Skip to main content

Logging with Datadog

Logging is a critical piece in any digital system. It enhances visibility, facilitates troubleshooting, and supports system reliability. With this recipe, you can implement a solution with Conscia that sends log entries to Datadog.

Overall Orchestration Flow​

Clients that invoke this orchestration flow can pass a logging message and log status (INFO, WARN, ERROR) to Datadog over HTTP.

In general, generic Components such as logging should not establish dependencies. Therefore, a common pattern is for a component that has something to log to use the logging component as a Sub Component.

Mapping Out DX Engine Elements​

This flow consists of three DX Engine elements:

  1. Datadog Connection (datadog-connection): A DX Engine Connection to Datadog.
  2. Datadog API Key (datadog-api-key): A Secret to store the Datadog token used by that Connection.
  3. Datadog Logger (datadog-log): An Orchestration Component that uses that Connection to log messages to Datadog.

DX Engine Configuration Details​

We can store the Datadog token as a secret in the DX Engine. To create the secret, in the Conscia UI:

  1. Click Settings in the top-nav, and then select Secrets.
  2. Click Add Secret.
  3. Set Secret Code to a value such as datadog-api-key.
  4. Set Secret Name to a value such as Datadog API Key.
  5. Set Secret Value to {Your Datadog token}.
  6. Optionally, enter a Description for the secret.
  7. Click Submit.

To create the Datadog Connection, in the Conscia UI:

  1. Click Settings in the top-nav, and then select Connections.
  2. Click Add Connection.
  3. Set Connection Code to a value such as datadog-connection.
  4. Set Connection Name to a value such as Datadog Connection.
  5. Optionally, enter a Connection Description for the connection.
  6. Select the Universal API Connector.
  7. Click Submit.

To configure the Datadog Connection, in the Conscia UI:

  1. Click the Edit button for the Datadog Logging Connection.
  2. Under Base URL, for Get value from, select Literal.
  3. Set the value of Base URL to the Datadog API base URL, such as https://http-intake.logs.{your-region}.datadoghq.com/api/v2/logs, replacing {your-region} with your Datadog server region.
  4. Under Base Headers, click Add another item.
  5. For the new header, set Header to DD-API-KEY.
  6. For the new header, set Value to Secret.
  7. For the value of the new header, select the secret created previously, such as Datadog API Key.
  8. Under Base Headers, click Add another item.
  9. For the new header, set Header to Content-Type.
  10. For the new header, set Value to Literal.
  11. For the value of the new header, enter application/json.
  12. Under Base Headers, click Add another item.
  13. For the new header, set Header to Accept.
  14. For the new header, set Value to Literal.
  15. For the value of the new header, enter application/json.
  16. Click Submit.

To create the Datadog Logger Orchestration Component, in the Conscia DX UI:

  1. In the top navigation, click Manage Experiences, and then click Components. The Manage Components page appears.
  2. Click Add Component. The Create Component wizard appears.
  3. For Component Code, enter an identifier for the component, such as datadog-log.
  4. For Name, enter a name for the component, such as Datadog Logger.
  5. Optionally, enter a Component Description.
  6. Select the No Rules checkbox. Experience Rules are not relevant to this Component.
  7. For Component Type, select Conscia - Universal API Connector. This Component Type can connect to almost any Webservice API.
  8. Optionally, select the Is Asynchronous checkbox.
  9. Click Submit.

The advantage of selecting Is Asynchronous is that the orchestration flow will not block until the Webservice call to Datadog completes. The disadvantage is that if logging fails, you will not receive notification.

Note that the Datadog API returns Request accepted for processing (always 202 empty JSON) when it successfully ingests a log entry.

To configure the Datadog Logger Orchestration Component, on the Manage Components page:

  1. Next to the Datadog Logger component, click the Edit button. The Edit Component wizard appears.
  2. For Connection, select the Datadog Connection Connection.
  3. For Webservice Path, select Literal, and leave the value blank.
  4. For Method, select POST.
  5. Under Body, for Get value from, select JS Expression, and then set the value to an expression such as the following:
[
  {
    "ddsource": "conscia",
    "ddtags": "env:staging",
    "hostname": "i-012345678",
    "message": "${contextField('logStatus')} ${contextField('logMessage')}",
    "service": "orchestration",
    "status": "${contextField('logStatus')}"
  }
]

Note that this body is an array; you could pass mulitple log entries in a single call.

You can pass context such as the following to the Datadog Logger Component:

{
    logEntry : {
        "logMessage": "This is the message to log.",
        "logSession": "{YOUR_INITIALS}",
        "logStatus": "DEBUG"
    }
}

Using Datadog Logger as a Sub Component​

To use the Datadog Logger Component as a Sub Component, in the DX Engine UI:

  1. In the top navigation, click Manage Experiences, and then click Components. The Manage Components page appears.
  2. Next to the component that will use the Datadog Logger Component as a Sub Component, click the Edit button. The Edit Component wizard appears.
  3. From the drop-down at the top right, select Sub Components. Note: the drop-down may have a scrollbar.
  4. Under Sub Components, click Add another item.
  5. For Property Name, enter a key in which the response from the Sub Component will appear in the response from the invoking Component.
  6. Under Component Codes, add the Datadog Logger (datadog-logger) Component.
  7. Under Context Field for Sub Component, click Add another item.
  8. For Context Field, enter logMessage.
  9. For Expression, enter a JavaScript expression that returns the data to log.
  10. Click Submit.

You can set Expression to a value such as response or a key within that value.

References​