Skip to main content

Getting Data into DX Graph

There are three ways to get data into the DX Graph:

  • Drop files in a bucket where DX Graph can load them on a schedule or based on an event trigger (JSON, delimited, XML)
  • Use APIs to push data into DX Graph
  • Schedule a job to pull data from an external source - this requires having an API endpoint that we we can connect to

Loading data from files​

Uploading a file to the Incoming Data Bucket​

Data files are uploaded to Data Buckets using the standard multi-part form POST. First, upload a file into the incoming Data Bucket.

curl --request POST \
  --url https://io.conscia.ai/vue/_api/v1/buckets/incoming/upload \
  --header 'authorization: Bearer {{apiKey}}' \
  --header 'X-Customer-Code: {{customerCode}}' \
  --form 'file[]=@customers01.csv.gz'

Importing data from uploaded files into a Collection​

For documentation on this topic, please visit Working with Data Files.

Push Data into DX Graph Using the API​

Records can be created, updated and removed synchronously and asynchronously. Synchronous means that when the called webservice returns, the change to the data has been made. Asynchronous calls will perform the actual change to the data as soon as possible.

Adding records to a Collection synchronously​

Synchronous endpoints are rate-limited to 10 calls per second.

POST https://io.conscia.ai/vue/_api/v1/collections/customer/records
Content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}

{
  "dataRecords": [
    { "customer_id": "123", "salutation": "Ms.", "first_name": "Pamela", "last_name": "Elaine" },
    { "customer_id": "222", "salutation": "Mr.", "first_name": "Hugo", "last_name": "Florence" }
  ],
  "ifExists": "ignore"
}

Sample response:

{
  "data": [
    {
      "new": {
        "customer_id": "123",
        "salutation": "Ms.",
        "first_name": "Florence",
        "last_name": "Remekie",
        "dataRecordIdentifier": "123",
        "@iat": 1657577883543,
        "@uat": 1657577883543
      }
    },
    {
      "new": {
        "customer_id": "222",
        "salutation": "Mr.",
        "first_name": "Hugo",
        "last_name": "Remekie",
        "dataRecordIdentifier": "222",
        "@iat": 1657577883543,
        "@uat": 1657577883543
      }
    }
  ]
}

Adding records to a Collection asynchronously​

Asynchronous endpoints will queue a job that will peform the actual work and return a job ID. The status of the job can be queried at a later point to get the status. Asynchronous endpoints have no rate limits.

POST https://io.conscia.ai/vue/_api/v1/collections/customer/records/_async
content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}

{
  "dataRecords": [
    { "customer_id": "123", "salutation": "Ms.", "first_name": "Pamela", "last_name": "Elaine" },
    { "customer_id": "222", "salutation": "Mr.", "first_name": "Hugo", "last_name": "Florence" }
  ],
  "ifExists": "ignore"
}

Sample response:

{
  "jobID": "20220711-1f30b3c3-542e-4c94-a029-d4128fe7f4ab"
}

To fetch the job's status:

GET https://io.conscia.ai/vue/_api/v1/jobs/20220711-1f30b3c3-542e-4c94-a029-d4128fe7f4ab/status
content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}

Pulling Data from an External API​

For information on how to pull data from an external system via APIs, please see this link.