Skip to main content

Getting Data into DX Graph

Using data from files​

Uploading a file to the Incoming Data Bucket​

Data files are uploaded to Data Buckets using the standard multi-part form POST. First, upload a file into the incoming Data Bucket.

curl --request POST \
  --url https://io.conscia.ai/vue/_api/v1/data-buckets/incoming/upload \
  --header 'authorization: Bearer {{apiKey}}' \
  --header 'X-Customer-Code: {{customerCode}}' \
  --form 'file[]=@customers01.csv.gz'

Importing data from a uploaded files into a Collection​

Once file(s) are uploaded into the incoming Data Bucket, the data in those files can be imported into a Data Collection using the importDataFeed endpoint.

POST https://io.conscia.ai/vue/_api/v1/collections/promotion/records/_importDataFeed
content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}

{
  "feedType": "full",
  "dataFeedImportConfiguration": {
    "format": "DELIMITED",
    "gzipped": true,
    "delimitedOptions": {
      "delimiter": ",",
      "escape": "\\",
      "quote": "\""
    },
    "filenamePattern": "customers*.csv.gz",
    "incomingDataBucketCode": "incoming",
    "skippedDataBucketCode": "skipped",
    "processedDataBucketCode": "processed"
  },  
  "sendCompletionEmail": false,
  "sendFailureEmail": false
}

Note that the incoming Data Bucket may have contain files that are not meant for the customer-profile collection. That is whhy the filenamePattern configuration is important as it specified which files should be imported.

Successfully imported files will be moved into the processed Data Bucket. Files that are skipped are moved into the skipped Data Bucket.

When scanning for files to process (that match filenamePattern), files are first sorted alphabetically.

Note: It is important that the filenames are named such that an alphabetical sort results in the order that the files should be scanned. This typically means it should be timestamped. Examples: customer-2022-02-07, customer-2022-02-08, customer-2022-02-09T17:00:21.

The feedType configuration can be full (each file will include a full set of data) or partial (each file includes only records to be updated). If the feedType is full, the last file found that matches the given filename pattern is loaded and all other files (if any) are moved to the skipped Data Bucket. If the feedType is partial, the each file loaded in the order the files are are found.

Using the API​

Records can be created, updated and removed synchronously and asynchronously. Synchronous means that when the called webservice returns, the change to the data has been made. Asynchronous calls will perform the actual change to the data at as soon as possible.

Adding records to a Collection synchronously​

Synchronous endpoints are rate-limited to 10 calls per second.

POST https://io.conscia.ai/vue/_api/v1/collections/customer/records
Content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}

{
  "dataRecords": [
    { "customer_id": "123", "salutation": "Ms.", "first_name": "Pamela", "last_name": "Elaine" },
    { "customer_id": "222", "salutation": "Mr.", "first_name": "Hugo", "last_name": "Florence" }
  ],
  "ifExists": "ignore"
}

Sample response:

{
  "data": [
    {
      "new": {
        "customer_id": "123",
        "salutation": "Ms.",
        "first_name": "Florence",
        "last_name": "Remekie",
        "dataRecordIdentifier": "123",
        "@iat": 1657577883543,
        "@uat": 1657577883543
      }
    },
    {
      "new": {
        "customer_id": "222",
        "salutation": "Mr.",
        "first_name": "Hugo",
        "last_name": "Remekie",
        "dataRecordIdentifier": "222",
        "@iat": 1657577883543,
        "@uat": 1657577883543
      }
    }
  ]
}

Adding records to a Collection asynchronously​

Asynchronous endpoints will queue a job that will peform the actual work and return a job ID. The status of the job can be queried at a later point to get the status Asynchronous endpoints have no rate limits.

POST https://io.conscia.ai/vue/_api/v1/collections/customer/records/_async
content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}

{
  "dataRecords": [
    { "customer_id": "123", "salutation": "Ms.", "first_name": "Pamela", "last_name": "Elaine" },
    { "customer_id": "222", "salutation": "Mr.", "first_name": "Hugo", "last_name": "Florence" }
  ],
  "ifExists": "ignore"
}

Sample response:

{
  "jobID": "20220711-1f30b3c3-542e-4c94-a029-d4128fe7f4ab"
}

To fetch the job's status:

GET https://io.conscia.ai/vue/_api/v1/jobs/20220711-1f30b3c3-542e-4c94-a029-d4128fe7f4ab/status
content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}