Getting Data into DX Graph
Using data from files
Uploading a file to the Incoming Data Bucket
Data files are uploaded to Data Buckets using the standard multi-part form POST. First, upload a file into the incoming
Data Bucket.
curl --request POST \
--url https://io.conscia.ai/vue/_api/v1/data-buckets/incoming/upload \
--header 'authorization: Bearer {{apiKey}}' \
--header 'X-Customer-Code: {{customerCode}}' \
--form 'file[]=@customers01.csv.gz'
Importing data from a uploaded files into a Collection
Once file(s) are uploaded into the incoming
Data Bucket, the data in those files can be imported into a Data Collection using the importDataFeed endpoint.
POST https://io.conscia.ai/vue/_api/v1/collections/promotion/records/_importDataFeed
content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}
{
"feedType": "full",
"dataFeedImportConfiguration": {
"format": "DELIMITED",
"gzipped": true,
"delimitedOptions": {
"delimiter": ",",
"escape": "\\",
"quote": "\""
},
"filenamePattern": "customers*.csv.gz",
"incomingDataBucketCode": "incoming",
"skippedDataBucketCode": "skipped",
"processedDataBucketCode": "processed"
},
"sendCompletionEmail": false,
"sendFailureEmail": false
}
Note that the incoming Data Bucket may have contain files that are not meant for the customer-profile
collection. That is whhy the filenamePattern
configuration is important as it specified which files should be imported.
Successfully imported files will be moved into the processed
Data Bucket. Files that are skipped are moved into the skipped
Data Bucket.
When scanning for files to process (that match filenamePattern
), files are first sorted alphabetically.
Note: It is important that the filenames are named such that an alphabetical sort results in the order that the files should be scanned. This typically means it should be timestamped. Examples:
customer-2022-02-07
,customer-2022-02-08
,customer-2022-02-09T17:00:21
.
The feedType
configuration can be full
(each file will include a full set of data) or partial
(each file includes only records to be updated). If the feedType
is full
, the last file found that matches the given filename pattern is loaded and all other files (if any) are moved to the skipped
Data Bucket. If the feedType
is partial
, the each file loaded in the order the files are are found.
Using the API
Records can be created, updated and removed synchronously and asynchronously. Synchronous means that when the called webservice returns, the change to the data has been made. Asynchronous calls will perform the actual change to the data at as soon as possible.
Adding records to a Collection synchronously
Synchronous endpoints are rate-limited to 10 calls per second.
POST https://io.conscia.ai/vue/_api/v1/collections/customer/records
Content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}
{
"dataRecords": [
{ "customer_id": "123", "salutation": "Ms.", "first_name": "Pamela", "last_name": "Elaine" },
{ "customer_id": "222", "salutation": "Mr.", "first_name": "Hugo", "last_name": "Florence" }
],
"ifExists": "ignore"
}
Sample response:
{
"data": [
{
"new": {
"customer_id": "123",
"salutation": "Ms.",
"first_name": "Florence",
"last_name": "Remekie",
"dataRecordIdentifier": "123",
"@iat": 1657577883543,
"@uat": 1657577883543
}
},
{
"new": {
"customer_id": "222",
"salutation": "Mr.",
"first_name": "Hugo",
"last_name": "Remekie",
"dataRecordIdentifier": "222",
"@iat": 1657577883543,
"@uat": 1657577883543
}
}
]
}
Adding records to a Collection asynchronously
Asynchronous endpoints will queue a job that will peform the actual work and return a job ID. The status of the job can be queried at a later point to get the status Asynchronous endpoints have no rate limits.
POST https://io.conscia.ai/vue/_api/v1/collections/customer/records/_async
content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}
{
"dataRecords": [
{ "customer_id": "123", "salutation": "Ms.", "first_name": "Pamela", "last_name": "Elaine" },
{ "customer_id": "222", "salutation": "Mr.", "first_name": "Hugo", "last_name": "Florence" }
],
"ifExists": "ignore"
}
Sample response:
{
"jobID": "20220711-1f30b3c3-542e-4c94-a029-d4128fe7f4ab"
}
To fetch the job's status:
GET https://io.conscia.ai/vue/_api/v1/jobs/20220711-1f30b3c3-542e-4c94-a029-d4128fe7f4ab/status
content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}