Skip to main content

Getting Data into Intel

Create a Data Bucket to hold transaction data​

PUT https://io.conscia.ai/vue/_api/v1/data-buckets/clickstream
content-type: application/json
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}

{
  "name": "Clickstream Data"
}

Create a Data Table within the Data Bucket​

PUT https://io.conscia.ai/vue/_api/v1/data-buckets/clickstream/data-tables/clickstream_user_event
Authorization: Bearer {{apiKey}}
content-type: application/json
X-Customer-Code: {{customerCode}}

{
  "name": "Clickstream - User Events",
  "dataTableConfig": {
    "schema": [
      {"fieldName": "DATE", "incomingDatatype": "varchar", "processedDatatype": "DATE"},
      {"fieldName": "VISITOR_ID", "incomingDatatype": "varchar", "processedDatatype": "varchar" },
      {"fieldName": "VISIT_NUM", "incomingDatatype": "varchar", "processedDatatype": "varchar"},
      {"fieldName": "DEVICE_TYPE", "incomingDatatype": "varchar", "processedDatatype": "varchar"},
      {"fieldName": "PAGE_VIEWS", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "PRODUCT_VIEWS", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "VISITS_PIP", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "CART_ADDITIONS", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "VISITS_CART_ADDITIONS", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "CART_REMOVALS", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "VISITS_CART_REMOVALS", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "ORDERS", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "VISITS_ORDERS", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "REVENUE", "incomingDatatype": "varchar", "processedDatatype": "double"},
      {"fieldName": "SEARCHES", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "VISITS_SEARCHES", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "CHECKOUT_INTENT", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "VISITS_CHECKOUT_INTENT", "incomingDatatype": "varchar", "processedDatatype": "integer"},
      {"fieldName": "PRODUCT_ENGAGED", "incomingDatatype": "varchar", "processedDatatype": "integer"}
    ],
    "format": "CSV",
    "csv_escape": "\\",
    "csv_quote": "\"",
    "csv_separator": ",",
    "skip_header_line_count": 1,
    "partition": ["bucket(VISITOR_ID, 10)"],
    "bucketCount": 50,
    "timestampField": "DATE"
  },
  "partitionGranularity": "MONTH"
}

There are two underlying datasets to a Data Table. The incoming and processed dataset. The uploaded files are considered the incoming dataset. When a time partitiopn is "processed", data from the incoming dataset (that fall within the specified time partition) are put into the processed dataset in a highly-efficient columnar data format for querying.

Upload data​

POST https://io.conscia.ai/vue/_api/v1/data-buckets/clickstream/data-tables/clickstream_user_event/incoming/upload
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}
Content-Type: multipart/form-data; boundary=----someboundary

------someboundary
Content-Disposition: form-data; name="file[]"; filename="clickstream-user-event-000000000000.gz"

< ./data/clickstream-user-event-000000000000.gz
------someboundary--

Get incoming record counts​

The following endpoint will get the record count of the uploaded files. Specific files can be scanned by specifying filename in the query parameter as a filename pattern. e.g. clickstream-user-event-%10.gz

GET https://io.conscia.ai/vue/_api/v1/data-buckets/clickstream/data-tables/clickstream_user_event/incoming/counts
Authorization: Bearer {{apiKey}}
content-type: application/json
X-Customer-Code: {{customerCode}}

Get a sample of the data records from an uploaded file​

GET https://io.conscia.ai/vue/_api/v1/data-buckets/clickstream/data-tables/clickstream_user_event/incoming/records?filename=clickstream-user-event-000000000009.gz
Authorization: Bearer {{apiKey}}
content-type: application/json
X-Customer-Code: {{customerCode}}

Sample response:

{
  "results": [
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... }
  ],
  "recordCount": 10
}

Process incoming data records​

The filenamePattern specifies which uploaded files should be processed.

POST https://io.conscia.ai/vue/_api/v1/data-buckets/clickstream/data-tables/clickstream_user_event/incoming/_processDataRecords
Authorization: Bearer {{apiKey}}
content-type: application/json
X-Customer-Code: {{customerCode}}

{
  "filenamePattern": "%",
  "partition": {
    "year": 2022,
    "month": 6
  }
}

List the processed records counts for a Data Table​

GET https://io.conscia.ai/vue/_api/v1/data-buckets/clickstream/data-tables/clickstream_user_event/processed/counts?partitionGranularity=MONTH
Authorization: Bearer {{apiKey}}
content-type: application/json
X-Customer-Code: {{customerCode}}

Sample response:

{
  "results": [
    {
      "year": 2021,
      "month": 11,
      "nbrOfRecords": 27221310
    },
    {
      "year": 2021,
      "month": 12,
      "nbrOfRecords": 24323328
    },
    {
      "year": 2022,
      "month": 1,
      "nbrOfRecords": 23778992
    },
    {
      "year": 2022,
      "month": 2,
      "nbrOfRecords": 18937676
    },
    {
      "year": 2022,
      "month": 3,
      "nbrOfRecords": 23206144
    },
    {
      "year": 2022,
      "month": 4,
      "nbrOfRecords": 27176528
    },
    {
      "year": 2022,
      "month": 5,
      "nbrOfRecords": 31089591
    },
    {
      "year": 2022,
      "month": 6,
      "nbrOfRecords": 947719
    }
  ],
  "recordCount": 8
}
GET https://io.conscia.ai/vue/_api/v1/data-buckets/clickstream/data-tables/clickstream_user_event/processed/counts?partitionGranularity=YEAR
Authorization: Bearer {{apiKey}}
content-type: application/json
X-Customer-Code: {{customerCode}}

Sample response:

{
  "results": [
    {
      "year": 2021,
      "nbrOfRecords": 51544638
    },
    {
      "year": 2022,
      "nbrOfRecords": 125136650
    }
  ],
  "recordCount": 2
}