Skip to main content

Stitching Data from Multiple Sources

In this recipe, we will show you how to use the DX Engine to stitch data from multiple sources. We will use GitHub Gists to store different parts of a product detail page. We will then use the DX Engine to fetch the data from the Gists and stitch it together into a single JSON object.

  • Ensure you are logged into GitHub
  • Navigate to (https://gist.github.com/)
  • Enter a Gist description: DX Engine Tutorial
  • Add the following four files to the Gist:

product-detail.json

{
"id": "p01",
"categoryId": "cat05",
"name": "Bose QuietComfort Ultra",
"price": 549.00
}

product-category.json

{
"category_id": "cat05",
"name": "Headphones",
"description": "Tune out the world with these awesome headphones."
}

product-inventory-nl.json

{
"product_id": "01",
"nbrUnits": 341
}

product-inventory-gb.json

{
"product_id": "01",
"nbrUnits": 889
}

Alt text

Create a public GitHub Gist by clicking the Create secret gist option and the click the Create secret gist button.

Alt text

You should now see a screen that look like this:

Alt text

Take note of your GitHub username and Gist ID. It is in the URL of the browser. For example, the URL for the Gist above is: (https://gist.github.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d). The username is dremekie and the Gist ID is the last part of the URL: b4d7970ec9ac6eeca20de3ed82dfb43d. You will need this ID in the next step. The URL used to fetch the raw data or each of the four files is:

FileRaw Gist URL
product-detail.json(https://gist.githubusercontent.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d/raw/product-detail.json)
product-category.json(https://gist.githubusercontent.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d/raw/product-category.json)
product-inventory-gb.json(https://gist.githubusercontent.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d/raw/product-inventory-gb.json)
product-inventory-nl.json(https://gist.githubusercontent.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d/raw/product-inventory-nl.json)

We will create a new Component for each of the four files. Each Component will be responsible for fetching the data from a Gist. Then we will create a fifth Component that will use the data from the other four Components to create a single JSON object that could be used to, for example, render the product detail page.

Create a Connection to be used to fetch Gists

  • Navigate to the Connections page (Settings --> Connections).
  • Click the Configure Connection button
  • Enter the following and click Submit:
FieldValue
Connection Codeproduct-gists-connection
Connection NameConnection used to fetch Product Gists
ConnectorUniversal API Connector
Base URLGet value from: Literal
https://gist.githubusercontent.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d/raw
MethodGET

Create a Component to fetch product detail

  • Navigate to the Experience Components page (Manage Experiences --> Experience Components).
  • Click the Create Experience Component button
  • Enter the following and click Submit:
FieldValue
Component Codeproduct-detail-gist
Component NameProduct Detail Gist
No RulesChecked
Component TypeConscia - Universal API Connector

The DX Engine will create and prepare the Component for further configuration. You should see the Product Detail Gist component appear in the Component listing.

  • Click Edit.
  • Enter the following and click Submit.
FieldForm TabValue
ConnectionMainConnection used to fetch Product Gists
Webservice PathMainGet value from: Literal
/product-detail.json
MethodMainGET
HeadersMainHeader: content-type

Value (Literal):
application/json
tip

Click the Duplicate button on the Product Detail Gist component to create the next three components quickly.

Create a Component to fetch product category

FieldValue
Component Codeproduct-category-gist
Component NameProduct Detail Gist
No RulesChecked
Component TypeConscia - Universal API Connector
FieldForm TabValue
ConnectionMainConnection used to fetch Product Gists
Webservice PathMainGet value from: Literal
/product-category.json
MethodMainGET
HeadersMainHeader: content-type

Value (Literal):
application/json

Create a Component to fetch product inventory for Great Britain

FieldValue
Component Codeproduct-inventory-gb-gist
Component NameProduct Inventory Gist (GB)
No RulesChecked
Component TypeConscia - Universal API Connector
FieldForm TabValue
ConnectionMainConnection used to fetch Product Gists
Webservice PathMainGet value from: Literal
/product-inventory-gb.json
MethodMainGET
HeadersMainHeader: content-type

Value (Literal):
application/json

Create a Component to fetch product inventory for the Netherlands

FieldValue
Component Codeproduct-inventory-nl-gist
Component NameProduct Inventory Gist (NL)
No RulesChecked
Component TypeConscia - Universal API Connector
FieldForm TabValue
ConnectionMainConnection used to fetch Product Gists
Webservice PathMainGet value from: Literal
/product-inventory-nl.json
MethodMainGET
HeadersMainHeader: content-type

Value (Literal):
application/json

Query the data from the Gists

Let's query the data from the Gists. We will use the following query:

POST {{engineUrl}}/experience/components/_query
X-Customer-Code: {{customerCode}}
Authorization: Bearer {{dxEngineToken}}

{
"componentCodes": ["product-detail-gist", "product-category-gist", "product-inventory-gb-gist", "product-inventory-nl-gist"],
"context": {}
}

The response will be:

{
"duration": 114,
"components": {
"product-category-gist": {
"@extras": {
"rule": {
"metadata": [],
"attributes": {}
}
},
"status": "VALID",
"response": {
"category_id": "cat05",
"name": "Headphones",
"description": "Tune out the world with these awesome headphones!"
}
},
"product-inventory-nl-gist": {
"@extras": {
"rule": {
"metadata": [],
"attributes": {}
}
},
"status": "VALID",
"response": {
"product_id": "01",
"nbrUnits": 341
}
},
"product-detail-gist": {
"@extras": {
"rule": {
"metadata": [],
"attributes": {}
}
},
"status": "VALID",
"response": {
"id": "p01",
"categoryId": "cat05",
"name": "Bose QuietComfort Ultra",
"price": 549
}
},
"product-inventory-gb-gist": {
"@extras": {
"rule": {
"metadata": [],
"attributes": {}
}
},
"status": "VALID",
"response": {
"product_id": "01",
"nbrUnits": 889
}
}
}
}

Create a Component to stitch the data from the Gists

Now we will use the Object Mapper Component to stitch together the data from the four Gists into one product JSON.

FieldValue
Component Codeproduct-object
Component NameThe combined product record from various sources
No RulesChecked
Component TypeConscia - Object Mapper

The DX Engine will create and prepare the Component for further configuration. You should see the mapper component appear in the Component listing.

  • Click Edit.
  • Enter the following and click Submit.

We're going to add four Object Maps (one for each Gist) to the Object Mapper Component. Each Object Map will be responsible for mapping the data from a single Gist to the final product JSON.

Under each the Object Maps section of the form click the + button to add a new Object Map.

Object Map for product-detail-gist

FieldValue
Source DataGet value from: Component Response
Product Detail Gist
Expression TypeJavaScript
SchemaproductId = data.id
name = data.name
value = data.price * 100
unit = 'CENTS'

The schema to look like this:

Alt text

Object Map for product-category-gist

FieldValue
Source DataGet value from: Component Response
Product Category Gist
Expression TypeJSONPath
SchemacategoryName = $.name

The schema to look like this:

Alt text

Object Map for product-inventory-gb-gist

FieldValue
Source DataGet value from: Component Response
Product Inventory Gist (GB)
Expression TypeJavaScript
Schemainventory_gb.count = data.nbrUnits

The schema to look like this:

Alt text

Object Map for product-inventory-nl-gist

FieldValue
Source DataGet value from: Component Response
Product Inventory Gist (NL)
Expression TypeJavaScript
Schemainventory_nl.count = data.nbrUnits

The schema to look like this:

Alt text

Query the merged data

POST {{engineUrl}}/experience/components/_query
X-Customer-Code: {{customerCode}}
Authorization: Bearer {{dxEngineToken}}

{
"componentCodes": ["product-object"],
"context": {}
}

The response will look this:

{
"duration": 110,
"components": {
"product-object": {
"@extras": {
"rule": {
"metadata": [],
"attributes": {}
}
},
"status": "VALID",
"response": {
"productId": "p01",
"name": "Bose QuietComfort Ultra",
"value": 54900,
"unit": "CENTS",
"categoryName": [
"Headphones"
],
"inventory_gb": {
"count": 889
},
"inventory_nl": {
"count": 341
}
}
}
}
}

Note that DX Engine will execute the four other Components in parallel to fetch the Gists and then execute the Object Mapper Component to stitch the data together.