Stitching Data from Multiple Sources

In this recipe, we will show you how to use the DX Engine to stitch data from multiple sources. We will use GitHub Gists to store different parts of a product detail page. We will then use the DX Engine to fetch the data from the Gists and stitch it together into a single JSON object.

• Ensure you are logged into GitHub
• Navigate to: https://gist.github.com/
• Enter a Gist description: DX Engine Tutorial
• Add the following four files to the Gist:

product-detail.json

{
  "id": "p01",
  "categoryId": "cat05",
  "name": "Bose QuietComfort Ultra",
  "price": 549.00
}

product-category.json

{
  "category_id": "cat05",
  "name": "Headphones",
  "description": "Tune out the world with these awesome headphones."
}

product-inventory-nl.json

{
  "product_id": "01",
  "nbrUnits": 341
}

product-inventory-gb.json

{
  "product_id": "01",
  "nbrUnits": 889
}

Create a public GitHub Gist by clicking the Create secret gist option and the click the Create secret gist button.

You should now see a screen that look like this:

Take note of your GitHub username and Gist ID. It is in the URL of the browser. For example, the URL for the Gist above is: https://gist.github.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d. The username is dremekie and the Gist ID is the last part of the URL: b4d7970ec9ac6eeca20de3ed82dfb43d. You will need this ID in the next step. The URL used to fetch the raw data or each of the four files is:

File	Raw Gist URL
product-detail.json	https://gist.githubusercontent.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d/raw/product-detail.json
product-category.json	https://gist.githubusercontent.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d/raw/product-category.json
product-inventory-gb.json	https://gist.githubusercontent.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d/raw/product-inventory-gb.json
product-inventory-nl.json	https://gist.githubusercontent.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d/raw/product-inventory-nl.json

We will create a new Component for each of the four files. Each Component will be responsible for fetching the data from a Gist. Then we will create a fifth Component that will use the data from the other four Components to create a single JSON object that could be used to, for example, render the product detail page.

Create a Connection to be used to fetch Gists

• Navigate to the Connections page (Settings --> Connections).
• Click the Configure Connection button
• Enter the following and click Submit:

Field	Value
Connection Code	product-gists-connection
Connection Name	Connection used to fetch Product Gists
Connector	Universal API Connector
Base URL	Get value from: Literal https://gist.githubusercontent.com/dremekie/b4d7970ec9ac6eeca20de3ed82dfb43d/raw
Method	GET

Create a Component to fetch product detail

• Navigate to the Experience Components page (Manage Experiences --> Experience Components).
• Click the Create Experience Component button
• Enter the following and click Submit:

Field	Value
Component Code	product-detail-gist
Component Name	Product Detail Gist
No Rules	Checked
Component Type	Conscia - Universal API Connector

The DX Engine will create and prepare the Component for further configuration. You should see the Product Detail Gist component appear in the Component listing.

• Click Edit.
• Enter the following and click Submit.

Field	Form Tab	Value
Connection	Main	Connection used to fetch Product Gists
Webservice Path	Main	Get value from: Literal /product-detail.json
Method	Main	GET
Headers	Main	Header: content-type Value (Literal): application/json

tip

Click the Duplicate button on the Product Detail Gist component to create the next three components quickly.

Create a Component to fetch product category

Field	Value
Component Code	product-category-gist
Component Name	Product Detail Gist
No Rules	Checked
Component Type	Conscia - Universal API Connector

Field	Form Tab	Value
Connection	Main	Connection used to fetch Product Gists
Webservice Path	Main	Get value from: Literal /product-category.json
Method	Main	GET
Headers	Main	Header: content-type Value (Literal): application/json

Create a Component to fetch product inventory for Great Britain

Field	Value
Component Code	product-inventory-gb-gist
Component Name	Product Inventory Gist (GB)
No Rules	Checked
Component Type	Conscia - Universal API Connector

Field	Form Tab	Value
Connection	Main	Connection used to fetch Product Gists
Webservice Path	Main	Get value from: Literal /product-inventory-gb.json
Method	Main	GET
Headers	Main	Header: content-type Value (Literal): application/json

Create a Component to fetch product inventory for the Netherlands

Field	Value
Component Code	product-inventory-nl-gist
Component Name	Product Inventory Gist (NL)
No Rules	Checked
Component Type	Conscia - Universal API Connector

Field	Form Tab	Value
Connection	Main	Connection used to fetch Product Gists
Webservice Path	Main	Get value from: Literal /product-inventory-nl.json
Method	Main	GET
Headers	Main	Header: content-type Value (Literal): application/json

Query the data from the Gists

Let's query the data from the Gists. We will use the following query:

POST {{engineUrl}}/experience/components/_query
X-Customer-Code: {{customerCode}}
Authorization: Bearer {{dxEngineToken}}

{
  "componentCodes": ["product-detail-gist", "product-category-gist", "product-inventory-gb-gist", "product-inventory-nl-gist"],
  "context": {}
}

The response will be:

{
  "duration": 114,
  "components": {
    "product-category-gist": {
      "@extras": {
        "rule": {
          "metadata": [],
          "attributes": {}
        }
      },
      "status": "VALID",
      "response": {
        "category_id": "cat05",
        "name": "Headphones",
        "description": "Tune out the world with these awesome headphones!"
      }
    },
    "product-inventory-nl-gist": {
      "@extras": {
        "rule": {
          "metadata": [],
          "attributes": {}
        }
      },
      "status": "VALID",
      "response": {
        "product_id": "01",
        "nbrUnits": 341
      }
    },
    "product-detail-gist": {
      "@extras": {
        "rule": {
          "metadata": [],
          "attributes": {}
        }
      },
      "status": "VALID",
      "response": {
        "id": "p01",
        "categoryId": "cat05",
        "name": "Bose QuietComfort Ultra",
        "price": 549
      }
    },
    "product-inventory-gb-gist": {
      "@extras": {
        "rule": {
          "metadata": [],
          "attributes": {}
        }
      },
      "status": "VALID",
      "response": {
        "product_id": "01",
        "nbrUnits": 889
      }
    }
  }
}

Create a Component to stitch the data from the Gists

Now we will use the Object Mapper Component to stitch together the data from the four Gists into one product JSON.

Field	Value
Component Code	product-object
Component Name	The combined product record from various sources
No Rules	Checked
Component Type	Conscia - Object Mapper

The DX Engine will create and prepare the Component for further configuration. You should see the mapper component appear in the Component listing.

• Click Edit.
• Enter the following and click Submit.

We're going to add four Object Maps (one for each Gist) to the Object Mapper Component. Each Object Map will be responsible for mapping the data from a single Gist to the final product JSON.

Under each the Object Maps section of the form click the + button to add a new Object Map.

Object Map for product-detail-gist

Field	Value
Source Data	Get value from: Component Response Product Detail Gist
Expression Type	JavaScript
Schema	productId = data.id name = data.name value = data.price * 100 unit = 'CENTS'

The schema to look like this:

Object Map for product-category-gist

Field	Value
Source Data	Get value from: Component Response Product Category Gist
Expression Type	JSONPath
Schema	categoryName = $.name

The schema to look like this:

Object Map for product-inventory-gb-gist

Field	Value
Source Data	Get value from: Component Response Product Inventory Gist (GB)
Expression Type	JavaScript
Schema	inventory_gb.count = data.nbrUnits

The schema to look like this:

Object Map for product-inventory-nl-gist

Field	Value
Source Data	Get value from: Component Response Product Inventory Gist (NL)
Expression Type	JavaScript
Schema	inventory_nl.count = data.nbrUnits

The schema to look like this:

Query the merged data

POST {{engineUrl}}/experience/components/_query
X-Customer-Code: {{customerCode}}
Authorization: Bearer {{dxEngineToken}}

{
  "componentCodes": ["product-object"],
  "context": {}
}

The response will look this:

{
  "duration": 110,
  "components": {
    "product-object": {
      "@extras": {
        "rule": {
          "metadata": [],
          "attributes": {}
        }
      },
      "status": "VALID",
      "response": {
        "productId": "p01",
        "name": "Bose QuietComfort Ultra",
        "value": 54900,
        "unit": "CENTS",
        "categoryName": [
          "Headphones"
        ],
        "inventory_gb": {
          "count": 889
        },
        "inventory_nl": {
          "count": 341
        }
      }
    }
  }
}

Note that DX Engine will execute the four other Components in parallel to fetch the Gists and then execute the Object Mapper Component to stitch the data together.