Merge Two Sources into a Single Data Model
In the following recipe, we will merge content items from the CMS with product records from a commerce engine in order to populate product pages, whether a Product Detail Page (PDP), Product Listing Page (PLP), landing page, or anywhere products are featured with extended information. This is a typical use case in a headless commerce implementation -- needing to match up additional data with product data, be it pricing or inventory or product care instructions. Frequently developers write glue code in the frontend to address this use case and end up making multiple API calls and stitching together data from the backends. An orchestration layer can configure this data unification.
Overall Orchestration Flow
We will be using Storyblok as our headless CMS and commercetools as our commerce system. All commercetools products have a SKU, and any product related content items in Storyblok have a SKU field where business users can relate the records.
We will create two Orchestration Components to connect to each data source, and a third to match the records up by product SKU.
For a simpler mapping example that does not require matching up a series of objects by some value, see the Stitching Data from Multiple Sources recipe.
For a breakdown on different methods to merge and transform responses, see the Blending Responses tutorial.
Mapping Out DX Engine Elements
- Get Book Products: This Component connects to commercetools and contains a webservice call to retrieve all products of
productType = Book
. - Get Author Articles: This Component connects to Storyblok and contains a webservice call to retrieve all stories in a folder called "Product-Related Articles".
- Merge Book Product Data: This is a Data Transformation Script Component. It takes the responses from each of the components above, matches the records based on SKU, and maps the required properties for a final response to be consumed by the front end. In this example we're doing the equivalent of a left join, where all Book Products will be returned and if there is associated Storyblok data, it will be included. This Component type is entirely scripted, unlike the Object and Property Mapper Components which use a UI to do the majority of the mapping and transforming.
The end result will look something like this:
{
"response": [
{
"productId": "13ea40eb-1b9a-438d-8824-6347f96862e0",
"productName": {
"en-US": "The Hobbit"
},
"productSku": "hobbit-123",
"productDescription": {
"en-US": "Best selling fantasy fiction"
},
"productSlug": {
"en-US": "the-hobbit"
},
"articleName": "J.R.R. Tolkien: A Literary Legend and Father of Modern Fantasy",
"articleTeaser": "Explore the captivating life and extraordinary imagination of J.R.R. Tolkien, the brilliant author behind the beloved Middle-earth tales and the creator of a genre that continues to inspire generations of readers and writers alike."
},
{
"productId": "c836f0a8-25e1-4066-a2a0-29cbccbd6f05",
"productName": {
"en-US": "The Greatest Showman"
},
"productSku": "showman",
"productDescription": {
"en-US": "film adaptation"
},
"productSlug": {
"en-US": "the-greatest-showman"
}
},
{
"productId": "74555781-cf33-42c8-b8ec-aeeea5ee3230",
"productName": {
"en-US": "The Best Cookbook"
},
"productSku": "mealprep",
"productDescription": {
"en-US": "Student guide for quick eating"
},
"productSlug": {
"en-US": "the-best-cookbook"
}
},
{
"productId": "74e4c00b-3421-42fc-ab79-8df82dcb7160",
"productName": {
"en-US": "Mastering The Art Of French Cooking"
},
"productSku": "mastering-french-cooking-123",
"productDescription": {
"en-US": "The definitive cookbook on French cuisine for American readers"
},
"productSlug": {
"en-US": "mastering-the-art-of-french-cooking"
},
"articleName": "Julia Child: A Culinary TV Star",
"articleTeaser": "Explore the enduring legacy of Julia Child, the charismatic chef who revolutionized television with her approachable yet sophisticated cooking style. From her iconic French recipes to her captivating on-screen presence, discover how Julia Child became a beloved culinary icon."
}
]
}
DX Engine Configuration Details
Create a Connection to commercetools
- Navigate to the Connections page (Settings --> Connections).
- Click the Configure Connection button.
- Enter the following and click Submit:
Field | Value |
---|---|
Connection Code | commercetools-connection |
Connection Name | commercetools Connection |
Connector | Universal API Connector |
Base URL | Get value from: Literal https://api.{region}.gcp.commercetools.com/{channel} |
Base Headers | Header: Authorization Value: Bearer {token} |
Create a Connection to Storyblok
- Navigate to the Connections page (Settings --> Connections).
- Click the Configure Connection button.
- Enter the following and click Submit:
Field | Value |
---|---|
Connection Code | storyblok-connection |
Connection Name | Storyblok Connection |
Connector | Universal API Connector |
Base URL | Get value from: Literal https://api.storyblok.com/v2/cdn/stories |
Query Parameters | Parameter: token Value: {token} |
Create a Component to retrieve products from commercetools
- Navigate to the Experience Components page (Manage Experiences --> Components)
- Click the + Add Component button.
- Enter the following and click Submit:
Field | Value |
---|---|
Component Code | commercetools-get-products-books |
Component Name | Get Book Products - commercetools |
No Rules | Checked |
Component Type | Conscia - Universal API Connector |
The DX Engine will create and prepare the Component for further configuration. You should see the new component appear in the Component listing.
- Click Edit.
- Enter the following and click Submit:
Field | Form Tab | Value |
---|---|---|
Connection | Main | commercetools Connection |
Webservice Path | Main | Get value from: Literal/product-projections/search |
Method | Main | GET |
Query Parameters | Main | Parameter: filter Value (Literal): productType.id:"{id}" |
Create a Component to retrieve articles from Storyblok
- Navigate to the Experience Components page (Manage Experiences --> Components)
- Click the + Add Component button.
- Enter the following and click Submit:
Field | Value |
---|---|
Component Code | storyblok-get-author-articles |
Component Name | Get Author Articles - Storyblok |
No Rules | Checked |
Component Type | Conscia - Universal API Connector |
The DX Engine will create and prepare the Component for further configuration. You should see the new component appear in the Component listing.
- Click Edit.
- Enter the following and click Submit:
Field | Form Tab | Value |
---|---|---|
Connection | Main | Storyblok Connection |
Method | Main | GET |
Headers | Main | Header: Accept Value (Literal): application/json |
Headers | Main | Header: Content-Type Value (Literal): application/json |
Query Parameters | Main | Parameter: starts_with Value (Literal): product-related |
Create a Component to merge product and article data
- Navigate to the Experience Components page (Manage Experiences --> Components)
- Click the + Add Component button.
- Enter the following and click Submit:
Field | Value |
---|---|
Component Code | merge-book-product-data |
Component Name | Merge Book Product Data |
No Rules | Checked |
Component Type | Conscia - Data Transformation Script |
The DX Engine will create and prepare the Component for further configuration. You should see the new component appear in the Component listing.
- Click Edit.
- Enter the following and click Submit:
Field | Form Tab | Value |
---|---|---|
Connection | Main | Storyblok Connection |
Configuration - Data to Modify | Main | Get value from(JS Expression):_.assign({},{ "books" : componentResponse('commercetools-get-products-books').results, "articles" : componentResponse('storyblok-get-author-articles').stories }) Script: |
let bookProducts = data.books.map((book) => ({
productId: book.id,
productName: book.name,
productSku: book.masterVariant.sku,
productDescription: book.description,
productSlug: book.slug
}));
let result = bookProducts.filter((book) => {
let articleInfo = data.articles.filter((article) => {
return book.productSku === article.content.productsku;
});
book.articleName = articleInfo[0]?.name;
book.articleTeaser = articleInfo[0]?.content.teaser;
return book;
});
result