Skip to main content

Merge Two Sources into a Single Data Model

In the following recipe, we will merge content items from the CMS with product records from a commerce engine in order to populate product pages, whether a Product Detail Page (PDP), Product Listing Page (PLP), landing page, or anywhere products are featured with extended information. This is a typical use case in a headless commerce implementation -- needing to match up additional data with product data, be it pricing or inventory or product care instructions. Frequently developers write glue code in the frontend to address this use case and end up making multiple API calls and stitching together data from the backends. An orchestration layer can configure this data unification.

Overall Orchestration Flow

We will be using Storyblok as our headless CMS and commercetools as our commerce system. All commercetools products have a SKU, and any product related content items in Storyblok have a SKU field where business users can relate the records.

We will create two Orchestration Components to connect to each data source, and a third to match the records up by product SKU.

For a simpler mapping example that does not require matching up a series of objects by some value, see the Stitching Data from Multiple Sources recipe.

For a breakdown on different methods to merge and transform responses, see the Blending Responses tutorial.

Mapping Out DX Engine Elements

Merge Visualizer

  • Get Book Products: This Component connects to commercetools and contains a webservice call to retrieve all products of productType = Book.
  • Get Author Articles: This Component connects to Storyblok and contains a webservice call to retrieve all stories in a folder called "Product-Related Articles".
  • Merge Book Product Data: This is a Data Transformation Script Component. It takes the responses from each of the components above, matches the records based on SKU, and maps the required properties for a final response to be consumed by the front end. In this example we're doing the equivalent of a left join, where all Book Products will be returned and if there is associated Storyblok data, it will be included. This Component type is entirely scripted, unlike the Object and Property Mapper Components which use a UI to do the majority of the mapping and transforming.

The end result will look something like this:

{
"response": [
{
"productId": "13ea40eb-1b9a-438d-8824-6347f96862e0",
"productName": {
"en-US": "The Hobbit"
},
"productSku": "hobbit-123",
"productDescription": {
"en-US": "Best selling fantasy fiction"
},
"productSlug": {
"en-US": "the-hobbit"
},
"articleName": "J.R.R. Tolkien: A Literary Legend and Father of Modern Fantasy",
"articleTeaser": "Explore the captivating life and extraordinary imagination of J.R.R. Tolkien, the brilliant author behind the beloved Middle-earth tales and the creator of a genre that continues to inspire generations of readers and writers alike."
},
{
"productId": "c836f0a8-25e1-4066-a2a0-29cbccbd6f05",
"productName": {
"en-US": "The Greatest Showman"
},
"productSku": "showman",
"productDescription": {
"en-US": "film adaptation"
},
"productSlug": {
"en-US": "the-greatest-showman"
}
},
{
"productId": "74555781-cf33-42c8-b8ec-aeeea5ee3230",
"productName": {
"en-US": "The Best Cookbook"
},
"productSku": "mealprep",
"productDescription": {
"en-US": "Student guide for quick eating"
},
"productSlug": {
"en-US": "the-best-cookbook"
}
},
{
"productId": "74e4c00b-3421-42fc-ab79-8df82dcb7160",
"productName": {
"en-US": "Mastering The Art Of French Cooking"
},
"productSku": "mastering-french-cooking-123",
"productDescription": {
"en-US": "The definitive cookbook on French cuisine for American readers"
},
"productSlug": {
"en-US": "mastering-the-art-of-french-cooking"
},
"articleName": "Julia Child: A Culinary TV Star",
"articleTeaser": "Explore the enduring legacy of Julia Child, the charismatic chef who revolutionized television with her approachable yet sophisticated cooking style. From her iconic French recipes to her captivating on-screen presence, discover how Julia Child became a beloved culinary icon."
}
]
}

DX Engine Configuration Details

Create a Connection to commercetools

  • Navigate to the Connections page (Settings --> Connections).
  • Click the Configure Connection button.
  • Enter the following and click Submit:
FieldValue
Connection Codecommercetools-connection
Connection Namecommercetools Connection
ConnectorUniversal API Connector
Base URLGet value from: Literal
https://api.{region}.gcp.commercetools.com/{channel}
Base HeadersHeader: Authorization
Value: Bearer {token}

Create a Connection to Storyblok

  • Navigate to the Connections page (Settings --> Connections).
  • Click the Configure Connection button.
  • Enter the following and click Submit:
FieldValue
Connection Codestoryblok-connection
Connection NameStoryblok Connection
ConnectorUniversal API Connector
Base URLGet value from: Literal
https://api.storyblok.com/v2/cdn/stories
Query ParametersParameter: token
Value: {token}

Create a Component to retrieve products from commercetools

  • Navigate to the Experience Components page (Manage Experiences --> Components)
  • Click the + Add Component button.
  • Enter the following and click Submit:
FieldValue
Component Codecommercetools-get-products-books
Component NameGet Book Products - commercetools
No RulesChecked
Component TypeConscia - Universal API Connector

The DX Engine will create and prepare the Component for further configuration. You should see the new component appear in the Component listing.

  • Click Edit.
  • Enter the following and click Submit:
FieldForm TabValue
ConnectionMaincommercetools Connection
Webservice PathMainGet value from: Literal
/product-projections/search
MethodMainGET
Query ParametersMainParameter: filter
Value (Literal): productType.id:"{id}"

Create a Component to retrieve articles from Storyblok

  • Navigate to the Experience Components page (Manage Experiences --> Components)
  • Click the + Add Component button.
  • Enter the following and click Submit:
FieldValue
Component Codestoryblok-get-author-articles
Component NameGet Author Articles - Storyblok
No RulesChecked
Component TypeConscia - Universal API Connector

The DX Engine will create and prepare the Component for further configuration. You should see the new component appear in the Component listing.

  • Click Edit.
  • Enter the following and click Submit:
FieldForm TabValue
ConnectionMainStoryblok Connection
MethodMainGET
HeadersMainHeader: Accept
Value (Literal): application/json
HeadersMainHeader: Content-Type
Value (Literal): application/json
Query ParametersMainParameter: starts_with
Value (Literal): product-related

Create a Component to merge product and article data

  • Navigate to the Experience Components page (Manage Experiences --> Components)
  • Click the + Add Component button.
  • Enter the following and click Submit:
FieldValue
Component Codemerge-book-product-data
Component NameMerge Book Product Data
No RulesChecked
Component TypeConscia - Data Transformation Script

The DX Engine will create and prepare the Component for further configuration. You should see the new component appear in the Component listing.

  • Click Edit.
  • Enter the following and click Submit:
FieldForm TabValue
ConnectionMainStoryblok Connection
Configuration - Data to ModifyMainGet value from(JS Expression):
_.assign({},{ "books" : componentResponse('commercetools-get-products-books').results, "articles" : componentResponse('storyblok-get-author-articles').stories })
Script:
let bookProducts = data.books.map((book) => ({
productId: book.id,
productName: book.name,
productSku: book.masterVariant.sku,
productDescription: book.description,
productSlug: book.slug
}));

let result = bookProducts.filter((book) => {
let articleInfo = data.articles.filter((article) => {
return book.productSku === article.content.productsku;
});
book.articleName = articleInfo[0]?.name;
book.articleTeaser = articleInfo[0]?.content.teaser;
return book;
});

result

References