Merge Two Sources into a Single Data Model

In the following recipe, we will merge content items from the CMS with product records from a commerce engine in order to populate product pages, whether a Product Detail Page (PDP), Product Listing Page (PLP), landing page, or anywhere products are featured with extended information. This is a typical use case in a headless commerce implementation -- needing to match up additional data with product data, be it pricing or inventory or product care instructions. Frequently developers write glue code in the frontend to address this use case and end up making multiple API calls and stitching together data from the backends. An orchestration layer can configure this data unification.

Overall Orchestration Flow

We will be using Storyblok as our headless CMS and commercetools as our commerce system. All commercetools products have a SKU, and any product related content items in Storyblok have a SKU field where business users can relate the records.

We will create two Orchestration Components to connect to each data source, and a third to match the records up by product SKU.

For a simpler mapping example that does not require matching up a series of objects by some value, see the Stitching Data from Multiple Sources recipe.

For a breakdown on different methods to merge and transform responses, see the Blending Responses tutorial.

Mapping Out DX Engine Elements

• Get Book Products: This Component connects to commercetools and contains a webservice call to retrieve all products of productType = Book.
• Get Author Articles: This Component connects to Storyblok and contains a webservice call to retrieve all stories in a folder called "Product-Related Articles".
• Merge Book Product Data: This is a Data Transformation Script Component. It takes the responses from each of the components above, matches the records based on SKU, and maps the required properties for a final response to be consumed by the front end. In this example we're doing the equivalent of a left join, where all Book Products will be returned and if there is associated Storyblok data, it will be included. This Component type is entirely scripted, unlike the Object and Property Mapper Components which use a UI to do the majority of the mapping and transforming.

The end result will look something like this:

{
  "response": [
    {
      "productId": "13ea40eb-1b9a-438d-8824-6347f96862e0",
      "productName": {
        "en-US": "The Hobbit"
      },
      "productSku": "hobbit-123",
      "productDescription": {
        "en-US": "Best selling fantasy fiction"
      },
      "productSlug": {
        "en-US": "the-hobbit"
      },
      "articleName": "J.R.R. Tolkien: A Literary Legend and Father of Modern Fantasy",
      "articleTeaser": "Explore the captivating life and extraordinary imagination of J.R.R. Tolkien, the brilliant author behind the beloved Middle-earth tales and the creator of a genre that continues to inspire generations of readers and writers alike."
    },
    {
      "productId": "c836f0a8-25e1-4066-a2a0-29cbccbd6f05",
      "productName": {
        "en-US": "The Greatest Showman"
      },
      "productSku": "showman",
      "productDescription": {
        "en-US": "film adaptation"
      },
      "productSlug": {
        "en-US": "the-greatest-showman"
      }
    },
    {
      "productId": "74555781-cf33-42c8-b8ec-aeeea5ee3230",
      "productName": {
        "en-US": "The Best Cookbook"
      },
      "productSku": "mealprep",
      "productDescription": {
        "en-US": "Student guide for quick eating"
      },
      "productSlug": {
        "en-US": "the-best-cookbook"
      }
    },
    {
      "productId": "74e4c00b-3421-42fc-ab79-8df82dcb7160",
      "productName": {
        "en-US": "Mastering The Art Of French Cooking"
      },
      "productSku": "mastering-french-cooking-123",
      "productDescription": {
        "en-US": "The definitive cookbook on French cuisine for American readers"
      },
      "productSlug": {
        "en-US": "mastering-the-art-of-french-cooking"
      },
      "articleName": "Julia Child: A Culinary TV Star",
      "articleTeaser": "Explore the enduring legacy of Julia Child, the charismatic chef who revolutionized television with her approachable yet sophisticated cooking style. From her iconic French recipes to her captivating on-screen presence, discover how Julia Child became a beloved culinary icon."
    }
  ]
}

DX Engine Configuration Details

Create a Connection to commercetools

• Navigate to the Connections page (Settings --> Connections).
• Click the Configure Connection button.
• Enter the following and click Submit:

Field	Value
Connection Code	commercetools-connection
Connection Name	commercetools Connection
Connector	Universal API Connector
Base URL	Get value from: Literal https://api.{region}.gcp.commercetools.com/{channel}
Base Headers	Header: Authorization Value: Bearer {token}

Create a Connection to Storyblok

• Navigate to the Connections page (Settings --> Connections).
• Click the Configure Connection button.
• Enter the following and click Submit:

Field	Value
Connection Code	storyblok-connection
Connection Name	Storyblok Connection
Connector	Universal API Connector
Base URL	Get value from: Literal https://api.storyblok.com/v2/cdn/stories
Query Parameters	Parameter: token Value: {token}

Create a Component to retrieve products from commercetools

• Navigate to the Experience Components page (Manage Experiences --> Components)
• Click the + Add Component button.
• Enter the following and click Submit:

Field	Value
Component Code	commercetools-get-products-books
Component Name	Get Book Products - commercetools
No Rules	Checked
Component Type	Conscia - Universal API Connector

The DX Engine will create and prepare the Component for further configuration. You should see the new component appear in the Component listing.

• Click Edit.
• Enter the following and click Submit:

Field	Form Tab	Value
Connection	Main	commercetools Connection
Webservice Path	Main	Get value from: Literal /product-projections/search
Method	Main	GET
Query Parameters	Main	Parameter: filter Value (Literal): productType.id:"{id}"

Create a Component to retrieve articles from Storyblok

• Navigate to the Experience Components page (Manage Experiences --> Components)
• Click the + Add Component button.
• Enter the following and click Submit:

Field	Value
Component Code	storyblok-get-author-articles
Component Name	Get Author Articles - Storyblok
No Rules	Checked
Component Type	Conscia - Universal API Connector

The DX Engine will create and prepare the Component for further configuration. You should see the new component appear in the Component listing.

• Click Edit.
• Enter the following and click Submit:

Field	Form Tab	Value
Connection	Main	Storyblok Connection
Method	Main	GET
Headers	Main	Header: Accept Value (Literal): application/json
Headers	Main	Header: Content-Type Value (Literal): application/json
Query Parameters	Main	Parameter: starts_with Value (Literal): product-related

Create a Component to merge product and article data

• Navigate to the Experience Components page (Manage Experiences --> Components)
• Click the + Add Component button.
• Enter the following and click Submit:

Field	Value
Component Code	merge-book-product-data
Component Name	Merge Book Product Data
No Rules	Checked
Component Type	Conscia - Data Transformation Script

The DX Engine will create and prepare the Component for further configuration. You should see the new component appear in the Component listing.

• Click Edit.
• Enter the following and click Submit:

Field	Form Tab	Value
Connection	Main	Storyblok Connection
Configuration - Data to Modify	Main	Get value from(JS Expression): _.assign({},{ "books" : componentResponse('commercetools-get-products-books').results, "articles" : componentResponse('storyblok-get-author-articles').stories }) Script:

let bookProducts = data.books.map((book) => ({
  productId: book.id,
  productName: book.name,
  productSku: book.masterVariant.sku,
  productDescription: book.description,
  productSlug: book.slug
}));

let result = bookProducts.filter((book) => {
  let articleInfo = data.articles.filter((article) => {
    return book.productSku === article.content.productsku;
  });
  book.articleName = articleInfo[0]?.name;
  book.articleTeaser = articleInfo[0]?.content.teaser;
  return book;
});

result

References

• Stitching Data from Multiple Sources
• Blending Responses
• Data Transformation Scripts