Skip to main content

How DX Engine Works

The Digital Experience Orchestration Engine (DX Engine) processes every request received from the frontend (or any other client) via the Experience API and responds to it by orchestrating various backend APIs based on business rules registered with the embedded rules engine. The DX Engine performs the logic required by breaking the work down into Components, which can be chained or processed in parallel depending on the orchestration needs of the application.

Conscia's DX Engine is agnostic of the both the backend and the frontend. For this reason, we offer a zero-code orchestration to our partners and customers.

alt_text

The Anatomy of the DX Engine​

Connections​

You can define a catalog of connections for each application. A Connection is configured with information about the API endpoint to connect to such as a headless CMS, Commerce Engine, Promotions Engine etc. This includes credentials, API tokens, etc. Conscia offers a library of pre-defined connections to various popular backend applications for convenience, however, you can easily connect to any backend API using our 'Universal Connector'.

Universal API Connector​

Conscia offers a Universal API Connector which allows for reading from (or writing to) any webservice-enabled datasource. This allows organizations to personalize/orchestrate with data sources that unique to them and/or are not already offered as a pre-built Connector.

TODO - Diagram

Components​

A Component is an atomic processing unit that processes data and business logic based on Component configuration. Here are the main features of Component.

A Component:

  • may or may not use a Connection
  • can retrieve data directly from the backend system via the Connection or from other Components
  • can be passed a real-time Context, which may be received from the client application or set by one of the other Components
  • may or may not process business rules to drive aspects of the business logic based on the Context passed to it
  • can cache it's response to avoid unnecessarily calling backend APIs or process information that has already been processed
  • can manipulate and transform data received from various inputs
  • can be chained with other Components

Context​

Business Rules processed by the DX Engine are triggered on the real-time Context of the current query being processed.

The Context can be set in one of the following ways:

  • Passed to the DX Engine in the API Request. For example, the query can set the location, device, CustomerSegment, Current Navigation State, Customer ID, and anything else that is relevant to the application
  • Set by any of the Components through a process called 'Context Enrichment'. For example, a Component that is responsible for retrieving the Customer Profile by Customer ID from a CDP may extract the Customer Segment of the Customer and set it as Context that any other Component can react to. Each Component has the ability to enrich the Context and each Component can also react to the Context set by any other Component.

Business Rules​

Each Component can behave differently depending on the real-time Context provided to it based on the Business Rules configured for that Component. These business rules can be configured via the Experience Studio by business teams or via APIs by developers.

Each Business Rule has the following elements:

  • Trigger - The Conditions that must be present for this rule to fire. This includes Active Start and End date, Priority, as well as the Context

  • Target - The data that the Component will return when the condition is met. This data could be selected from a backend system by the business users or via APIs. Experience Studio offers an intuitive UI for business users to select content from source systems such as CMSs, DAMs and Commerce Engines.

This functionality forms the basis of Personalization in a Composable tech stack.

Caching​

Every Component has the option to cache its output for a configurable amount of time. Since the Components can be assembled in any way, the DX Engine allows for any Component to configure how its output is cached.

These controls include:

  • Time-to-live of a cached response
  • What the cache key should be

The actual work performed by the DX Engine itself contributes almost negligibly to the response time. Any latency is due to IO to/from the various backend systems that are being orchestrated.

We have benchmarked “extreme” orchestration scenarios by having 20+ of Components making mock calls (i.e. zero response latency) and evaluating various conditional logic and transformations under load and seen a response time of less than 10ms.

Caching Deep-Dive​

By default the cache key is the full Component-specific configuration of a Component.

The parameters of Component configuration are defined at the time the Component is created. There are some Component-specific configurations and some general configurations as described earlier.

These configurations are received at execution time from one of the following:

  • Context (that is passed in from the request or enriched by one of the upstream Components)
  • Business Rules
  • Another upstream Component

Let’s walk through a scenario. Let’s say that you are looking to display ‘relevant’ videos residing in Cloudinary to a customer based on their customer segment. The Cache Key will be the list of resourceIDs + transformations + any general Component config such as connection code.

The list of configurations for the Cloudinary Static Video List Components are defined here: https://docs.conscia.ai/reference/Components/cloudinary#cloudinary-static-video-list

alt_text

The resourceIDs configuration will be produced in one of the following ways - these are depicted in the diagram above.

1 - The business user will set rules based on which the videos will be selected for specific customer segments. 2 - The Recommendations engine will provide a list of recommended videos 3 - The list of relevant videos is passed to the engine within the Context itself (perhaps previously seen videos are available in the browser cookie?)

For (1) above, you only need the ‘Static Video List’ Component for Cloudinary. The business user will select the list of records and what transformations to apply to the video for each trigger condition (which in this case is the customer segment). The configuration that the cloudinary video list Component requires is simply the list of resourceIDs, which were received from the business rules associated with the Component.

For (2), a Recommendations Component and the Static Video List Component would be required. The Recommendations Component will return the list of resourceIDs and the Static Record List will use that as configuration to grab these video assets from Cloudinary.

For (3), the list of video IDs is passed in the Context by the calling application and the Static Video List Component will simply retrieve this list from the Context.

Chaining Components and the Dependency Graph​

Components are run in parallel wherever possible, but there are times when the output from one Component is required before another Component can be executed.

Digital Experiences often require an orchestration of various systems and applications. For example, consider a scenario where your customer profile is sitting in a Customer Data Platform (CDP) such as Segment. Segment holds on to customer traits such as 'brand affinity', 'product affinity', 'churn propensity', etc. When the customer comes to your website, you want to offer them a personalized experience.

Let’s say you want to offer personalized promotions (managed in Talone.One) and content (managed in Contentstack) to a customer based on their current cart contents as well as their historical purchases, customer segment and other profile attributes that live in a CDP (or a customer database). As for promotions, you also want to aggregate all of the promotions that the customer qualifies for and present to them on the front-end. This requires calling multiple systems in a very specific sequence i.e there are dependencies between API calls. Output from one call needs to be presented as an input to another API call. The following diagram demonstrates the sequence of API calls that must be orchestrated.

alt-text

Performance Considerations​

The DX Engine is responsible for orchestrating many API calls for various backends. Some of these calls are dependent on each other i.e they need a response from another API before they can be called. Others calls can be made in parallel.

The DX Engine creates a dependency graph of all the APIs based on the logic configured within your API flow. This ensures that the performance is optimized for your overall API query.

Secondly, each Component that makes a request to a backend can be provisioned with its own independent caching strategy. If you are making calls to slower backends that don't change very often, you can configure a longer cache invalidation period. You can also configure your caching strategy to reduce the load on any backend system to reduce the overall cost of your API calls.

The combination of caching and optimization of calling sequence together ensures the best possible performance.

Data Manipulation and Transformation​

Components can manipulate and transform data from upstream Components, or the data sources that they are connected to. There are several use cases that can be addressed by this capability.

Frontend Schema Mapping​

The DX Engine allows you to restructure and/or merge the data from your backend sources into a structure that your applications can more easily consume. For example, in order to truly decouple the frontend from the backend, you can use the DX Engine to translate the schema from the data sources to the schema required by the frontend. This greatly simplifies the frontennd development as you no longer need to write complex logic to stitch data from multiple APIs.

This involves identifying the data elements in one or more source systems and specifying how they should be transformed and mapped to the corresponding data elements in the target system(s).

Conscia provides the following Data Mapping Components which allow you to specify the target structure and how the target structure's elements are populated by the various sources:

Component Response Transformation​

A Component itself can also transform its own output to make it easier for other downstream components to consume it via JavaScript Expression.

Context Enrichment​

Sometimes, a Component may require certain information extracted from the output of an upstream component or a data source. In our previous example, we mentioned the Customer Profile Component that extracted the customer data from a CDP so that a downstream Component can use it to pull the right content from a CMS. In this case, the Component has the ability to enrich the 'Context' of the query with a new field called 'Customer Segment'. This, once again, is done by using Javascript Expressions.

A Single Experience API for Any Frontend​

The rise of Composable Commerce means that the number of best-of-breed applications that organizations have to manage and build experiences against has and will continue to rise. One challenge that naturally emerges from this revolution is that developers must integrate the frontend with each of the vendor APIs.

Conscia simplifies the presentation layer with a single unified API response so that the front-end can simply interface with the DX Engine, which in turn takes care of connecting to and assembling content and data from multiple backend systems. The API takes in a request and returns a JSON response with the data and content required to render any page/screen/display whatsoever.