Merge ERP Invoices in DX Engine
In the enterprise space, growth by merger is a common and successful strategy. However for the IT-facing teams, this kind of marriage can introduce duplicate systems - multiples of tax, content management, or search platforms servicing the frontend, but also multiple WMS, OMS and ERP systems in the backend. Customers could think they're placing one order, but multiple business units are receiving their own piece of the final delivery.
This recipe uses DX Engine to query a pair of disparate ERP systems (as mocked up in DX Graph) that have distinct data structures, and showcases how DX Graph can pair up overlapping records between them and deliver a single cohesive invoice to the end user.
Mapping Out DX Engine Elements
When a frontend calls this Recipe through Conscia's Experience API, it will either provide an OrderID context and only retrieve that Order's constituent pieces; or it can provide nothing, and receive all orders in the ERP. In production implementations, there would of course be a variety of more nuanced filtering options available, additional work performed with the unified payload, et cetera.
POST {{engineUrl}}/experience/components/_query
X-Customer-Code: {{customerCode}}
Authorization: Bearer {{dxEngineToken}}
{
"componentCodes": ["prepare-consolidated-invoice"],
"context": {
"orderId": "1001"
}
}
The response would be invoice inputs (and other order details) from one ERP system, or consolidated from both ERP systems:
Experience API Response
{
"duration": 192,
"components": {
"prepare-consolidated-invoice": {
"@extras": {
"rule": {
"metadata": [],
"attributes": {}
},
"metadata": []
},
"status": "VALID",
"response": [
{
"PCNAOrderID": "1001",
"PCNAOrderStatus": [
"PENDING",
"Open"
],
"Customer": {
"CustomerID": [
"CUST001",
"ACM-001"
],
"CustomerName": [
"ACME CORPORATION",
"Acme Corp"
],
"BillingAddress": {
"Street": "123 Business Street",
"City": "New York",
"State": "NY",
"Zip": "10001",
"Country": "USA"
},
"CustomerContact": {
"ContactName": "John Doe",
"ContactPhone": "+1 555 123 4567"
},
"ShippingAddress": {
"Street": "456 Commerce Avenue",
"City": "Los Angeles",
"State": "CA",
"Zip": "90001",
"Country": "US"
}
},
"Invoicing": {
"InvoiceDate": [
1730696400000,
1730433600000
],
"DueDate": [
1733288400000,
1733029200000
],
"Terms": [
"100% net @ 30 days",
"net/30."
],
"SoonestDueDate": "12/1/2024"
},
"Totals": {
"Subtotal": [
1200.5,
1160
],
"Discounts": [
50,
0
],
"Taxes": [
96.04,
80
],
"Total": [
1246.54,
1350
],
"GrandTotal": 2596.54
},
"LineItems": [
{
"Description": "COMPONENT A",
"Quantity": "10",
"PriceEach": "100",
"LineTotal": 1000,
"Metadata": {
"ItemID": "ITM001",
"ReferenceSystem": "ERP A",
"ReferenceOrderID": "INV-12345"
}
},
{
"Description": "INSTALLATION FEE",
"Quantity": "1",
"PriceEach": "200.50",
"LineTotal": 200.5,
"Metadata": {
"ItemID": "ITM002",
"ReferenceSystem": "ERP A",
"ReferenceOrderID": "INV-12345"
}
},
{
"Description": "Support Plan",
"Quantity": 1,
"PriceEach": 240,
"LineTotal": 240,
"Metadata": {
"ItemID": "P002",
"ReferenceSystem": "ERP B",
"ReferenceOrderID": "SAP-98765"
}
},
{
"Description": "Widget Alpha",
"Quantity": 5,
"PriceEach": 200,
"LineTotal": 1000,
"Metadata": {
"ItemID": "P001",
"ReferenceSystem": "ERP B",
"ReferenceOrderID": "SAP-98765"
}
}
],
"Discounts": [
{
"Discount": 50,
"Reason": "Loyalty Discount"
}
]
}
]
}
},
"errors": []
}
DX Graph Configuration
The topics in this section describe how the ERP data was structured, including relationships between entries and example Orders.
Data Model for ERP A
Here are the two Orders in ERP A, along with their constituent elements:
ERP A Orders
Query:
{
"limit": 100,
"filter": {},
"options": {
"recordLayoutConfig": {
"relationships": {
"LINE_ITEMS_ON_ORDER": {
"fieldsToReturn": [
"LINE_ITEM_ID",
"LINE_ITEM_DESCRIPTION",
"LINE_ITEM_QUANTITY",
"LINE_ITEM_UNIT_PRICE"
]
},
"DISCOUNTS_ON_ORDER": {
"fieldsToReturn": [
"DISCOUNT_ID",
"DISCOUNT_TYPE",
"DISCOUNT_AMOUNT"
]
}
}
}
}
}
Response:
{
"data": [
{
"values": {
"@iat": 1732654156662,
"@uat": 1732802762299,
"CUSTOMER_ID": "CUST001",
"CUSTOMER_NAME": "ACME CORPORATION",
"DISCOUNTS_ON_ORDER": [
"1"
],
"LINE_ITEMS_ON_ORDER": [
"ITM001",
"ITM002"
],
"ORDER_BILLING_ADDRESS": {
"ADDRESS_STREET": "123 Business Street",
"ADDRESS_CITY": "New York",
"ADDRESS_STATE": "NY",
"ADDRESS_POSTCODE": "10001",
"ADDRESS_COUNTRY": "USA"
},
"ORDER_DUE_DATE": 1733288400000,
"ORDER_ID": "INV-12345",
"ORDER_INVOICE_DATE": 1730696400000,
"ORDER_INVOICE_TOTAL": 1200.5,
"ORDER_PAYMENT_TERMS": "100% net @ 30 days",
"ORDER_STATUS": "PENDING",
"ORDER_TAX_AMOUNT": 96.04,
"UNIFIED_ORDER_ID": "1001",
"dataRecordIdentifier": "INV-12345"
},
"relationships": {
"LINE_ITEMS_ON_ORDER": {
"entities": [
{
"values": {
"LINE_ITEM_ID": "ITM001",
"LINE_ITEM_DESCRIPTION": "COMPONENT A",
"LINE_ITEM_QUANTITY": "10",
"LINE_ITEM_UNIT_PRICE": "100"
},
"relationships": {}
},
{
"values": {
"LINE_ITEM_ID": "ITM002",
"LINE_ITEM_DESCRIPTION": "INSTALLATION FEE",
"LINE_ITEM_QUANTITY": "1",
"LINE_ITEM_UNIT_PRICE": "200.50"
},
"relationships": {}
}
]
},
"DISCOUNTS_ON_ORDER": {
"entities": [
{
"values": {
"DISCOUNT_ID": "1",
"DISCOUNT_TYPE": "Loyalty Discount",
"DISCOUNT_AMOUNT": 50
},
"relationships": {}
}
]
}
}
},
{
"values": {
"@iat": 1733168526850,
"@uat": 1733168624934,
"CUSTOMER_ID": "CUST230",
"CUSTOMER_NAME": "OASIS ENTERTAINMENT",
"LINE_ITEMS_ON_ORDER": [
"ITA650",
"ITA651"
],
"ORDER_BILLING_ADDRESS": {
"ADDRESS_STREET": "1100-16 Business LN",
"ADDRESS_CITY": "New York",
"ADDRESS_STATE": "NY",
"ADDRESS_POSTCODE": "10001",
"ADDRESS_COUNTRY": "USA"
},
"ORDER_DUE_DATE": 1730347200000,
"ORDER_ID": "INV-12330",
"ORDER_INVOICE_DATE": 1728446400000,
"ORDER_INVOICE_TOTAL": 10000,
"ORDER_PAYMENT_TERMS": "100% net @ 30 days",
"ORDER_STATUS": "CLOSED",
"ORDER_TAX_AMOUNT": 130.3,
"UNIFIED_ORDER_ID": "1000",
"dataRecordIdentifier": "INV-12330"
},
"relationships": {
"LINE_ITEMS_ON_ORDER": {
"entities": [
{
"values": {
"LINE_ITEM_ID": "ITA650",
"LINE_ITEM_DESCRIPTION": "SELF SEALING STEM BOLTS",
"LINE_ITEM_QUANTITY": "1000",
"LINE_ITEM_UNIT_PRICE": "1"
},
"relationships": {}
},
{
"values": {
"LINE_ITEM_ID": "ITA651",
"LINE_ITEM_DESCRIPTION": "STEM BOLT BUCKET",
"LINE_ITEM_QUANTITY": "1",
"LINE_ITEM_UNIT_PRICE": "9000"
},
"relationships": {}
}
]
},
"DISCOUNTS_ON_ORDER": {
"entities": []
}
}
}
]
}
Data Model for ERP B
Here are the two Orders in ERP B, along with their constituent elements:
ERP B Orders
Query:
{
"limit": 100,
"filter": {},
"options": {
"recordLayoutConfig": {
"relationships": {
"item_details_for_invoice": {
"fieldsToReturn": [
"item_detail_id",
"product_code",
"name",
"quantity",
"price_per_unit",
"total_price"
]
}
}
}
}
}
Response:
{
"data": [
{
"values": {
"@iat": 1733168748223,
"@uat": 1733168841997,
"additional_fees": [
{
"fee_name": "FAST HANDS TAX",
"fee_amount": 5.3
}
],
"amount_due": 960.6,
"currency_code": "USD",
"customer": {
"name": "Oasis Entertainment",
"contact": "Nathan Oasis",
"phone": "+1 555 122 3345"
},
"customer_code": "APGTO",
"dataRecordIdentifier": "SAP-98770",
"due_date": 1725336000000,
"invoice_date": 1725336000000,
"invoice_state": "RESOLVED",
"item_details_for_invoice": [
"3"
],
"net_amount": 1021.2,
"order_id": "SAP-98770",
"payment_conditions": "Cash on delivery",
"unified_order_id": "999",
"shipping_address": {
"street_address": "220-404 Grant Street",
"city_name": "Boston",
"region": "MA",
"postal_code": "02134",
"country_code": "US"
},
"vat": 60.6
},
"relationships": {
"item_details_for_invoice": {
"entities": [
{
"values": {
"item_detail_id": "3",
"product_code": "P650",
"name": "PREMIUM BEYBLADES",
"quantity": 12,
"price_per_unit": 80.05,
"total_price": 960.6
},
"relationships": {}
}
]
}
}
},
{
"values": {
"@iat": 1732653367869,
"@uat": 1733258862590,
"additional_fees": [],
"amount_due": 1350,
"currency_code": "USD",
"customer": {
"name": "Acme Corp",
"contact": "John Doe",
"phone": "+1 555 123 4567"
},
"customer_code": "ACM-001",
"dataRecordIdentifier": "SAP-98765",
"due_date": 1733029200000,
"invoice_date": 1730433600000,
"invoice_state": "Open",
"item_details_for_invoice": [
"1",
"2"
],
"net_amount": 1160,
"order_id": "SAP-98765",
"payment_conditions": "net/30.",
"unified_order_id": "1001",
"shipping_address": {
"street_address": "456 Commerce Avenue",
"city_name": "Los Angeles",
"region": "CA",
"postal_code": "90001",
"country_code": "US"
},
"vat": 80
},
"relationships": {
"item_details_for_invoice": {
"entities": [
{
"values": {
"item_detail_id": "2",
"product_code": "P002",
"name": "Support Plan",
"quantity": 1,
"price_per_unit": 240,
"total_price": 240
},
"relationships": {}
},
{
"values": {
"item_detail_id": "1",
"product_code": "P001",
"name": "Widget Alpha",
"quantity": 5,
"price_per_unit": 200,
"total_price": 1000
},
"relationships": {}
}
]
}
}
}
]
}
We will execute the remainder of the recipe in DX Engine.
DX Engine Configuration Details
The topics in this section explain how to implement the Components that execute this recipe.
Secrets
DX Graph Secret
To create a Secret used to store the DX Graph Offline Token in the DX Engine UI:
- Navigate to the Secrets page (Settings --> Secrets).
- Click the + Add Secret button.
- Enter the following and click Submit:
Field | Value |
---|---|
Secret Code | dxgraph |
Secret Name | DX Graph |
Description | Specify the customer code this will connect to |
Secret Value | Enter your DX Graph Offline Token, generated via the Conscia Postman Collection. |
Connections
Connection to DX Graph
- Navigate to the Connections page (Settings --> Connections).
- Click the + Add Connection button.
- Enter the following and click Submit:
Field | Value |
---|---|
Connection Code | dx-graph |
Connection Name | DX Graph Connection |
Connector | DX Graph |
Conscia Endpoint | Staging |
API Key | **Get value from:**Secret DX Graph |
Customer Code | the relevant customer code |
Components
Component to fetch ERP A Orders
- Navigate to the Experience Components page (Manage Experiences --> Components).
- Click the + Add Component button.
- Enter the following and click Submit.
Field | Form Tab | Value |
---|---|---|
Component Code | Main | get-erp-a-orders |
Component Name | Main | ERP A Orders |
No Rules | Main | Checked |
Component Type | Main | Dx Graph - Dynamic Record List |
Connection | Main | Get value from: Literal DX Graph |
Collection ID | Main | erp-a-orders |
Query Filter | Main | Get value from: JS ExpressionJSON.stringify( {"$and": [ contextField('orderId') ? {"$eq": {"field": "UNIFIED_ORDER_ID", "value": contextField('orderId')}} : {} ]} ) |
Record Layout Configuration | Main | Get value from: JS Expression (below) |
({
"relationships": {
"LINE_ITEMS_ON_ORDER": {
"fieldsToReturn": ["LINE_ITEM_ID", "LINE_ITEM_DESCRIPTION", "LINE_ITEM_QUANTITY", "LINE_ITEM_UNIT_PRICE"]
},
"DISCOUNTS_ON_ORDER": {
"fieldsToReturn": ["DISCOUNT_ID", "DISCOUNT_TYPE", "DISCOUNT_AMOUNT"]
}
}
})
Component to fetch ERP B Orders
- Navigate to the Experience Components page (Manage Experiences --> Components).
- Click the + Add Component button.
- Enter the following and click Submit.
Field | Form Tab | Value |
---|---|---|
Component Code | Main | get-erp-b-orders |
Component Name | Main | ERP B Orders |
No Rules | Main | Checked |
Component Type | Main | Dx Graph - Dynamic Record List |
Connection | Main | Get value from: Literal DX Graph |
Collection ID | Main | erp-b-orders |
Query Filter | Main | Get value from: JS ExpressionJSON.stringify( {"$and": [ contextField('orderId') ? {"$eq": {"field": "unified_order_id", "value": contextField('orderId')}} : {} ]} ) |
Record Layout Configuration | Main | Get value from: JS Expression (below) |
({
"relationships": {
"item_details_for_invoice": {
"fieldsToReturn": ["item_detail_id", "product_code", "name", "quantity", "price_per_unit", "total_price"]
}
}
})
Component to Make ERP A Orders into Generic Orders
In production, a more comprehensive transform would be executed following the process introduced below. We demonstrate the conversion to generic Order with an Object Mapper (here) and a Property Mapper (below).
- Navigate to the Experience Components page (Manage Experiences --> Components).
- Click the + Add Component button.
- Enter the following and click Submit.
Component to Make ERP B Orders into Generic Orders
In production, a more comprehensive transform would be executed following the process introduced below. We demonstrate the conversion to generic Order with an Object Mapper (above) and a Property Mapper (here).
- Navigate to the Experience Components page (Manage Experiences --> Components).
- Click the + Add Component button.
- Enter the following and click Submit.
Field | Value |
---|---|
Component Code | erpb-generic |
Component Name | ERP B Order to Generic Order |
No Rules | Checked |
Component Type | Conscia - Property Mapper |
Default source | Get value from: Context Field Order |
Default Expression Type | JavaScript |
Property Map: UnifiedOrderID | data.values.unified_order_id |
Property Map: UnifiedOrderStatus | data.values.invoice_state |
Property Map: ReferenceOrderID | data.values.order_id |
Property Map: Customer | ({ "CustomerID": data.values.customer_code, "CustomerName": data.values.customer.name, "CustomerContact": { "ContactName": data.values.customer.contact, "ContactPhone": data.values.customer.phone }, "ShippingAddress": { "Street": data.values.shipping_address.street_address, "City": data.values.shipping_address.city_name, "State": data.values.shipping_address.region, "Zip": data.values.shipping_address.postal_code, "Country": data.values.shipping_address.country_code, } }) |
Property Map: Invoicing | ({ "InvoiceDate": data.values.invoice_date, "DueDate": data.values.due_date, "Terms": data.values.payment_conditions }) |
Property Map: Totals | ({ "Subtotal": data.values.net_amount, "Discounts": 0, "Taxes": data.values.vat, "Total": data.values.amount_due, }) |
Property Map: LineItems | _.map(data.relationships.item_details_for_invoice.entities, (item) => ({ "Description": item.values.name, "Quantity": item.values.quantity, "PriceEach": item.values.price_per_unit, "LineTotal": item.values.total_price, "Metadata": { "ItemID": item.values.product_code } })) |
Response Transform | response = { ...response, LineItems: _.map(response.LineItems, (item) => ({ ...item, Metadata: { ...item.Metadata, ReferenceOrderID: response.ReferenceOrderID } })) } |
Component to Prepare a Consolidated Invoice
The genericized orders are overlaid for presentation in this Data Mapper Component. In brief, this Component puts each Order from each ERP into a map, filing them by UnifiedOrderID. The first Order to arrive is inserted as-is. For subsequent Orders with the same UnifiedOrderID, we walk the Order Object and, for each field,
- if the fields' values are identical, we leave the old value as-is.
- if the field is a single string/number/etc., we replace the old value with an Array containing both the old and new values.
- if the field is an Array, we append the entries from the new Order into the existing array.
- if the field is an Object, we recursively repeat this process within that Object.
There is a demonstration of some straightforward custom logic at the end of the Script.
- A GrandTotal figure is calculated, as the sum of each ERP's Totals.
- We delete the ReferenceOrderIDs from the Invoice, as this is an internal field (present in the per-line-item Metadata).
- We present the SoonestDueDate, which is the soonest Invoicing DueDate displayed in a human-readable output.
In a production scenario, additional enterprise-specific logic could be inserted to manage the operations here, look up qualities against tables or dictionaries, et cetera.
To implement this Component:
- Navigate to the Experience Components page (Manage Experiences --> Components).
- Click the + Add Component button.
- Enter the following and click Submit.
Field | Value |
---|---|
Component Code | prepare-consolidated-invoice |
Component Name | Prepare Consolidated Invoice |
No Rules | Checked |
Component Type | Conscia - Data Transformation Script |
Data to modify | Source data - Get value from: JS Expression({ "ERP_A": _.map(componentResponse 'get-erp-a-orders'), (item) => ( item.genericOrders.erpa-generic.response )), "ERP_B": _.map componentResponse('get-erp-b-orders'), (item) => ( item.genericOrders.erpb-generic.response )) }) |
Script | below |
Data Transformation Script
const map = new Map();
function addToMap(map, arr) {
for (const item of arr) {
if (!map.has(item.PCNAOrderID)) {
map.set(item.PCNAOrderID, {});
}
const mapEntry = map.get(item.PCNAOrderID);
function recursiveMerge(value, mapEntry) {
console.log(JSON.stringify(value));
console.log(JSON.stringify(mapEntry));
Object.keys(value).forEach(key => {
console.log("key: " + key);
console.log("value:" + JSON.stringify(value[key]));
console.log("mapEntry:" + JSON.stringify(mapEntry[key]));
if ((key in mapEntry) & (mapEntry[key] !== value[key])) {
if (Array.isArray(value[key])) {
console.log('array');
mapEntry = mapEntry || [];
mapEntry[key] = mapEntry[key].concat(value[key]);
} else if (typeof value[key] === 'object' && value[key] !== null) {
console.log('object');
var ret = recursiveMerge(value[key], mapEntry[key]);
console.log(JSON.stringify(ret));
mapEntry[key] = ret;
} else {
console.log('concat');
mapEntry[key] = [].concat(mapEntry[key], value[key]);
}
} else {
console.log('instantiate');
mapEntry[key] = value[key];
}
});
return mapEntry;
}
recursiveMerge(item, mapEntry);
}
}
// Add all arrays to the map.
addToMap(map, data["ERP_A"]);
addToMap(map, data["ERP_B"]);
// Implement custom logic here.
map.forEach(entry => {
//Create a GrandTotal field that sums each ERP's Total.
if (Array.isArray(entry.Totals.Total)) {
entry.Totals.GrandTotal = entry.Totals.Total.reduce((sum, num) => sum + num, 0);
} else {
entry.Totals.GrandTotal = entry.Totals.Total;
}
//Do not expose ReferenceOrderID.
delete entry.ReferenceOrderID;
//Display the earliest chronological due date in human-readble format.
entry.Invoicing.SoonestDueDate = new Date(Math.min(...[].concat(entry.Invoicing.DueDate))).toLocaleDateString("en-US");
});
// Convert map values to array. Assigning to a variable returns that variable.
v = Array.from(map.values());
Sub Component Execution
Now, in order to complete the recipe, the ERP A Orders and ERP B Orders Components must call the "ERP to Generic" Components once per Order they have captured.
ERP A
We instantiate the ERP A Order to Generic Order as a Sub Component of ERP A Orders, making it a Parent Component.
- Navigate to the Experience Components page (Manage Experiences --> Components).
- Edit the ERP A Orders Component.
- Navigate to the Sub Components tab.
- Enter the following and click Submit.
Field | Value |
---|---|
Sub Components | Add one entry. |
Property Name | genericOrders |
Component Codes | Add one entry: erpa-generic |
Context Field for Sub Component | Add one entry: Context Field: order Expression: response |
As response
is an array of Orders, the Sub Component will be invoked response.length
times and each execution of erpa-generic
will receive one entry as response
.
ERP B
We instantiate the ERP B Order to Generic Order as a Sub Component of ERP B Orders, making it a Parent Component.
- Navigate to the Experience Components page (Manage Experiences --> Components).
- Edit the ERP B Orders Component.
- Navigate to the Sub Components tab.
- Enter the following and click Submit.
Field | Value |
---|---|
Sub Components | Add one entry. |
Property Name | genericOrders |
Component Codes | Add one entry: erpb-generic |
Context Field for Sub Component | Add one entry: Context Field: order Expression: response |
As response
is an array of Orders, the Sub Component will be invoked response.length
times and each execution of erpb-generic
will receive one entry as response
.
References
Conscia Object Mapper Conscia Property Mapper Data Transformation Script