Exporting Collections
After data has been imported into a Collection and enriched/transformed/etc., it can be exported to a file in a Data Bucket. The file can be in line-delimited JSON or delimited format. This mechanism is useful for exporting data to be consumed by another system such as a search engine, database, data warehouse, etc..
The DX Graph provides the Job Type, Export Collection, to export data from a Collection into a file in a Data Bucket.
Job Types
Export Collection
The Export Collection job exports every record from a Collection that matches a specified filter (if provided) into a file in a Data Bucket. Any transformation or schema errors are uploaded to an errors file in the same Data Bucket. More details on error files are here. Exported files are in line-delimited JSON format.
Job Parameters
| Parameter | Description |
|---|---|
| Collection Code | The Collection to export from. |
| Transformers | A list of transformations applied to each source record in the collection. |
| Target Data Bucket | The DX Graph Bucket that the Data File will be exported to. |
| Target Filename | The name of the file that the data will be written to. It has the placeholder, {{timestamp}}. {{timestamp}} is the timestamp of the export request. Example: products_{{timestamp}}.jsonl would result in files that look like: products_20230514_131001.jsonl. |
| filter | A DX Graph filter that will be applied to the source records. If the filter is not provided, then all records will be exported. |
| Record Layout Configuration | If this is specified, the export of the records are in the Expanded Record Format. This defines what fields and relationships to return. You can see details here. If not specified, the records are exported as-is. |
| Target Schema | JSON schema that is applied to the transformed records. |
Note: When specifying any transformers, you must keep in mind that the source record is in the Expanded Record Format. The transformers have access to Expanded Record Format functions