Skip to main content

Managing Data Files in Data Buckets

Data files can be uploaded to and downloaded from a Data Bucket using either the DX Graph API or the DX Graph UI. This is useful for uploading data files to be imported into a Collection or downloading data files that have been exported from a Collection.

Managing Files through the UI​

Files can be uploaded to and downloaded from a Data Bucket using the DX Graph UI. This is done on the Operations -> Files page.

Files and Buckets

Along the left hand side you will see a listing of Data Buckets that have been created for this environment.

Download and Remove Files​

When you click on a Bucket, a listing of files currently in that Bucket will populate in the center Files section. These can be removed by clicking the - button or downloaded by clicking the Download button.

Upload Files​

You can manually add files into a Data Bucket on this page. Click on the appropriate Bucket, then either drag and drop your file(s) or click the Browse link to upload your file(s) into the Bucket.

Managing Files through the API​

Listing Files in a Data Bucket​

To list Files in a Data Bucket, you may use the limit and pattern query parameters to define how many files you want to retrieve and what pattern the file name should match.

The following lists the first 5 files in the store-sales Data Bucket that match the pattern store*.*.

GET {{engineUrl}}/vue/_api/v1/buckets/store-sales/files?limit=5&pattern=store*.*
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}

Deleting Files in a Data Bucket​

You can delete one or multiple files at a time (to a maximum of 1000) using the DELETE endpoint. Use the pattern query parameter to define the pattern the file name should match.

It is good practice to clean up your buckets periodically unless you have contacted us about a predefined retention policy.

The following removes all files in the store-sales Data Bucket that match the pattern store*.*.

DELETE {{engineUrl}}/vue/_api/v1/buckets/store-sales/files?pattern=store*.*
Authorization: Bearer {{apiKey}}
X-Customer-Code: {{customerCode}}

Uploading Files​

You can upload one or more files at a time to a Bucket using the upload endpoint. The following uploads two files to the store-sales bucket.

curl --request POST \
  --url '{{engineUrl}}/vue/_api/v1/buckets/store-sales/upload' \
  --header 'Authorization: Bearer {{apiKey}}' \
  --header 'X-Customer-Code: {{customerCode}}' \
  --form 'file[]=@store-sales-01-2023.csv.gz' \
  --form 'file[]=@store-sales-02-2023.csv.gz'

Downloading Files​

When fetching a file from a Data Bucket, the filename query parameter is required. The filename is the name of the file that is contained in the Data Bucket. The following request streams the file to the client so it is important pipe the output to a file.

curl --request GET \
  --url '{{engineUrl}}/vue/_api/v1/buckets/store-sales/download?filename=store-sales-01-2023.csv.gz' \
  --header 'Accept: */*' \
  --header 'Authorization: Bearer {{apiKey}}' \
  --header 'User-Agent: httpyac' \
  --header 'X-Customer-Code: {{customerCode}}' > store-sales-01-2023.csv.gz