Enrich and Normalize Data
Clean Up Data Using Find/Replace And Other Bulk Operations
Oftentimes it is necessary to migrate data from one location to another. As part of this exercise, cleaning up the data is undertaken in order to create a “clean slate”. Conscia supports this by providing a toolbox of bulk operations that can be run on the data to perform various tasks such as find / replace, mass apply and type conversion.
.
To access the data operations:
- Navigate to the source collection and field of interest
- While hovering over the field header, click on the arrow to expose the inline menu
- From the menu, select TRANSFORM and the target operation.
Note that each field will display a different set of transformations (only those transformations applicable to the current data type are shown).
Data Normalization with Bulk-Apply Data Transformation Features
When you are pulling data from multiple data sources and/or is being managed by siloed departments and teams, it is common to have inconsistencies.
The following transformations are available to standardize and normalize your data through Control Center and can also be operationalized as data preparation recipes that can be applied on a schedule.
| Enrichment Feature | Description |
|---|---|
| Change Case | Changes the sentence case of values in the selected column and data set. |
| Concatenate Fields | Combines two columns into a new column with values separated by a user-specified delimiter. |
| Convert Array to String | Converts the values in an array to a delimited string. |
| Convert Date Field Format | Converts date values in a selected field to an alternative date format. |
| Convert Field to Number | Changes the type of a field by converting it from a text to number |
| Convert Field to Text | Converts the type of a field to text |
| Remove Non-printable Characters | Removes any non-printable characters found within the values in a field |
| Trim Whitespace | Removes whitespaces in values of the selection column |
| Find and Replace (Single) | Find a specified pattern in the data set and replaces it with the user defined value or pattern |
| Find and Replace (Multiple) | Allows a user to reference a collection containing a series of find and replace values. When a value is found in the target column, it is replaced with the corresponding value from the lookup table. |
| Mass Apply (Single) | Overwrites a field value with a user defined target value |
| Mass Apply (Multiple) | Overwrites values in the target column. The user can define either to overwrite the target column with the mass applied values, or append the values to the end. |
| Mass Remove | Removes a value from a selected field |
| Group Children Records | Aggregates child records under a parent record |
| Extract Pattern | Takes in a regular expression (regex) pattern and outputs a new column containing text which matches the user defined pattern. Each match is separated by a pipe (|) in the new column. |
| Extract Unique Values | Extracts all unique values stored within a field. Output is stored as a separate collection. |
| Spell Check | Checks the spellings of words in a string column and suggest correct spellings. |
| Split Field (Single) | Splits data into separate columns given a character. Contents of this column will be split on the given character and stored in a new single array column |
| Split Field (Multiple) | Splits a value within the target field into multiple fields. The number of generated fields is based on the number of delimiters found. |