What are the requirements and limits of Data Manager?
Data Manager is included in all Service Collection Premium and Enterprise plans.
Although Data Manager is built to handle very large data sets, it does have a maximum load. Each limit may have workarounds that involve changes to your local system or how your data is structured.
Minimum disk space
Calculate minimum disk space as:
(total size of local files) × (number of days of storage)
The Cleanse and Import client runs on your local system. Because it is middleware running locally, capacity depends on your system hardware and configuration.
If you load large files or add data frequently, ensure enough disk space is available. To save space, consider removing files processed more than 14 days ago.
If you use tools or databases without files, disk requirements may be lower.
Example
If you have 10 CSV files under 100 MB, you need:
100 MB × 14 days = 1.5 GB
of disk space (in addition to OS requirements).
Minimum memory
The Cleanse and Import client also runs locally. If you load large files or add data frequently, ensure you have enough RAM.
Memory requirements may be reduced if you only use files (such as CSV) and not tools or databases. Some adapters load all data into memory; monitor the first load and adjust memory if needed.
Examples
Flat files with 75,000 objects – at least 8 GB RAM recommended.
SQL or APIs with 175,000 objects – at least 8 GB RAM recommended.
Data Manager sizing guide
Flat Files | |||
Number of compute objects | Number of flat file sources | Memory on local server (Gb) | Additional storage on local server (Gb) |
1,000 | 3 | 8 | 50 |
25,000 | 4 | 8 | 50 |
50,000 | 5 | 8 | 50 |
100,000 | 6 | 8 | 50 |
250,000 | 8 | 16 | 80 |
500,000 | 10 | 16 | 80 |
1,000,000 | 12 | 16 | 100 |
| |||
SQL & ODBC or APIs | |||
Number of compute objects | Number of direct connections to source | Memory on local server (Gb) | Additional storage on local server (Gb) |
1,000 | 3 | 8 | N/A |
25,000 | 4 | 8 | N/A |
50,000 | 5 | 8 | N/A |
100,000 | 6 | 8 | N/A |
250,000 | 8 | 16 | N/A |
500,000 | 10 | 16 | N/A |
1,000,000 | 12 | 16 | N/A |
| |||
How long will a merge take?
Data Manager runs in the cloud. The merge size and duration depend on:
number of attributes in the merged data
number of data sources
total number of records (sum of records across data sources)
To speed up the merge:
Reduce attributes – assign unneeded attributes to
<ignore>in attribute mapping.Add cleansing rules – remove duplicate or erroneous records before merging.
Reduce data sources – remove sources that are not required.
What is the recommended scale for object count in an object class?
While Assets Data Manager does not enforce a hard limit on the number of objects in a golden dataset, our performance and scale testing has validated the following configuration for optimal stability and performance:
Golden Objects: Up to 2,000,000
Attributes per Object: 80
Data Sources: 5
If your dataset exceeds these parameters, you may experience longer processing times or potential import failures. We recommend reviewing your cleansing rules and mapping only essential attributes to maintain high performance.
Maximum number of Important attributes
Each object class can have up to 56 Important attributes.
Important attributes are compared across data sources to ensure they are correctly and consistently represented. You can mark any attribute as Important in the Attributes screen.
When an Important attribute is merged, it is compared against data sources. If values match across sources, the attribute is flagged as Verified.
Maximum number of tags
Tags are customisable, searchable metadata attached to Data Manager objects. Across all active object classes, you can use up to 56 tags.
Was this helpful?