データ マネージャーの要件と制限は何ですか?

Although Data Manager is built to handle very large data sets, it does have a maximum load. Each limit may have workarounds that involve changes to your local system or how your data is structured.

Minimum disk space

Calculate minimum disk space as:

(total size of local files) × (number of days of storage)

The Cleanse and Import client runs on your local system. Because it is middleware running locally, capacity depends on your system hardware and configuration.

If you load large files or add data frequently, ensure enough disk space is available. To save space, consider removing files processed more than 14 days ago.

If you use tools or databases without files, disk requirements may be lower.

例

If you have 10 CSV files under 100 MB, you need:

100 MB × 14 days = 1.5 GB

of disk space (in addition to OS requirements).

Minimum memory

The Cleanse and Import client also runs locally. If you load large files or add data frequently, ensure you have enough RAM.

Memory requirements may be reduced if you only use files (such as CSV) and not tools or databases. Some adapters load all data into memory; monitor the first load and adjust memory if needed.

例

• Flat files with 75,000 objects – at least 8 GB RAM recommended.

• SQL or APIs with 175,000 objects – at least 8 GB RAM recommended.

Data Manager sizing guide

How long will a merge take?

Data Manager runs in the cloud. The merge size and duration depend on:

• number of attributes in the merged data

• number of data sources

• total number of records (sum of records across data sources)

To speed up the merge:

• Reduce attributes – assign unneeded attributes to <ignore> in attribute mapping.

• Add cleansing rules – remove duplicate or erroneous records before merging.

• Reduce data sources – remove sources that are not required.

What is the recommended scale for object count in an object class?

While Assets Data Manager does not enforce a hard limit on the number of objects in a golden dataset, our performance and scale testing has validated the following configuration for optimal stability and performance:

• Golden Objects: Up to 2,000,000

• Attributes per Object: 80

• Data Sources: 5

If your dataset exceeds these parameters, you may experience longer processing times or potential import failures. We recommend reviewing your cleansing rules and mapping only essential attributes to maintain high performance.

Maximum number of Important attributes

Important attributes are compared across data sources to ensure they are correctly and consistently represented. You can mark any attribute as Important in the Attributes screen.

When an Important attribute is merged, it is compared against data sources. If values match across sources, the attribute is flagged as Verified.

Maximum number of tags

Tags are customisable, searchable metadata attached to Data Manager objects. Across all active object classes, you can use up to 56 tags.