How do I troubleshoot errors in my data?

Data Manager helps teams to connect, clean, and reconcile their data, by bringing in data from multiple types of data sources. Data Manager connects to these sources by using Adapters to run a job and bring data into Data Manager.

Troubleshooting Data Manager errors

If you encounter errors while running a job, check both your data and Adapter configuration to make sure they are set up correctly:

1. Check connectivity

   • Verify that your data is accessible by reviewing the Check connectivity section.

2. Check Authentication and Authorization

   • Find your Adapter in the Learn about Adapters section of the documentation.

   • Review the Authentication and Authorization requirements for your Adapter. If you are using a Flat File Adapter, you can skip this step.

3. Follow the troubleshooting flow for your Adapter type:

   • Flat File Adapter

   • SQL & ODBC Adapter

   • Product Adapter

Check connectivity

Since the Data Manager Client runs locally and can retrieve information from both local and cloud sources, you need a reliable path between the data source and the Data Manager Client. Network problems, such as closed firewalls, DNS failures, incorrect URLs, timeouts, and intermittent connectivity, can cause API calls to fail unpredictably.

1. Review Proxy Server Configuration - if a proxy server is being used, this must be specified when running the job in the Adapters Client, or the data source may not be available. Use the following arguments when configuring a job to run with a proxy service:

   --proxy Set a proxy server --proxy "Address" or -p Address --proxy-user Set username to authenticate against the proxy --proxy-user "Username" or --proxy-user Username --proxy-pass Set password to authenticate against the proxy --proxy-pass "Password" --proxy-default-credentials Use default credentials to authenticate against the proxy server

2. Review Server Certificate Status - verify the SSL for the server you are trying to hit is valid. If the SSL is invalid, you may encounter an error similar to the one below:

   The site cannot provide a secure connection - error: SSL protocol error

   If this error is encountered, you may be able to use the following argument to configure the job to run without a valid certificate:

   --bypass-invalid-certificate

3. Perform Handshake - Confirm that the Adapters Client can contact the data sources by locating the IP Address of the data source servers and using the ping command to verify connectivity in Terminal (in MacOS and Linux) or Command Prompt (in Windows/DOS). Here’s an example of ping in use, which confirms that data is coming from the listed site at 208.67.222.222 (substitute the IP Address you are attempting to reach):

   ping 208.67.222.222 PING 208.67.222.222 (208.67.222.222): 56 data bytes 64 bytes from 208.67.222.222: icmp_seq=0 ttl=58 time=21.006 ms 64 bytes from 208.67.222.222: icmp_seq=1 ttl=58 time=13.208 ms 64 bytes from 208.67.222.222: icmp_seq=2 ttl=58 time=11.132 ms 64 bytes from 208.67.222.222: icmp_seq=3 ttl=58 time=14.004 ms 64 bytes from 208.67.222.222: icmp_seq=4 ttl=58 time=17.659 ms 64 bytes from 208.67.222.222: icmp_seq=5 ttl=58 time=13.959 ms 64 bytes from 208.67.222.222: icmp_seq=6 ttl=58 time=13.512 ms 64 bytes from 208.67.222.222: icmp_seq=7 ttl=58 time=14.193 ms ...

Check Authentication and Authorisation

Each Adapter has different Authentication and Authorisation requirements. Based on the adapter you have selected.

These are listed in the documentation under “Authentication and Authorisation”.

Check the data from a Flat File Adapter

If you encounter an error while running a job with a Flat File Adapter (CSV or TXT data source), follow these steps to troubleshoot:

1. Verify Job Execution History

• Check if the job successfully imported data by reviewing the Job Execution History.

• A checkmark means success, while a warning icon indicates an issue.

• Click the warning icon ⚠️ for details on the problem.

2. Review the Raw Data

• Use the View Raw Data screen to check for missing attributes or fields.

• Missing fields may indicate a configuration error in the CSV Adapter.

3. Check the Delimiter Character

• The delimiter separates values in a CSV (e.g., a comma or tab character).

• If missing or misconfigured, errors like this may appear:

  Line 280: The column count is not correct.

4. Verify Column Names and Mappings

• Each column in your data source should have a corresponding Column Mapping.

• Errors may appear if:

  • A column name is incorrect.

  • The total number of columns is mismatched.

  • A column contains unexpected spaces.

  • Example error:

    Line 280: The column count is not correct. Error in line 1: Cannot find column 'cpu' in the Column Mappings. You have incorrectly entered a column name, or the name of the column in the data source contains a space.

5. Confirm the Qualifier Character

• A Qualifier character ensures text with commas is treated as a single value. The typical qualifier character is double quotes (").

• Example:

  • Without a qualifier: Microsoft Windows 10, v. 22H2 is read as two values because the comma is seen as a delimiter character.

  • With a qualifier: “Microsoft Windows 10, v. 22H2" is read as a single value because it is encased in quotes.

• Ensure you are using standard double quotes (") and not Smart Quotes.

6. Check Date Formats

• Ensure the Date Format matches the format of the source data exactly.

• Even a small mismatch can cause errors, which would typically looks like this:

  Column ‘whenCreated', date value '20/04/2009 1:47' cannot be converted to 'd/MM/yyyy HH:mm (8/02/2006 06:30).'

• In this case, the correct format should be d/MM/yyyy H:mm instead of d/MM/yyyy HH:mm.

Check the data from a SQL & ODBC Adapter

If you encounter an error when running a job with a SQL & ODBC Adapter running a job from a database source, review your data source and the configuration of your Adapter using the following steps:

1. Verify the data - The first step is to verify that data was brought in successfully when the job was run. You can do this by viewing the Job Execution History. A checkmark in the far-right column means the job was run successfully, while a warning icon indicates a problem was encountered. You can click on the warning icon to get more information about the nature of the problem. Remember - your account needs to have read only access to the database in order to retrieve data.

2. Review the Raw Data - The next step is to review the condition of the data itself using the View Raw Data screen. If there are missing attributes or fields from the retrieved data, you should review the query that you have entered when configuring the SQL & ODBC Adapter and ensure it is correct.

3. Review the Job Configuration screen - this is slightly different for SQL and OBDC connections:

   1. For SQL Connections:

      1. Review the Database Address and Database Name.

      2. Authentication - if you are using a standard Username and Password, ensure these are current and correct. If you are using Integrated Security, ensure that the correct credentials and the connection strings are provided by the database source owner.

      3. Test the SQL Query - It’s necessary to confirm that the query is correct. We suggest to reach out to the SME and provide them the query to execute against the source directly, to verify that the query itself is correct. After the query has been run, verify the structure of the retrieved data.

   2. For ODBC Connections - Ensure the driver has been correctly installed for whichever of the two techniques below you wish to use:

      1. If you are opening an ODBC Datasource directly, ensure that you have added a User DSN and selected a driver to create a new User Data Source Name 'DSN'.

      2. If you are using a Connection String, ensure it is correctly configured by reviewing with the database source owner.

Check the data from a Product Adapter

If you encounter an error running a job with a Product Adapter that connects to a third-party tool, review your data source and the configuration of your Adapter using the following steps:

1. Verify the data - The first step is to verify that data was brought in successfully when the job was run. You can do this by viewing the Job Execution History. A checkmark in the far-right column means the job was run successfully, while a warning icon indicates a problem was encountered. You can click on the warning icon to get more information about the nature of the problem.

2. Review packet sizing - In API connections, packet sizing could be an issue, depending on the number of records that reside in the source. If your Adapter consistently suffers from timeout errors, we recommend that you set a lower API limit so it does not timeout if the number of source records or “packet size” is large.

3. Review the Raw Data - The next step is to review the condition of the data itself using the View Raw Data screen. If there are missing attributes or fields from the retrieved data, these may be due to:

   1. The product Adapters having a specific call that defines the fields returned. This call may exclude the fields that you are expecting. See Step 4, below, for more information.

   2. If you are using a custom Adapter, the wrong selection may have been made, such as the fields returned by the custom Adapter or the API call used..

   3. If the call is customisable within the source app, the report available there may be incorrect or not accessible with the current permissions. Your tool SME will need to correct that.

4. Review the API Call - The next step is to review the API call that the Adapter is making to the Data Source. Please refer to the Adapters section of the documentation, where the API call for each Adapter is listed under “API Call”. For example, the Azure Virtual Machine Adapter has the following API Call listed:

   Device.Read.All

5. Review the Fields Returned - The next step is to review which fields are returned by the Adapter. These are also listed in the Adapters section of the documentation. For example, the Azure Virtual Machine Adapter returns the following fields:

   Name StorageProfileImageReferenceSku AdminUsername Id StorageProfileImageReferenceVersion ProvisionVMAgent Type StorageProfileImageReferenceExactVersion EnableAutomaticUpdates Location StorageProfileImageReferenceId PatchMode VmId OsDiskType AssessmentMode VmSize OsDiskName AllowExtensionOperations NumberOfCores OsDiskCreateOption RequireGuestProvisionSignal OsDiskSizeInMB OsDiskCaching BootDiagnosticsEnabled ResourceDiskSizeInMB OsDiskStorageAccountTyp BootDiagnosticsStorageUri MemoryInMB OsDiskId ProvisioningState MaxDataDiskCount OsDiskSizeGB StorageProfileImageReferencePublisher DataDiskCountTotalDataDiskSizeGB StorageProfileImageReferenceOffer ComputerName