Cleanse your data

The Cleanse process filters duplicate, erroneous, or unneeded information from your data. It is done after running a job, but before Importing. Cleansing can be triggered manually, but is usually triggered by running the Cleanse and Import client from the command-line.

Cleanse your Data manually

To cleanse a single data source manually from within Data Manager:

1. From within Data Manager, select Cleanse and import in the left menu bar.

2. Select a single Data sources to cleanse.

3. Select … and then select Cleanse. You will now see the results from the previous cleanse operation.

4. Select >>Start. The cleansing process will begin and the results from the cleansing will update automatically. If you leave this page, the cleanse will not be halted but will complete in the background.

To cleanse multiple data sources manually from within Data Manager:

1. From within Data Manager, select Cleanse and import in the left menu bar.

2. Select one or more Data sources in the left menu bar.

3. At the top of the page, select Cleanse selected. You will now see the results from the previous cleanse operation.

4. Select >>Start. The cleansing process will begin and the results from the cleansing will update automatically. If you leave this page, the cleanse will not be halted but will complete in the background.

You can view the results of your Cleansed data in the Data Manager > Cleansing Analysis screen, where you can see how the different cleansing rules have affected your data.

Cleanse your data using the Cleanse and Import client for MacOS

While Cleansing and Importing can be done through Data Manager, you can also Cleanse and Import your data using the Cleanse and Import client from the command-line.

To run the Cleanse and Import client client for Mac/Linux:

1. Open a command line interface by navigating to the Finder App in Mac (Applications > Utilities > Terminal).

2. Navigate to the location of the Data Manager clients.

   1 cd /Users/Shared/assets_data_manager

3. Select the version of the Cleanse and Import client that matches your CPU. Pick from:

   1 2 cd osx-arm64/assets-cleanse-and-import-client cd osx-x64/assets-cleanse-and-import-client

4. If you like, you can use the help command to find out more information about how to configure the Cleanse and Import client:

   1 ./dm-cleanseimport --help

5. You will need the following two pieces of information to run the app:

   1. Your Workspace ID. This should look something like this:

      1 e44regc98-8d2r-3af7-011t-209p44716366

   2. Your Cleanse and Import client token. You can create this token by following the instructions here. A complete token should look something like this:

      1 ATCTT3xFrTN0U64VR99p1VfpQtLlpar7R4hq6PSkiNNaNOyk37P_7URVmROOp1Orez-ijlluMRwJA7Cw1LsgMFq

6. Run the Cleanse and Import client app on the command line interface, replacing the placeholder text with your workspace ID and Cleanse and Import client token:

   1 ./dm-cleanseimport --run "cleanse" --workspace-id <workspaceId> --token <your-Cleanse-and-Import-token>

7. You can also use these optional parameters when running the app:

   1. Choose both Import and Cleanse the data simultaneously by replacing the “cleanse" parameter with “full".

   2. Choose a single Data Source Type to Cleanse or Import. If your Data Source Type name includes spaces, you will have to encase it in quotes:

      1 --type "My Data Source Type Name"

   3. Choose a single Data Source to Cleanse or Import. If your Data Source name includes spaces, you will have to encase it in quotes:

      1 --ds "My Data Source"

   4. Choose a single Object Class to Cleanse or Import.

      1 --object "Compute"

   5. Choose a proxy server:

      1 --proxy "Address" or -p "Address"

   6. Choose a username to authenticate against the proxy server:

      1 --proxy-user "Username" or --proxy-user Username

   7. Choose a password to authenticate against the proxy server:

      1 --proxy-pass "Password" or --proxy-pass Password

   8. Use the default credentials to authenticate against the proxy server:

      1 --proxy-default-credentials

   9. Bypass any invalid server certificates that appear when authenticating to the proxy server:

      1 --bypass-invalid-certificate

8. A success message should tell you when the job is complete and if any errors were encountered while the Cleanse and Import client was running.

Cleanse your Data using the Cleanse and Import Client for Linux

While Cleansing and Importing can be done through Data Manager, you can also Cleanse and Import your data using the Cleanse and Import client from the command-line.

To cleanse your data using the Cleanse and Import client in a Linux environment:

1. Open a command line interface by using Cmd + Alt + T.

2. Navigate to the location of the Data Manager clients.

   1 cd /Users/Shared/assets_data_manager

3. Select the version of the Cleanse and Import client that matches your CPU. Pick from:

   1 2 cd linux-arm64/assets-cleanse-and-import-client cd linux-x64/assets-cleanse-and-import-client

1. If you like, you can use the help command to find out more information about how to configure the Cleanse and Import client:

1. You will need the following two pieces of information to run the app:

   1. Your Workspace ID. This should look something like this:

      1 e44regc98-8d2r-3af7-011t-209p44716366

   2. Your Cleanse and Import client token. You can create this token by following the instructions here. A complete token should look something like this:

      1 ATCTT3xFrTN0U64VR99p1VfpQtLlpar7R4hq6PSkiNNaNOyk37P_7URVmROOp1Orez-ijlluMRwJA7Cw1LsgMFq

2. Run the Cleanse and Import client app on the command line interface, replacing the placeholder text with your workspace ID and Cleanse and Import client token:

   1 ./dm-cleanseimport --run "cleanse" --workspace-id <workspaceId> --token <your-Cleanse-and-Import-token>

3. You can also use these optional parameters when running the app:

   1. Choose to Import and Cleanse the data simultaneously by replacing the “cleanse" parameter with “full".

   2. Choose a single Data Source Type to Cleanse or Import. If your Data Source Type name includes spaces, you will have to encase it in quotes:

      1 --type "My Data Source Type Name"

   3. Choose a single Data Source to Cleanse or Import. If your Data Source name includes spaces, you will have to encase it in quotes:

      1 --ds "My Data Source"

   4. Choose a single Object Class to Cleanse or Import.

      1 --object "Compute"

   5. Choose a proxy server:

      1 --proxy "Address" or -p "Address"

   6. Choose a username to authenticate against the proxy server:

      1 --proxy-user "Username" or --proxy-user Username

   7. Choose a password to authenticate against the proxy server:

      1 --proxy-pass "Password" or --proxy-pass Password

   8. Use the default credentials to authenticate against the proxy server:

      1 --proxy-default-credentials

   9. Bypass any invalid server certificates that appear when authenticating to the proxy server:

      1 --bypass-invalid-certificate

4. A success message should tell you when the job is complete and if any errors were encountered while the Cleanse and Import client was running.

Cleanse your data using the Cleanse and Import client for Windows/DOS

While Cleansing and Importing can be done through Data Manager, you can also Cleanse and Import your data using the Cleanse and Import client from the command-line.

To cleanse your data using the Cleanse and Import client in a Windows/DOS environment:

1. Open a command-line interface using Ctrl + Shift + Enter (to open as an Administrator), or by clicking on the Windows icon using the mouse, then typing in “CMD” and select “Run as administrator”.

2. Navigate to the location of the Data Manager clients.

   1 cd Users/Shared/assets_data_manager

3. Select the version of the Cleanse and Import client that matches your CPU. Pick from:

   1 2 cd win-x64/assets-cleanse-and-import-client cd win-x86/assets-cleanse-and-import-client

4. If you like, you can use the help command to find out more information about how to configure the Cleanse and Import app:

   1 dm-cleanseimport.app --help

5. You will need the following two pieces of information to run the app:

   1. Your Workspace ID. This should look something like this:

      1 e44regc98-8d2r-3af7-011t-209p44716366

   2. Your Cleanse and Import client token. You can create this token by following the instructions here. A complete token should look something like this:

      1 ATCTT3xFrTN0U64VR99p1VfpQtLlpar7R4hq6PSkiNNaNOyk37P_7URVmROOp1Orez-ijlluMRwJA7Cw1LsgMFq

6. Run the Cleanse and Import app on the command line interface, replacing the placeholder text with your Workspace ID and Cleanse and Import token:

   1 dm-cleanseimport.app --run "cleanse" --workspace-id <workspaceId> --token <your-Cleanse-and-Import-token>

7. You can also use these optional parameters when running the app:

   1. Choose to Import and Cleanse the data simultaneously by replacing the “cleanse" parameter with “full".

   2. Choose a single Data Source Type to Cleanse or Import. If your Data Source Type name includes spaces, you will have to encase it in quotes:

      1 --type my_data_source_name

   3. Choose a single Data Source to Cleanse or Import. If your Data Source name includes spaces, you will have to encase it in quotes:

      1 --ds my_data_source

   4. Choose a single Object Class to Cleanse or Import. Currently, Data Manager supports three types of objects classes: Compute, User, and Network.

      1 --object Compute

   5. Choose a proxy server:

      1 --proxy my_proxy_server_address

   6. Choose a username to authenticate against the proxy server:

      1 --proxy-user my_username

   7. Choose a password to authenticate against the proxy server:

      1 --proxy-pass my_password

   8. Use the default credentials to authenticate against the proxy server:

      1 --proxy-default-credentials

   9. Bypass any invalid server certificates that appear when authenticating to the proxy server:

      1 --bypass-invalid-certificate

8. A success message should tell you when the job is complete and if any errors were encountered while the Cleanse and Import client was running.