Schedule an import of your data

The instructions below are for setting up a scheduled task to import data into Data Manager using third-party scheduling apps and the Cleanse and Import client.

The two apps used for demonstration are Cron (for MacOS/Linux) and Windows Scheduler (for Windows/DOS), but any scheduling app may be used as long as the command running the Cleanse and Import client is correctly configured.

If you have a process of placing the files on a server for consumption, additional scheduled tasks may be required to copy the files to the appropriate location for the Cleanse and Import client to consume.

You should leave an appropriate amount of time (1 hour) between scheduled tasks to allow them to complete.

Import your data using the Cleanse and Import client with a Cron job in MacOS and Linux

To create a Cron job to run the Cleanse and Import client:

1. Open a command line interface by navigating to the Finder App in Mac (Applications > Utilities > Terminal).

2. Navigate to the location of the Cleanse and Import client. This may be in a different location depending on how you have set up your Data Manager Clients. The Cleanse and Import Client will have the following filename:

   1 dm-cleanseimport.app

3. Create a new script to run the Cleanse and Import client. Make a new .sh file using a text editor like Nano. The example below creates a file named test-script.sh:

   1 nano test-script.sh

4. Now enter the command that runs the Cleanse and Import client. You will need both your WorkspaceID and a Cleanse and Import token. Here’s a sample script:

   1 2 3 #!/bin/bash echo "This is an sample Bash script that runs the Importers client." ./dm-cleanseimport --workspace-id <workspaceId> --token <your-token> –-run "import"

1. Save the script.

2. Make the script executable using the chmod command.

   1 chmod +x test-script.sh

3. Next, open the Crontab file by typing the following: The file opens in the vi editor. If the crontab had not previously existed, a new file is created.

   1 crontab -e

1. Add a line containing a Cron expression and the path to the script:

   1 [cron-expression] [path-to-script]

   A Cron expression consists of five fields that accept the respective number values:

   1 [minute: 0-59] [hour: 0-23] [day: 1-31] [months: 1-12] [weekday: 0-6]

1. Press i to turn on the Insert Mode.

2. Add the following line to the crontab file to run our test script every Tuesday and Thursday at 6 p.m.:

   1 0 18 * * 2,4 /Users/[username]/test-script.sh

3. Press ESC to leave insert mode.

4. Save the file and exit by pressing the colon : and then x to save the edits to your crontab.

5. You can view your list of existing cron jobs using the crontab command. The output displays all the previously added Cron expressions.

   1 crontab -l

Import your Data using the Cleanse and Import client with Windows Scheduler in Windows/DOS

To create a task in Windows Task Scheduler to run the Cleanse and Import Client:

1. Open Windows Task Scheduler

2. Create a subfolder called Data Manager.

3. Select Create Task from the right pane, and you are presented with the Create Task box. 

4. Enter the name of the task, which should be meaningful for the job to be done. In this example, we’ll use Test, but a more suitable name could be “Import JSM Assets.”

5. Select the radio button "Run whether user is logged on or not".

6. Select Change User or Group and enter the service account details. 

1. Select the Trigger tab and select New….

2. Enter the details of when the job will be executed (daily, weekly, monthly, the time of day to start, etc).

3. Select OK.

4. Select the Action tab and select New…. 

5. Ensure Actions: is set to Start a program.

6. Enter the Program/script as dm-cleanseimport.exe.

7. You must enter the following in the Add arguments (optional) field:

   1. Click on the Trigger tab and click on New….

   2. Enter the details of when the job will be executed (daily, weekly, monthly, the time of day to start etc).

   3. Click OK.

   4. Click in the Action tab and click on New…. 

   5. Ensure Actions: is set to Start a program.

   6. Enter the Program/script as dm-cleanseimport.exe.

   7. You must enter the following in the Add arguments (optional) field:

      1. Click on the Trigger tab and click on New….

      2. Enter the details of when the job will be executed (daily, weekly, monthly, the time of day to start etc).

      3. Click OK.

      4. Click in the Action tab and click on New…. 

      5. Ensure Actions: is set to Start a program.

      6. Enter the Program/script as dm-cleanseimport.exe.

      7. You must enter the following in the Add arguments (optional) field:

         1. Click on the Trigger tab and click on New….

         2. Enter the details of when the job will be executed (daily, weekly, monthly, the time of day to start etc).

         3. Click OK.

         4. Click in the Action tab and click on New…. 

         5. Ensure Actions: is set to Start a program.

         6. Enter the Program/script as dm-cleanseimport.exe.

         7. You must enter the following in the Add arguments (optional) field:

            1. You must specify the WorkspaceID, for example:

               1 --workspaceid e44regc98-8d2r-3af7-011t-209p44716366

            2. You must specify the Cleanse and Import Client token, for example:

               1 --token ATCTT3xFrTN0U64VR99p1VfpQtLlpar7R4hq6PSkiNNaNOyk37P_7URVmROOp1Orez-ijlluMRwJA7Cw1LsgMFqFnvhJapdniv4ewDc0twt7vPBhrbDts1fyPOW6Tw3S-Gzg6rRaEULWl2_JqQNvictFu_hK6wGxpGRvPTpyaQXzr9MtQ_8yKDms=0433HC52

            3. You must specify if you want to Cleanse, Import, or run both operations at the same time:

               1 2 3 --run import --run cleanse --run full

1. The Start in (optional): field is where the Data Manager client executable resides in the server (refer to the prerequisites). Specify the destination folder where the Data Manager client resides.

2. Click in the Settings tab.

3. Change the Stop the task if it runs longer than: from 3 days to 1 hour.

4. Click OK.

5. If the service account is used to initiate the task, you will be asked to enter the password for the service account. Enter the corresponding password and click on the SAVE button.

The task is now saved and can be executed from the task scheduler, it will be ran at the time defined in the above steps. The scheduled tasks can be tested by selecting the Task Name and selecting Run.