Project import takes a long time validating the data
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
When running a project import, the entire XML archive data is mapped and validated in the Import Validation step. This step takes a long time on one or many tables.
Example:
Mapping and validating import data for project 'PKEY'.
Extracting project data. Processing AO_7DEABF_TESTSTEP: Entity 1859358 of 2198873.
Diagnosis
Inspect the table entries in the XML file for big records. You may use this Python script to print the length of all non-empty <string> elements in data rows that belong to the AO_7DEABF_TESTSTEPtable:
Python code
import xml.etree.ElementTree as ET
from unidecode import unidecode
tree = ET.parse('activeobjects.xml')
root = tree.getroot()
for child in root.findall('{http://www.atlassian.com/ao}data'):
if child.attrib.get('tableName') == 'AO_7DEABF_TESTSTEP':
for row in child.findall('{http://www.atlassian.com/ao}row'):
for string in row.findall('{http://www.atlassian.com/ao}string'):
if string.text != None:
print(len(string.text))
XML data sample
<data tableName="AO_7DEABF_TESTSTEP">
<column name="DATA" />
<column name="ID" />
<column name="ISSUE_ID" />
<column name="ORDER_ID" />
<column name="RESULT" />
<column name="STEP" />
<column name="CREATED_BY" />
<column name="MODIFIED_BY" />
<row>
<string xsinil="true" />
<integer>121</integer>
<integer>11620</integer>
<integer>2</integer>
<string xsinil="true" />
<string>Step 1 2 3</string>
<string>user1</string>
<string>user1</string>
</row>
<row>
<string xsinil="true" />
<integer>122</integer>
<integer>11621</integer>
<integer>1</integer>
<string xsinil="true" />
<string>Step 1 2 3</string>
<string>user1</string>
<string>user1</string>
</row>
...Cause
Having large data elements in the XML may significantly delay the import validation steps. In our case, we saw that elements of more than 6M have caused the validation of the table to take over 4 hours.
Solution
Find those records in the source system. For example, in the AO table above, you can refer to the
ISSUE_IDfield to open the issues that have the problematic entries.Note:
ISSUE_IDis the third element defined in the columns of the data so will always be the third element in the rows:<column name="DATA" /> <column name="ID" /> <column name="ISSUE_ID" /> <column name="ORDER_ID" /> <column name="RESULT" /> <column name="STEP" /> <column name="CREATED_BY" /> <column name="MODIFIED_BY" />Fix the source system so those large records aren't present in the XML anymore. In this example, the problematic issues couldn't not be opened at all due to the large Zephyr data so the admin deleted them via the API.
Re-export the XML backup.
Other Notes
The table above is used as an example and is coming from the Zephyr third-party app. Zephyr test steps do not have a limit on the number of characters the user can enter and therefore, may cause this problem.
Was this helpful?