How to troubleshoot data-pipeline export jobs
Platform Notice: Data Center Only - This article only applies to Atlassian apps on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
In some cases, the Data pipeline export might be incomplete or missing some files. Therefore, it is recommended to troubleshoot it and see if it is generating the expected output based on the data or if it's breaking up.
Environment
Jira X
Diagnosis
The pipeline export path is missing some files mentioned in Data pipeline export schema
<shared-home>/data-pipeline/export/<job-id>if you run Jira in a cluster<local-home>/data-pipeline/export/<job-id>you are using non-clustered Jira
Errors during the export
2025-02-07 06:11:12,473+0000 data-pipeline-export-executor-0 ERROR abarnwal [c.a.b.i.core.service.DefaultDataExportOrchestrator] Failed writing entities to file - processId: 1222
java.lang.NullPointerException: Cannot invoke "java.lang.Long.longValue()" because the return value of "com.atlassian.jira.issue.link.IssueLink.getSourceId()" is null
at com.atlassian.business.insights.jira.extract.issuelinks.IssueLinksExtractor.lambda$getIssueLinks$1(IssueLinksExtractor.java:58)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
at com.atlassian.business.insights.jira.extract.issuelinks.IssueLinksExtractor.getIssueLinks(IssueLinksExtractor.java:63)
at com.atlassian.business.insights.jira.extract.issuelinks.IssueLinksExtractor.enrich(IssueLinksExtractor.java:71)
at com.atlassian.business.insights.jira.extract.serialize.IssueSerializer.lambda$serialize$0(IssueSerializer.java:59)Enabling debugging for the following packages reveals that jobs are not running during their scheduled time or throw errors during execution
com.atlassian.business.insights.core.ao.dao.entity.Metadata
com.atlassian.business.insights.core.service.DataExportJobExecutor
com.atlassian.business.insights.core.service.DefaultDataExportOrchestrator
com.atlassian.business.insights.core.writer.ScatterDatasetWriter
com.atlassian.business.insights.core.writer.CsvMapWriterCause
If there are errors during the export with/without enabling the debugging packages raise the issue to Atlassian support for help ,
Some of the missing files might be expected when there are no entries to be exported in their respective tables like AO_D530BB_CANNEDRESPONSE or AO_56464C_APPROVAL at export time
Solution
The errors and exceptions root cause might be inconsistency like orphan entries in one of the tables (issue link)
For missing files, once the corresponding tables get populated the files will start showing up.
Was this helpful?