Jira Data Center 11.x Reindex Cannot Change DocValues Type from SORTED_NUMERIC to SORTED_SET
Platform Notice: Data Center Only - This article only applies to Atlassian apps on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Upgrading Jira Data Center to version 11.0.0 or later is incomplete with one or more of the following symptoms:
Jira fails to start after upgrade or upgrade tasks do not execute
OSGi components fail to initialize (e.g., webhooks throw
WebhooksNotInitializedException)Installed plugins appear enabled but are not functional (or missing modules)
Bundled gadgets are not available
Diagnosis
These messages may be found in atlassian-jira.log:
WARN [c.a.jira.index.AccumulatingResultBuilder] cannot change DocValues type from SORTED_NUMERIC to SORTED_SET for field "issue_subtasks"
java.lang.IllegalArgumentException: cannot change DocValues type from SORTED_NUMERIC to SORTED_SET for field "issue_subtasks"
at org.apache.lucene.index.FieldInfos$FieldNumbers.verifyConsistent(FieldInfos.java:289)
at org.apache.lucene.index.FieldInfos$FieldNumbers.setDocValuesType(FieldInfos.java:337)ERROR [c.a.j.search.lucene.LuceneClusterIndex] The recover of index from snapshot failed due to individual issues failure. We are skipping full reindex as it will also fail due to the exact same issues failures.
com.atlassian.jira.search.lucene.ReindexErrorDuringCatchUp:
om.atlassian.jira.index.IndexingFailureException: Indexing completed with errors
at com.atlassian.jira.search.lucene.LuceneClusterIndex.pickIndexSnapshotFromSharedHome(LuceneClusterIndex.java:370)INFO [c.a.j.search.lucene.LuceneClusterIndex] Failed to get index on this node. Blocking the start of this instance
INFO [c.a.jira.scheduler.JiraSchedulerLauncher] (Event: Level = (EventLevel: warning) , Key = (EventType: no-index) , Desc = We could not find a valid search index on the current node , Exception = Jira needs to have a valid search index after starting. During the node startup, all of the following steps failed to pass: <ul><li>Re-index missing data if a local issue index is less than 10% behind the database.</li><li>Check if Jira shared-home directory contains a recent index snapshot.</li><li>Trigger a full re-index.</li></ul><B>You need to fix this issue manually. Do not start other nodes until the indexing problem is resolved. </B>)[c.a.g.embedded.internal.GadgetUrlBuilder] GadgetUrlBuilder: could not parse spec at rest/gadgets/1.0/g/com.atlassian.jira.gadgets:two-dimensional-stats-gadget/gadgets/two-dimensional-stats-gadget.xml[c.a.j.p.webhooks.spi.JiraWebhookEventPublisher] Unexpected exception during webhookService.publish() call happened: The webhooks plugin hasn't been initialized yet. Webhook will not be published.
com.atlassian.webhooks.WebhooksNotInitializedException: The webhooks plugin hasn't been initialized yet. Webhook will not be published.
at com.atlassian.webhooks.internal.DefaultWebhookService.getExecutorOrThrow(DefaultWebhookService.java:390)
at com.atlassian.webhooks.internal.DefaultWebhookService.publish(DefaultWebhookService.java:264)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)Cause
Important: Jira Data Center 11 requires a full reindex on startup. Plan for appropriate downtime based on your instance size, or follow Upgrade Jira Data Center 10 to 11 with minimal downtime.
Jira Data Center 11 introduced two breaking changes to OpenSearch/Lucene indexing that modify DocValues field types for subtask sorting and issue properties.
When Jira attempts to start with an existing index from a pre-11.0 version, the DocValues type mismatch prevents OpenSearch from loading the index. This blocks Jira startup at a critical point:
Jira cannot complete its startup sequence
Upgrade tasks are never triggered
The JiraStartedEvent never fires
Components listening for startup events remain uninitialized
Solution
To successfully upgrade to Jira 11.0.0 or later, when encountering these errors:
Stop the entire Jira cluster.
Jira application upgrade on a single node.
Set the following JVM args:
-Dcom.atlassian.jira.startup.rebuild.local.index=false -Dcom.atlassian.jira.startup.pick.indexsnapshot.from.shared=false -Dcom.atlassian.jira.startup.allow.full.reindex=trueThis will run a full reindex upon startup.
Once indexing is complete, perform the necessary plugin updates.
Initiate a manual full reindex.
Once indexing is done, verify a successful upgrade:
Jira starts completely.
Upgrade tasks complete successfully.
All plugins and features are functional.
Restart the single node.
Remove the following JVM args:
-Dcom.atlassian.jira.startup.rebuild.local.index=false -Dcom.atlassian.jira.startup.pick.indexsnapshot.from.shared=false -Dcom.atlassian.jira.startup.allow.full.reindex=true
Propagate the upgrade to other nodes—once the first node is healthy and reindex is 100% complete, upgrade the remaining Data Center nodes.
The JVM parameters force Jira to:
Not use a local index from the previous version (
rebuild.local.index=false)Not attempt to copy index snapshots from other nodes (
pick.indexsnapshot.from.shared=false)Perform a full reindex from the database (
allow.full.reindex=true)
Was this helpful?