Bitbucket Data Center Index creation fails in AWS OpenSearch with 'illegal_argument_exception' error
Platform Notice: Data Center Only - This article only applies to Atlassian apps on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bitbucket fails to create indexes in AWS OpenSearch, resulting in 'illegal_argument_exception' errors. This article provides steps to resolve the issue by verifying and adjusting shard awareness configurations.
Solution
When setting up a clustered architecture connecting to AWS OpenSearch Service (version 2.11) for Bitbucket, indexing may fail, preventing code search from working. This issue occurs because Bitbucket cannot create an index/mapping in OpenSearch.
The following error appears in the logs:
DEBUG [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Attempting to create mapping in the search server
ERROR [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Failed to create indexes: ''[illegal_argument_exception, illegal_argument_exception, illegal_argument_exception, illegal_argument_exception]''
ERROR [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to synchronize the mapping in the search serverThe error means Bitbucket and the search server (Elasticsearch or OpenSearch) are incompatible.
To resolve this issue, follow these steps:
Delete all Bitbucket indexes in your OpenSearch Service or create a new domain.
Stop the Bitbucket instance.
In the start-webapp.sh file located at $BITBUCKET_INSTALLATION_DIR/bin, uncomment the line that says JVM_SUPPORT_RECOMMENDED_ARGS= and add -Dindex.codesearch.node.replicas.num=2.
Start the Bitbucket instance and verify the JVM argument is picked up by analysing the Bitbucket process in the output of ps -ef | grep java.
Enable wire debugging using the following API:
curl -u <ADMIN_USERNAME> -v -X PUT -d "" -H "Content-Type: application/json" "<BITBUCKET_BASE_URL>/rest/api/latest/logs/logger/org.apache.wire/debug"Replace <ADMIN_USERNAME> with a Bitbucket admin user and <BITBUCKET_BASE_URL> with your Bitbucket base URL.
Connect Bitbucket to your new OpenSearch service in AWS and trigger an index by adding a commit.
This issue occurs because the shard awareness configuration in AWS OpenSearch Service requires shard copies to be evenly distributed across awareness attributes (usually Availability Zones). If you have configured 3 awareness attributes (typically 3 Availability Zones) in AWS OpenSearch Service, you need at least 2 replicas (1 primary + 2 replica) so that the total shard copies are evenly distributed across availability zones. For more details, see the AWS article on configuring Amazon OpenSearch Service for high availability.
Was this helpful?