Bitbucket on AWS Amazon Linux instances fail to start due to the Hotpatch for Apache Log4j tool
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
AWS started installing a Log4jHotPatch tool for JDKs shipped on Amazon Linux instances on 17 December 2021 as a means to address the CVE-2021-44228 security vulnerability with Log4j.
The tool injects a Java agent into a running JVM process and adds an "agent" jar file in Bitbucket's class path.
The jar file gets removed once Log4jHotPatch has been loaded. This causes an error during Bitbucket start up because it is unable to find the jar file.
References:
Environment
Bitbucket Server and Data Center on AWS Amazon Linux
Diagnosis
The atlassian-bitbucket.log file shows the following error during start up:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
2021-12-20 20:48:26,913 ERROR [spring-startup] l.servicelocator.ServiceLocator Cannot build ServiceLocator
liquibase.exception.UnexpectedLiquibaseException: java.io.FileNotFoundException: /tmp/agent1234567890123456789.jar (No such file or directory)
at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:129)
at liquibase.servicelocator.ServiceLocator.<init>(ServiceLocator.java:59)
at liquibase.servicelocator.ServiceLocator.<clinit>(ServiceLocator.java:43)
at liquibase.database.DatabaseFactory.<init>(DatabaseFactory.java:28)
at liquibase.database.DatabaseFactory.getInstance(DatabaseFactory.java:46)
at liquibase.integration.spring.SpringLiquibase.createDatabase(SpringLiquibase.java:410)
at liquibase.database.ExtendedSpringLiquibase.createDatabase(ExtendedSpringLiquibase.java:57)
at liquibase.integration.spring.SpringLiquibase.createLiquibase(SpringLiquibase.java:375)
at liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:36)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at java.lang.Thread.run(Thread.java:748)
... 63 frames trimmed
Caused by: java.io.FileNotFoundException: /tmp/agent1234567890123456789.jar (No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:228)
at java.util.zip.ZipFile.<init>(ZipFile.java:157)
at java.util.jar.JarFile.<init>(JarFile.java:171)
at java.util.jar.JarFile.<init>(JarFile.java:108)
at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:152)
at liquibase.resource.ClassLoaderResourceAccessor.getResourcesAsStream(ClassLoaderResourceAccessor.java:53)
at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:115)
... 13 common frames omittedIn the sample stack trace above, the jar file that could not be found is:
/tmp/agent1234567890123456789.jarThe numeric portion in the filename changes for each restart
Resolution
Disable the
Log4jHotPatchtool by calling:1sudo touch /etc/log4j-cve-2021-44228-hotpatch.killAfterwards, start up Bitbucket
Was this helpful?