Unable to access JMX monitoring via Jconsole when Bitbucket Server is hosted in AWS
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
Unable to access JMX counters when Bitbucket Server is hosted in an Amazon EC2 instance which is part of a VPC.
Specifically jconsole says the connection to the remote JMX fails.
Diagnosis
Environment
Amazon EC2
VPC using a Security Group with restrictive rules
Diagnostic Steps
Verify that the JMX port you specified is open and listening
Run jconsole with debug logging enabled for more data on the issue:
jconsole -J -Djava.util.logging.config.file=path_to_logging.properties_for_jconsoleYou should see something similar to the following in the jconsole debug output:
May 02, 2017 10:09:51 AM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=Proxy[RMIServer,RemoteObjectInvocationHandler[UnicastRef [liveRef: [endpoint:[52.XX.XXX.XXX:33920](remote),objID:[-6cdfb6b5:15bc6963763:-7fff, 2103495553046990789]]]]]] failed to connect: java.rmi.ConnectException: Connection refused to host: 52.XX.XXX.XXX; nested exception is:
java.net.ConnectException: Operation timed out (Connection timed out)Cause
JMX uses 2 ports, the one specified in the setenv.sh file and a port that is randomly assigned. In environments where access to ports is restricted, binding to the randomly assigned port may be prevented.
Solution
Resolution
Set RMI port to the same JMX port specified in the setenv file by adding the following to the JVM_SUPPORT_RECOMMENDED_ARG configuration of the setenv file:
JVM_SUPPORT_RECOMMENDED_ARG="-Dcom.sun.management.jmxremote.rmi.port=3333"setenv and environment variable changes in Bitbucket Server 5.0+
Starting with Bitbucket Server 5.0, setenv.sh and setenv.bat have been removed. The options that were set in this file can now be set via environment variables. Where to set the environment variable depends on which Operating System you're running on.
Linux
When using the atlbitbucket service on Linux, the environment variables are ignored. You must set the parameters in _start-webapp.sh (or start-bitbucket.sh). These values will be read when the service starts.
As an example, to set JVM_SUPPORT_RECOMMENDED_ARGS, you would add this line to the file:
Example
JVM_SUPPORT_RECOMMENDED_ARGS=-XX:+HeapDumpOnOutOfMemoryErrorWindows
Set the parameter as an environment variable for the user running Bitbucket Server. For example, if you want to set JVM_SUPPORT_RECOMMENDED_ARGS, create it as an environment variable and assign the appropriate value to it. When Bitbucket Server starts using the startup scripts or service, it will pick up and apply this value.
Was this helpful?