Users being logged out from JIRA

Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.

Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

User sessions are being terminated on a Crowd SSO configuration.

Environment

8.13.6

CROWD SSO (not SSO 2.0)

Diagnosis

How to Replicate

The user is logged out moving around on Jira (Dashboards, Issue searches, Boards, editing Issues), or when working on another Tab with Confluence or BitBucket.

Just editing the issue and remaining there, sporadically adding a comment, reproduces the issue consistently:

Incognito window
Sign it
Edit an issue (wait 5-10min)
Try to add a comment - if the login is not presented, add a comment and wait for more
Login is presented

Collecting data for the Diagnostic

Enable Network Debug:
- Navigate to the Jira Admin > System > Logging and profiling page;
- Scroll down until you see the Configure logging level for another package link.
- When clicking this link, you will be presented with a dialog box. We can use this to add the following classes with DEBUG level:org.apache.http

Reproduce the problem:
- Sign in using an Incognito window;
- Enable on your Browser the Developer Tools and clean up the content of the network panel;
- Edit an Issue;
- Try to add a comment - if the login is not presented, add a comment and wait more 5-10 min
- When the login is presented, save the har file on the Developer Tools.

Data Analysis

Crowd SSO is using the option Require Consistent Client IP Address;
The architecture is using Load Balancer and Proxy, for instance:

Browser Request → Load Balancer → HTTP Server → Load Balancer → Application Server

We can observe that the problem occurs after Jira submit a REST request to update the Development panel, receiving HTTP error 400:

Access Log

1
2

10.444.222.10 716x107921x1 - [19/Oct/2021:11:56:47 -0500] "GET /track/rest/dev-status/1.0/issue/summary?issueId=314447&_=1634662178331 HTTP/1.0" 400 71 147 "https://xxx.com/track/projects/APS/issues/APS-3982?filter=allopenissues" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36" "xxxxxx"

When the Development Task was executed previously, the cookie was validated and the user was identified:

Atlassian-Jira.log

2021-10-19 11:54:12,097-0500 http-nio-8080-exec-17 DEBUG anonymous 714x107358x3 xxxxxx 10.777.11.252,10.444.222.10,10.444.44.178,10.444.222.10 /rest/dev-status/1.0/issue/summary [o.a.h.client.protocol.ResponseProcessCookies] Cookie accepted [crowd.rememberme.token="", version:0, domain:xxx.com, path:/crowd, expiry:Wed Dec 31 18:00:10 CST 1969] 2021-10-19 11:54:12,106-0500 http-nio-8080-exec-17 DEBUG v3ry8 714x107358x3 xxxxxx 10.777.11.252,10.444.222.10,10.444.44.178,10.444.222.10 /rest/dev-status/1.0/issue/summary [o.a.h.client.protocol.RequestAddCookies] CookieSpec selected: default

However, later on, the same task could not use the cookie for the authentication, the user was not identified and the request for authentication failed with Error 400, and Invalid Token:

Atlassian-Jira.log

2021-10-19 11:56:47,306-0500 http-nio-8080-exec-7 DEBUG anonymous 716x107921x1 xxxxxx 10.777.11.252,10.444.222.10,10.444.44.178,10.444.222.10 /rest/dev-status/1.0/issue/summary [o.a.h.client.protocol.RequestAddCookies] CookieSpec selected: default

2021-10-19 11:56:47,306-0500 http-nio-8080-exec-7 DEBUG anonymous 716x107921x1 xxxxxx 10.777.11.252,10.444.222.10,10.444.44.178,10.444.222.10 /rest/dev-status/1.0/issue/summary [o.a.h.client.protocol.RequestAddCookies] Cookie [version: 0][name: JSESSIONID][value: 7E0796E379AE8445167F3FEDBEBBBCA3][domain: xxx.com][path: /crowd][expiry: null] match [(secure)xxx.com:443/crowd/rest/usermanagement/1/session/JM5VfhOd_325c6VlEYiFfwAAAAAAAIACdjNyeTg] 
... 
2021-10-19 11:56:47,375-0500 http-nio-8080-exec-7 DEBUG anonymous 716x107921x1 xxxxxx 10.777.11.252,10.444.222.10,10.444.44.178,10.444.222.10 /rest/dev-status/1.0/issue/summary [o.apache.http.wire] http-outgoing-18708 << "<?xml version="1.0" encoding="UTF-8" standalone="yes"?><error><reason>INVALID_SSO_TOKEN</reason><message>Token doesn't match the existing token.</message></error>" 
... 
2021-10-19 11:56:47,387-0500 http-nio-8080-exec-7 DEBUG anonymous 716x107921x1 xxxxxx 10.777.11.252,10.444.222.10,10.444.44.178,10.444.222.10 /rest/dev-status/1.0/issue/summary [o.apache.http.wire] http-outgoing-18709 >> "<?xml version="1.0" encoding="UTF-8" standalone="yes"?><validation-factors><validation-factor><name>remote_address</name><value>10.444.222.10</value></validation-factor><validation-factor><name>X-Forwarded-For</name><value>10.777.11.252, 10.444.222.10, 10.444.44.178</value></validation-factor></validation-factors>" 
...
2021-10-19 11:56:47,406-0500 http-nio-8080-exec-7 DEBUG anonymous 716x107921x1 xxxxxx 10.777.11.252,10.444.222.10,10.444.44.178,10.444.222.10 /rest/dev-status/1.0/issue/summary [o.apache.http.wire] http-outgoing-18709 << "HTTP/1.1 400 Bad Request[\r][\n]"

We can observe that is occurring a forward action:

Atlassian-jira.log

2021-10-19 11:56:47,387-0500 http-nio-8080-exec-7 DEBUG anonymous 716x107921x1 xxxxxxx 10.777.11.252,10.444.222.10,10.444.44.888,10.444.222.10 /rest/dev-status/1.0/issue/summary [o.apache.http.wire] http-outgoing-18709 >> "<?xml version="1.0" encoding="UTF-8" standalone="yes"?><validation-factors><validation-factor><name>remote_address</name><value>10.444.222.10</value></validation-factor><validation-factor><name>X-Forwarded-For</name><value>10.777.11.252, 10.444.222.10, 10.444.44.178</value></validation-factor></validation-factors>"

Following the document Configuring Trusted Proxy Servers, it was missing the proxy Servers IP at the trusted list.

The information provided on the debug lines were anonymized.

Cause

When a user first logs into Crowd, the application generates a token for the user with validation factors. The validation factors above include a remote_address (which is the IP address of the proxy), and X-Forwarded-For address (which should be the client/user's original IP address).

When a user tries going to another application, Crowd tries to validate that token with the original X-Forwarded-For address. As long as the remote_address is in your trusted proxy settings, Crowd will not expect these IPs to be the same. Since one of the proxies is generating a new IP each time the user tries to access, Crowd thinks that these requests are coming from two different IPs, even though they are both coming from the same user on the same machine.

Solution

Add the domain and the range of IPs from the proxy Servers

Log in to the Crowd Administration Console.
In the top navigation bar, click > Trusted proxy servers.
Type the IP address or the hostname of the proxy server. Possible values are:
- A full IP address, e.g. 192.168.10.12 (IPv4) or 2001:db8:85a3:0:0:8a2e:370:7334 (IPv6).
- An IPv4 subnet using wildcard notation, e.g. 192.168...
- An IPv4 or IPv6 subnet, using CIDR notation, e.g. 192.168.10.1/16 (IPv4) or 2001:db8:85a3::/64 (IPv6). For more information, see the introduction to CIDR notation on Wikipedia and RFC 4632.
- A hostname, e.g. proxy.example.org. All IP addresses bound to the given hostname will be trusted.
  ⚠️ Using hostnames will cause DNS requests to be sent, which might affect Crowd performance.
Click Add.
It is not required to restart any server.
Clean up session cookies.