A Jira or JSM Mail Handler configured with the OAuth 2.0 integration fails to create new issues from any incoming email
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Since Jira 8.10.0 (or in Jira Service Management 4.10.0) was released, it has become possible to configure incoming Mail Handlers using the Oauth 2.0 Authentication instead of the Basic Authentication, which will be eventually deprecated by Google and Microsoft.
The purpose of this KB article is to list all the known scenarios where a mail handler (Jira Mail Handler or JSM Email Requests) configured with Oauth 2.0 stops processing new incoming emails.
ℹ️ Please note that this KB article only applies to mail handlers configured with Oauth 2.0 (It does not apply to mail handlers configured with the Basic Authentication)
Solution
Environment
Jira 8.10.0 or higher, configured with a Jira Mail Handler using the Oauth 2.0 authentication, via ⚙ > System > Incoming Mail
Or JSM (Jira Service Management) 4.10.0 or higher, configured with a JSM Mail Handler using the Oauth 2.0 authentication, via Project Settings > Email Requests
Causes
Root Cause 1 - The Jira (or JSM) Mail Handler is configured with a GMAIL Mail Server and fails to fetch new emails after some time
If a Jira or a JSM Mail Handler is configured using a GMAIL server with Oauth 2.0, the connection between the Jira application and GMAIL might break after a certain amount of time and new incoming emails will stop being processed. This issue is tracked in 2 different bugs:
This bug causes a failure after 1h when using GMAIL with SECURE_POP
This bug has not been fixed
This bug causes a failure after 30 days when using GMAIL with SECURE_IMAP
This bug has been fixed (please check the bug details for the fixed versions)
This scenario is detailed in the KB article The Jira or JSM mail handler fails to connect to the GMAIL server after some time when using Oauth 2.0, so please refer to that article to check how to diagnose it and fix it.
We also recommend watching the 2 bugs linked above to get notified when updates are made to them.
Root Cause 2 - The Jira (or JSM) Mail Handler is configured with any type of Mail Server (GMAIL, Microsoft...) and gets stuck due to a temporary connectivity issue
When configured to use Oauth 2.0, the Jira Mail Handler (configured in ⚙ > System > Incoming Mail) and the JSM Mail Handler (configured in Project Settings > Email Requests) might at some point get stuck and stop processing any new incoming email.
This problem is caused by 2 bugs, and usually occurs when there is a temporary connectivity issue between the Jira application and the Mail Server. Due to the nature of these bugs, when a temporary connectivity issue occurs, the Mail Handler will indefinitely try to fetch a new Oauth 2.0, and will never get out of this loop.
We have 2 known bugs raised for each type of Mail Handler:
This bug was raised for the Jira Mail Handler
It will be fixed in a future Jira Version (please check the bug details for the fixed versions)
This bug was raised for the JSM Mail Handler
It has been already fixed in Jira Service Management (please check the bug details for the fixed versions)
To check whether this root cause is valid, collect thread dumps while the issue is happening. If you see a long running Caesium thread with the following stack trace, then this root cause is relevant:
Stack Trace from the Thread Dump
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
"Caesium-1-1" daemon prio=5 tid=0x00000000000000eb nid=0 runnable
java.lang.Thread.State: RUNNABLE
at java.base@11.0.13/java.net.SocketInputStream.socketRead0(Native Method)
at java.base@11.0.13/java.net.SocketInputStream.socketRead(Unknown Source)
at java.base@11.0.13/java.net.SocketInputStream.read(Unknown Source)
at java.base@11.0.13/java.net.SocketInputStream.read(Unknown Source)
at java.base@11.0.13/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source)
at java.base@11.0.13/sun.security.ssl.SSLSocketInputRecord.readFully(Unknown Source)
at java.base@11.0.13/sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(Unknown Source)
at java.base@11.0.13/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source)
at java.base@11.0.13/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base@11.0.13/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
at java.base@11.0.13/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source)
at java.base@11.0.13/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source)
at java.base@11.0.13/java.io.BufferedInputStream.fill(Unknown Source)
at java.base@11.0.13/java.io.BufferedInputStream.read1(Unknown Source)
at java.base@11.0.13/java.io.BufferedInputStream.read(Unknown Source)
- locked <0x00000000739144b7> (a java.io.BufferedInputStream)
at java.base@11.0.13/sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at java.base@11.0.13/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at java.base@11.0.13/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
- locked <0x0000000062f1103d> (a sun.net.www.protocol.https.DelegateHttpsURLConnection)
at java.base@11.0.13/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
- locked <0x0000000062f1103d> (a sun.net.www.protocol.https.DelegateHttpsURLConnection)
at java.base@11.0.13/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
- locked <0x000000005a1b5fc1> (a sun.net.www.protocol.https.HttpsURLConnectionImpl)
at com.nimbusds.oauth2.sdk.http.HTTPRequest.send(HTTPRequest.java:899)
at com.atlassian.oauth2.client.lib.token.DefaultTokenService.getToken(DefaultTokenService.java:139)
at com.atlassian.oauth2.client.lib.token.DefaultTokenService.forceRefresh(DefaultTokenService.java:71)
at com.atlassian.oauth2.client.storage.DefaultTokenHandler.refreshToken(DefaultTokenHandler.java:144)
at com.atlassian.oauth2.client.storage.DefaultTokenHandler.refreshTokenIfNeeded(DefaultTokenHandler.java:131)
at com.atlassian.oauth2.client.storage.DefaultTokenHandler.lambda$getRefreshedToken$1(DefaultTokenHandler.java:110)
at com.atlassian.oauth2.client.storage.DefaultTokenHandler$$Lambda$6901/0x0000000804b17840.call(Unknown Source)
at com.atlassian.oauth2.client.util.concurrent.KeyedLocks.executeWithLock(KeyedLocks.java:33)
- locked <0x000000007ce01059> (a java.util.concurrent.atomic.AtomicInteger)
at com.atlassian.oauth2.client.storage.DefaultTokenHandler.getRefreshedToken(DefaultTokenHandler.java:109)
at com.atlassian.oauth2.client.storage.DefaultTokenHandler.getRefreshedToken(DefaultTokenHandler.java:102)
at com.atlassian.oauth2.client.storage.DefaultTokenHandler.getRefreshedToken(DefaultTokenHandler.java:35)
at jdk.internal.reflect.GeneratedMethodAccessor1341.invoke(Unknown Source)
at java.base@11.0.13/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base@11.0.13/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy1519.getRefreshedToken(Unknown Source)
at com.atlassian.jira.internal.mail.processor.feature.oauth.MailOAuthServiceImpl.getOAuthToken(MailOAuthServiceImpl.java:53)
at com.atlassian.jira.internal.mail.processor.feature.authentication.OAuthAuthenticationStrategy.getPassword(OAuthAuthenticationStrategy.java:31)
at com.atlassian.jira.internal.mail.processor.feature.puller.MailPullerWorker.getMailServerPassword(MailPullerWorker.java:248)
at com.atlassian.jira.internal.mail.processor.feature.puller.MailPullerWorker.pullEmailForConnection(MailPullerWorker.java:137)
at com.atlassian.jira.internal.mail.processor.feature.puller.MailPullerWorker.pullMailFromAllValidChannels(MailPullerWorker.java:100)
at com.atlassian.jira.internal.mail.processor.feature.puller.MailPullerService.run(MailPullerService.java:33)
at com.atlassian.jira.internal.mail.processor.services.MailPullerExecutor.run(MailPullerExecutor.java:29)
at com.atlassian.jira.internal.mail.processor.services.AbstractMailExecutor.execute(AbstractMailExecutor.java:45)
at com.atlassian.jira.internal.mail.processor.services.MailJobRunner.runJob(MailJobRunner.java:35)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$$Lambda$4070/0x00000008040c7c40.accept(Unknown Source)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base@11.0.13/java.lang.Thread.run(Unknown Source)To resolve this issue:
the temporary solution consists in resetting the Mail Handler job by re-starting the Jira application (or the Jira node where the job is stuck)
the long term solution is to upgrade both Jira (and JSM) to their respective fixed versions listed in the bug tickets
Root Cause 3 - The Jira (or JSM) Mail Handler is configured with a Microsoft Mail Server, and the secret used for the Oauth 2.0 integration has expired
While configuring the Mail Handler with Oauth 2.0 and Microsoft, a client secret should have been created on the Azure portal, along with an expiration date (step 22 of Detailed steps to configure OAuth 2.0 integration with Microsoft Azure). If this secret expires, the Mail Handler will fail refresh the Oauth 2.0 token, and to process new incoming emails.
This scenario is detailed in the KB article The Jira (or JSM) Mail Handler stops processing new incoming emails due to the error "The provided client secret keys for app XXXXXXXX are expired", so please refer to that article to check how to diagnose it and fix it.
Was this helpful?