Tuning webhook performance parameters in Jira Data Center
Platform Notice: Data Center Only - This article only applies to Atlassian apps on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
The article discusses tuning webhook performance parameters in Jira Data Center.
Asynchronous webhooks
In Jira 9 and earlier, webhooks are processed synchronously, with the most computationally intensive work performed during the HTTP request. Administrators can opt in to asynchronous webhooks by adding a feature flag and performing a rolling restart.
Starting in Jira 10.0.0, asynchronous webhooks have become the default and only option. This change significantly enhances the end-user experience, as processing is now handled by a dedicated thread pool instead of being tied to the user request.
However, in both Jira 9 and 10, environments with exceptional webhook usage may exceed the guardrails set by the default parameters. The following parameters allow you to customize these limits to meet your organization’s needs.
Environment
Jira Data Center 10.3.5 LTS / 10.6.0 and later.
These parameters are unavailable in Jira from 10.0.0 – 10.3.4 LTS and 10.4.x – 10.5.x. Please upgrade to 10.3.5+ or 10.6.0+.
For all Jira Data Center 9 versions, please review the workarounds on JRASERVER-74492 - Enabling asynchronous WebHooks may result in the Web-Hook-Events-Processor queue overflowing.
Diagnosis
Problem 1: processing queue
You observe the following error in your atlassian-jira.log file:
2025-03-25 14:05:00,000+0000 http-nio-8080-exec-1 ERROR charlie 1x1x1 abcdef 0.0.0.0 /secure/AjaxIssueAction.jspa [c.a.event.internal.AsynchronousAbleEventDispatcher] There was an exception thrown trying to dispatch event [com.atlassian.jira.event.issue.IssueEvent@cafebabe[issue=TIS-1,comment=<null>,worklog=<null>,changelog=[GenericEntity:ChangeGroup][issue,10000][author,charlie][created,2025-03-25 14:04:59.00][id,10000],eventTypeId=2,sendMail=true,params={eventsource=action, baseurl=https://jira.example.com},subtasksUpdated=true,spanningOperation=Optional.empty]] from the invoker [com.atlassian.event.internal.ComparableListenerInvoker@cafebabe]
java.lang.RuntimeException: Task com.atlassian.webhooks.internal.DefaultWebhookService$$Lambda$1234/0x0000000cafebebe@cafebabe rejected from java.util.concurrent.ThreadPoolExecutor@cafebabe[Running, pool size = 5, active threads = 5, queued tasks = 250, completed tasks = 80000]. Listener: com.atlassian.jira.plugins.webhooks.spi.JiraWebhookEventPublisher event: com.atlassian.jira.event.issue.IssueEventSimplified:
Task rejected from ThreadPoolExecutor[Running, pool size = 5, active threads = 5, queued tasks = 250, completed tasks = 80000]Problem 2: HTTP dispatch queue
You observe the following error in your atlassian-jira.log file:
2025-03-25 14:05:00,000+0000 webhook-dispatcher:thread-1 WARN anonymous [c.a.w.internal.publish.DefaultWebhookDispatcher] Could not dispatch jira:issue_updated webhook to <WEBHOOK_URL>; a maximum of 500 dispatches are already in flight
2025-03-25 14:05:00,000+0000 webhook-dispatcher:thread-1 DEBUG anonymous [c.a.w.internal.publish.DefaultWebhookDispatcher] Skipping webhook invocation [f63f8397-7328-48d8-9c22-2a4eb2e30954] to <WEBHOOK_URL> (Too many webhook dispatches already in flight)Simplified:
Could not dispatch <EVENT_NAME> webhook ... a maximum of <MAX_IN_FLIGHT_DISPATCHES_COUNT> dispatches are already in flight
Too many webhook dispatches already in flightProblem 3: failure exponential backoff
You observe the following error in your atlassian-jira.log file:
2025-03-25 14:05:00,000+0000 webhook-dispatcher:thread-1 WARN anonymous [c.a.w.internal.history.DefaultInvocationHistoryService] Request to <WEBHOOK_URL> resulted in an error: Webhook failed too many times. Skipping this webhook for the next 11933msSimplified:
Webhook failed too many times. Skipping this webhook for the next <DELAY>msProblem 4: new webhooks are not processed despite being enabled
May happen when instance have more than 250 enabled webhooks configured.
There are no logs indicating that webhooks are rejected, but those are not sent to the destination.
Solution
Tuning webhook event/JQL configuration
Have you considered adjusting your webhook configuration within Jira? Reducing the scope of webhooks will decrease workload and avoid limits. For recommendations, please consult Best practices on working with webhooks in Jira Data Center.
These parameters differ from Jira 9, which used JVM arguments instead of application properties. Any webhook JVM arguments carried over from Jira 9 will not affect Jira 10.
The following parameters can be added to the jira-config.properties file (Advanced Jira application configuration).
Parameters for Problem 1
Title | Property | Description | Default Value |
|---|---|---|---|
Queue size |
| The size of the webhook processing queue. Any additional pending requests will be dropped and unrecoverable. | 250 |
Executor thread pool size |
| The number of | 5 |
Enqueue timeout |
| How long (milliseconds) a webhook event-producing thread will wait to add to a full queue before timing out. Setting this value too high could force users to wait indefinitely for their HTTP request to return. | 100 |
Parameters for Problem 2
Title | Property | Description | Default Value |
|---|---|---|---|
Max in-flight webhooks |
| The maximum number of webhooks that can be dispatched concurrently. This includes webhook requests that are currently in transit over the network, as well as those waiting for HTTP connections. Any additional requests will be dropped and unrecoverable. Before increasing this limit, consider tuning the I/O thread pool size, max number of HTTP connections per host, and max HTTP connections. | 500 |
Dispatch timeout |
| How long (milliseconds) a If unsuccessful, the webhook will be dropped. | 250 |
HTTP connections per host |
| The maximum number of concurrent HTTP connections allowed per host for dispatching webhooks. | 5 |
Total HTTP connections |
| The maximum number of concurrent HTTP connections for dispatching webhooks. A higher number of HTTP connections with insufficient I/O threads will increase the number of in-flight webhooks. | 200 |
I/O thread pool size |
| The number of HTTP I/O threads. Additional threads may increase CPU usage but will improve queue throughput. | 3 |
Parameters for Problem 3
Jira considers a webhook invocation "failed" if it fails to receive a valid, 2xx response. Before adjusting these parameters, investigate any potential errors with the integration or network.
Title | Property | Description | Default Value |
|---|---|---|---|
Backoff trigger count |
| The maximum number of consecutive failures before a webhook is deemed unhealthy and subsequently skipped. A single successful dispatch will mark the webhook as healthy. | 5 |
Backoff initial delay |
| How long (seconds) an unhealthy webhook is skipped when first deemed unhealthy. | 10 |
Backoff exponent |
| An unhealthy webhook is skipped with an initial delay. If the webhook continues to fail, this duration will keep on increasing based on this delay exponent up to the max delay or until the webhook is healthy. The backoff delay is computed as:
up to the configured max delay. | 1.2 |
Backoff max delay |
| The maximum duration for which an unhealthy webhook is skipped. | 2 |
Parameters for Problem 4
Important: that property is not controlled by the jira-config.properties file! It should be added as the system property to the the setenv.sh file e.g. -Datlassian.webhooks.search.default.limit=500
Title | Property | Description | Default Value |
|---|---|---|---|
Webhooks default search limit |
| Default limit of webhooks that will be processed, if property is not defined Jira uses the default. It affects the amount of active webhooks that will be processed by the system. | 250 |
Was this helpful?