Creating DBR message fails with: KryoException: Buffer overflow. Available: 0, required: 1
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Jira Data Center 8.13andlater
Solution
Context
Since JIRA DC 8.12 we are using Document Based Replication to replicate the index across the cluster. When a change on the issue is triggered on one node, JIRA synchronously re-indexes this issue then asynchronously serialises the object with all Lucene document(s) and distributes it to other nodes.
JIRA is using Kryo for the serialisation/deserialisation of Lucene documents. JIRA comes with some assumptions about how big the serialised documents may be. By default the maximum size of the object with Lucene documents is set to 16MB.
Problem
It is possible that a full issue reindex (including all related entities) is triggered by a plugin on an issue with a large number of comments, worklogs and history and will produce a document larger than 16MB. If this happens you will see a similar log on the node which tried to create the DBR message:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[c.a.j.cluster.dbr.DefaultDBRSender] [DBR] [SENDER] Error when creating a dbr update with related message for documents: ...
...
com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 1
Serialization trace:
type (org.apache.lucene.document.TextField)
fields (org.apache.lucene.document.Document)
worklogs (com.atlassian.jira.cluster.dbr.DBRMessageUpdateWithRelatedData)
at com.esotericsoftware.kryo.io.Output.require(Output.java:186)
at com.esotericsoftware.kryo.io.Output.writeBoolean(Output.java:635)
at com.esotericsoftware.kryo.serializers.UnsafeField$BooleanUnsafeField.write(UnsafeField.java:151)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:107)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:568)
at com.esotericsoftware.kryo.serializers.ReflectField.write(ReflectField.java:70)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:107)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:637)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:158)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:43)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:568)
at com.esotericsoftware.kryo.serializers.ReflectField.write(ReflectField.java:70)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:107)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:568)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:154)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:43)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:568)
at com.esotericsoftware.kryo.serializers.ReflectField.write(ReflectField.java:70)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:107)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:553)
at com.atlassian.jira.cluster.dbr.KryoDBRMessageDataSerializer.serialize(KryoDBRMessageDataSerializer.java:55)Side note: In general, it is fine for DBR messages to fail sometimes (~5% rate) as there is another replay mechanism that will make sure indexes on all nodes are consistent and will re-index missing data.
However, there is another issue that may cause those "big" issues to fail index replay when the number of related entities (comments, worklogs) is greater then 1000: JRASERVER-71980, so in this case, both problems amplify each other.
Workaround
Serialisation exception when creating DBR message
Issue: JRASERVER-71976
The maximum size of the serialised data in a single DBR message is set to 16MB. It can be overridden with the following system property (example: overriding the maximum size to 32MB). Please don't set this parameter to a very high value.
1
-Dcom.atlassian.jira.cluster.dbr.serialization.max.size.bytes=33554432⚠️ Note: you will have to set this property on every node and this will require a rolling restart of all nodes.
Replay failing for a large list of comments or worklogs
Issue: JRASERVER-71980
Currently there is no workaround for this. Note that most of the time this should not be a problem and the index will be consistent across the cluster ✔️. Every worklog or comment item on this list (when created o updated) was replicated (via DBR and the backup replay mechanism) via individual DBR messages and index replay operations. Those messages and operations will not be affected by JRASERVER-71976 and JRASERVER-71980.
The problem only affects re-index issue operations which trigger a full issue reindex (with all comments and worklogs). This is usually caused by misuse of JIRA indexing API: plugins update the issue only but trigger a full issue re-index (issue with all comments and worklogs) issue re-index instead of reindexing the issue itself. ℹ️ Usually disabling the plugin triggering this re-indexing action should solve the problem.
Was this helpful?