SVN indexing stuck in a loop due to the pipeline batch path limit being reached
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Subversion repository indexing never finishes and the indexing problem reported in the atlassian-fisheye.log file is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
2021-03-23 00:00:19,129 INFO [InitPing1 REPO ] fisheye BaseRepositoryScanner-ping - processing repository REPO (REPO)
2021-03-23 00:00:19,129 INFO [InitPing1 REPO ] fisheye Svn2Scanner-doSlurpTransaction - Starting slurp of REPO (REPO)
2021-03-23 00:00:20,321 WARN [InitPing1 REPO ] fisheye BaseRepositoryScanner-handleSlurpException - Problem processing revisions from repository REPO (REPO) due to class com.cenqua.fisheye.rep.RepositoryClientException - org.apache.subversion.javahl.ClientException: svn: Wrapped Java Exception
com.cenqua.fisheye.rep.RepositoryClientException: org.apache.subversion.javahl.ClientException: svn: Wrapped Java Exception
at com.cenqua.fisheye.svn.SvnThrottledClient.executeNoThrottle(SvnThrottledClient.java:192) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient.execute(SvnThrottledClient.java:161) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient.logMessages(SvnThrottledClient.java:143) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Scanner.doChangesetScan(Svn2Scanner.java:337) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Scanner.slurpRepository(Svn2Scanner.java:207) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Scanner.doSlurpTransaction(Svn2Scanner.java:184) [fisheye.jar:?]
at com.cenqua.fisheye.rep.BaseRepositoryScanner.ping(BaseRepositoryScanner.java:73) [fisheye.jar:?]
at com.cenqua.fisheye.rep.BaseRepositoryEngine.doSlurp(BaseRepositoryEngine.java:85) [fisheye.jar:?]
at com.cenqua.fisheye.rep.RepositoryEngine.slurp(RepositoryEngine.java:419) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.IndexingPingRequest.doRequest(IndexingPingRequest.java:28) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.IncrementalPingRequest.doRequest(IncrementalPingRequest.java:30) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.PingRequest$1.run(PingRequest.java:55) [fisheye.jar:?]
at com.cenqua.fisheye.util.NamedExecution.run(NamedExecution.java:27) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.PingRequest.process(PingRequest.java:52) [fisheye.jar:?]
at com.cenqua.fisheye.rep.RepositoryHandle.processPingRequests(RepositoryHandle.java:211) [fisheye.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
Caused by: org.apache.subversion.javahl.ClientException: svn: Wrapped Java Exception
at org.apache.subversion.javahl.SVNClient.logMessages(Native Method) [svn-javahl.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient$2.call(SvnThrottledClient.java:149) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient$2.call(SvnThrottledClient.java:144) [fisheye.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
at com.cenqua.fisheye.svn.SvnTask.access$101(SvnTask.java:13) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnTask$1.run(SvnTask.java:35) [fisheye.jar:?]
at com.cenqua.fisheye.util.NamedExecution.run(NamedExecution.java:27) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnTask.run(SvnTask.java:30) [fisheye.jar:?]
... 3 more
Caused by: com.cenqua.fisheye.svn.SvnThrottledClient$BreakException: Reached 100755 which is beyond the pipeline batch path limit (100000). Breaking out from execution. Please use "fisheye.pipeline.batch.pathlimit" and "fisheye.pipeline.batch.cslimit" properties to tune it if needed.
at com.atlassian.fisheye.svn.Svn2Scanner$2.singleMessage(Svn2Scanner.java:366) [fisheye.jar:?]
at com.cenqua.fisheye.svn.DecoupledLogMessageCallback.singleMessage(DecoupledLogMessageCallback.java:24) [fisheye.jar:?]
... 11 moreCause
The pipelined indexing processes changesets in batches and the default maximum allowed number of unique paths in a batch (summed across changesets) was reached.
As the message indicates, the default limit is 100000 but the number of unique paths being processed in that use case was 100755, causing the indexing failure.
Solution
According to the document JVM system properties - Fisheye pipeline indexing (and as seen in the log error), the fisheye.pipeline.batch.pathlimit property controls this limit, the default value being 100000. A higher value needs to be set for fixing this issue.
This setting needs to be configured in the FISHEYE_OPTSenvironment variable. After configuring this setting, Fisheye needs to be restarted so that this change in the environment variable gets picked, and the repository needs to be re-indexed from scratch.
Detailed steps:
Create / edit the
FISHEYE_OPTSenvironment variable;Make sure that it contains at least this property (as there may be other ones already set):
1FISHEYE_OPTS="-Dfisheye.pipeline.batch.pathlimit=150000"Feel free to use any limit value higher than 100000.
Restart Fisheye;
Monitor the indexing activity on the repository that was facing this issue in order to determine if the new limit set is enough or if it should be increased further.
Was this helpful?