Assets unable to delete .lock files during indexing
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Assets is having problems running indexing operations with errors appearing stating that it is "unable to get the index lock" or that it is unable to delete files in the insight cache directory. This may prevent specific nodes from recognizing that updates have been made to Assets objects on other nodes.
Environment
Observed in JSM 5.12 but likely exists in other environments.
Diagnosis
Log messages stating that existing cache lock files cannot be deleted:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
ThreadPoolAsyncTaskExecutor::Thread 122 WARN [i.r.insight.persistence.FileManagerImpl] Failed to delete file: D:\Atlassian\Jira\caches\insight_ehcache_disk\cache-10540370692047528105
org.apache.commons.io.IOExceptionList: 1 exceptions: [java.io.IOException: Unable to delete file: D:\Atlassian\Jira\caches\insight_ehcache_disk\cache-10540370692047528105\.lock]
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:345)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1206)
at io.riada.insight.persistence.FileManagerImpl.quiteDeleteFiles(FileManagerImpl.java:269)
at io.riada.insight.persistence.FileManagerImpl.clearInsightDiskCacheDirectory(FileManagerImpl.java:261)
...
Caused by: java.io.IOException: Unable to delete file: D:\Atlassian\Jira\caches\insight_ehcache_disk\cache-10540370692047528105\.lock
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1425)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:338)
... 45 more
Caused by: java.nio.file.FileSystemException: D:\Atlassian\Jira\caches\insight_ehcache_disk\cache-10540370692047528105\.lock: The process cannot access the file because it is being used by another processIn addition there are log messages stating that indexing cannot begin because the index lock is not available. All attempts to claim the index lock time out.
1
2
3
4
5
6
insight-InsightImportThreadGroup-worker-thread-5 ERROR user [c.a.j.issue.index.DefaultIndexManager] Could not reindex: com.atlassian.jira.issue.util.IssueObjectIssuesIterable (1 items): [ISD-140527]
com.atlassian.jira.issue.index.exception.CannotGetIndexLockException: Can not get index lock.
at com.atlassian.jira.issue.index.DefaultIndexManager.executeWithIndexLock(DefaultIndexManager.java:919)
insight-InsightImportThreadGroup-worker-thread-5 ERROR user [c.a.j.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Wait attempt timed out - waited 30000 millisecondsCause
Another program such as an anti-virus scanner or other security tool is holding these files open to inspect them. Jira cannot complete indexing because of this, potentially keeping Assets from properly updating values from node to node.
Ina Windows environment, it may be possible to determine what program is holding these files open.Run the WindowsResource Monitor program (resmon.exe) and consult the CPU tab. In the "search" field, enter the directory path or the file name for the files being held open. This should tell you which programs are holding these files open.
In a Linux environment, use the lsof command to determine what program is holding open a file.
Solution
Perform the following steps. The first step can be skipped if you do not have any programs that are scanning the directory in question. Note that after deleting the cache files and restarting Jira, it will rebuild the cache on its own automatically.
Whitelist the
caches\insight_ehcache_disk\directory in your antivirus/scanning programShut Jira down
Delete all files in
caches\insight_ehcache_disk\.Restart Jira afterward.
Was this helpful?