Is it possible to ingest Jira logs to ELK stack?.
Platform Notice: Data Center Only - This article only applies to Atlassian apps on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
There may be requirement where users want to ingest Jira log files like atlassian-jira-security log file for into Elasticsearch/Logstash stack. But log appears unstructured and does not seems to match logging pattern. Thus it is difficult to ingest log files.
Environment
Jira 8.x and later
Diagnosis
Log appender for security log files is set following way. But it could be observed sometime that log files does not exactly follow the conversion pattern.
Log pattern
log4j.appender.securitylog=com.atlassian.jira.logging.JiraHomeAppender
log4j.appender.securitylog.File=atlassian-jira-security.log
log4j.appender.securitylog.MaxFileSize=20480KB
log4j.appender.securitylog.MaxBackupIndex=5
log4j.appender.securitylog.layout=com.atlassian.logging.log4j.NewLineIndentingFilteringPatternLayout
log4j.appender.securitylog.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSSZ} %t %X{jira.username} %X{jira.request.id} %X{jira.request.assession.id} %X{jira.request.ipaddr} %X{jira.request.url} %m%nSample logs
2022-01-09 16:32:12,025+0000 http-nio-8081-exec-16 url: /secure/404.jspa - HttpSession created [lmn456]
2022-01-09 16:32:12,029+0000 http-nio-8080-exec-8 url: /, /secure/MyJiraHome.jspa anonymous 992x1744103x1 - 10.98.5.9 / HttpSession
created [abc123]
2022-01-09 16:32:12,039+0000 http-nio-8080-exec-51 url: /, /secure/MyJiraHome.jspa anonymous 992x1744104x2 - 10.98.5.9 / HttpSessio
n created [xyz987]Cause
There is no contract as such where log files will have a fix format. There is no enforcement to follow any log pattern. The logs which users are observing are complex to parse but there is no issue with logging or it's format. Log files can have any arbitrary string with no specific format.
There is no specific contract for the output format of log files. So log patterns are not very relevant here.
Solution
User would have to use grok patterns as described in Grok Filter Plugin to ingest data as per requirement.
Few customer published all their grok patterns that are used to parse our logs, one such example is Maximizing business value from your log files.
Was this helpful?