• Products
  • Get started
  • Documentation
  • Resources

Integrate Opsgenie with Splunk

Splunk logo

Splunk alerts can be used to monitor for and respond to specific events. Alerts use a saved search to look for events in real time or on a schedule. Alerts trigger when search results meet specific conditions. Alert actions can be used to respond when alerts trigger.

What does Opsgenie offers Splunk users?

Opsgenie provides a two-way integration with Splunk. Splunk's Searching and Reporting app lets users search their data, create data models and pivots, save searches and pivots as reports, configure alerts, and create dashboards. Through Opsgenie’s Alerts app, forward Splunk alerts to Opsgenie. With the Splunk Integration, Opsgenie acts as a dispatcher for these alerts, determines the right people to notify based on on-call schedules– notifies via email, text messages (SMS), phone calls, iOS & Android push notifications, and escalates alerts until the alert is acknowledged or closed. Through OEC (Opsgenie Edge Connector) you can forward Opsgenie alerts to Splunk as events for additional indexing and search.

Functionality of the integration

  • Opsgenie has a Splunk-specific alert app to send Splunk alerts to Opsgenie. Opsgenie also has a specific API for Splunk Integration, Splunk sends alerts through Opsgenie Alerts app to Opsgenie and Opsgenie handles the automatic creation of alerts.

  • If Create Splunk Events for Opsgenie Alerts is enabled, alert specific actions (Create Alert, Acknowledge Alert, etc.) will be sent to Splunk as events through OEC.

Add Splunk Integration in Opsgenie

  1. Go to Opsgenie's Splunk Integration page.

Under the Free and Essentials plans, the tabs under the Team dashboard are limited to Main, Integrations, Members, and Activity Stream.

2. Specify who is notified for Splunk alerts using the Teams field. Auto-complete suggestions are provided as you type.

An alternative for Step 1) and Step 2) is to add the integration from the Team Dashboard of the team which will own the integration. To add an integration directly to a team, navigate to the Team Dashboard and open Integrations tab. Click Add Integration and select the integration that you would like to add.

3. Copy the API Key.

4. Click Save Integration.

Splunk integration settings

Configuration in Splunk

  1. In Splunk, install the Opsgenie App from Splunkbase.

Splunk install app

2. After installation, navigate to "Apps" and click Set Up to configure the Opsgenie App.

Splunk Setup App

3. Paste the API Key copied previously in the chapter: Add Splunk Integration in Opsgenie

Splunk Setup App

Updating API Key

If using Splunk Cloud and need to update the API Key you've set in this step, please file a Splunk support case to uninstall the app.

4. Run a search in Splunk to create an Alert.

5. Click Save As and select "Alert" from the dropdown list.

Splunk alert

6. Populate alert title and specify conditions.

7. Click the +Add Actions button to access the dropdown list and select "Opsgenie".

Splunk trigger action
Splunk trigger action

8. Click Save.

Integration via OEC

Use Opsgenie’s OEC and the Splunk script to update alerts on Splunk. This enables deployment of your own scripts/ability to modify the ones provided and execute customized actions on Splunk.

To use the Splunk integration package, follow the steps in the sections below through the end.

Download the latest version of OEC

To download the latest version of Splunk package, please use readme file of oec-scripts repository.

Installation

For Red Hat Based Distributions

  • Run following command : rpm -i opsgenie-splunk-<your_version>.rpm

For Debian Based Distributions

  • Run following command : dpkg -i opsgenie-splunk-<your_version>.deb

For Windows

  • Unzip opsgenie integration zip file which contains OEC package into a directory (C:\opsgenie\oec is the preferred path.) Follow the instructions here to install OEC on Windows.

  • In order to learn more about how to run OEC you can refer to the Running OEC documentation.

To be able to execute actions in Splunk, OEC gets the configuration parameters from the configuration file. The configuration file can be found under /home/opsgenie/oec/conf/config.json and for windows C:\opsgenie\oec\conf\config.json

Configuring Splunk Integration for OEC

  • In order to use OEC utility for your Splunk integration, Send Via OEC in your integration settings.

  • Configurations regarding Splunk are optional if you select send via OEC, you can choose to configure them in your OEC config file.

Splunk OEC

Configuring Splunk Integration for Splunk Cloud

  • If you are using Splunk Cloud, Opsgenie can now deliver events to your Splunk Cloud instance.

  • DO NOT select Send Via OEC and enter your Splunk Cloud Url and Splunk Token in the respective fields. Please enter Splunk Cloud Url along with port.

Splunk OEC configuration

Configuration in Splunk

  1. Log in to Splunk as an administrator.

  2. From the home page, select Add Data.

Splunk add data

3. From bottom, select Monitor.

Splunk monitor

4. From data input options, select HTTP Event Collector and give your new event collector a name.

Splunk HTTP Event Collector

5. Complete rest of the steps without modification unless desired. In final step note the token value given by Splunk.

Splunk token created

6. To enable data input through HTTP, select Settings > Data Input from top bar, then navigate to HTTP Event Collector. Afterwards make sure All Tokens is set to enabled in Global Settings menu.

Splunk edit global settings

Now that collector is configured, Splunk will create events whenever Opsgenie issues alert related actions.

OEC Configuration

To be able to create events in Splunk, OEC gets the configuration parameters from integration settings.

Configuration Parameter

Description

Location

Splunk URL

URL of your Splunk HTTP Event Collector Server including port. For ex: http://<splunk_server>:

/home/opsgenie/oec/conf/config.json

Splunk Token

Token of your Splunk HTTP Event Collector data input

/home/opsgenie/oec/conf/config.json

Verify SSL

The request to your Splunk Server might fail if the SSL certificate verification fails. You can choose to not to verify ssl cert, default is false

/home/opsgenie/oec/conf/config.json

Sample payload sent from Splunk and draggable fields in Opsgenie

The result field of the content below differs according to the fields of the lines that match the search. That's why Opsgenie provides common fields of the result object in available fields. Raw, Index, Serial, Source Type etc. are examples of the common fields.

We also added "Result Object" to the available fields, to make it possible to extract custom fields from the result object.

For example, to put the date_month field of result object to the alert: {{result.date_month}} solves the problem.

Create Alert payload:

JSON

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 { "session_key": "r41vK7psTN9iIp1HQXqgNxTHPz2AW_Ee3ELbdYM4FBqiBbI7L6f82o6f6IENt6Q_Xdq2V4jBSkjkyIfXIm56xbbcFcpWlcJNB0ZUZaezsImsTQ2lGWH26yiZ8l854Or8SPETrWuVgTKVeC", "search_name": "fail", "results_link": "http://Tuba-MacBook-Pro.local:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__fail_at_1464802733_32.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now", "app": "search", "sid": "rt_scheduler__admin__search__fail_at_1464802733_32.0", "configuration": { "api_url": "http://4kmm916oxm9m.runscope.net" }, "server_host": "Tuba-MacBook-Pro.local", "owner": "admin", "results_file": "/Applications/Splunk/var/run/splunk/dispatch/rt_scheduler__admin__search__fail_at_1464802733_32.0/per_result_alert/tmp_0.csv.gz", "server_uri": "https://127.0.0.1:8089", "result": { "date_month": "may", "index": "main", "_indextime": "1464802756", "date_minute": "15", "date_hour": "0", "splunk_server": "Tuba-MacBook-Pro.local", "date_mday": "11", "sourcetype": "secure", "source": "tutorialdata copy 2.zip:./www1/secure.log", "date_second": "2", "_serial": "0", "_sourcetype": "secure", "date_year": "2016", "eventtype": "", "_kv": "1", "timeendpos": "25", "timestartpos": "4", "linecount": "1", "date_zone": "local", "date_wday": "wednesday", "punct": "____::__[]:________...___", "_raw": "Thu May 11 2016 00:15:02 www1 sshd[4747]: Failed password for invalid user jabber from 118.142.68.222 port 3187 ssh2", "_eventtype_color": "", "_confstr": "source::tutorialdata copy 2.zip:./www1/secure.log|host::Tuba-MacBook-Pro.local|secure", "_time": "1462914902", "host": "Tuba-MacBook-Pro.local" } }
Splunk alert fields

Sample payload sent to Splunk

JSON

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 { "alertDetails":{ }, "customerDomain":"opsgenie", "sendViaMarid":false, "alertId":"2ce35c31-a8e1-4be5-a781-110564a45e75-1568102450408", "url":"http://localhost:8088", "type":"Splunk", "alert":{ "alertId":"2ce35c31-a8e1-4be5-a781-110564a45e75-1568102450408" }, "source":null, "token":"4c40855d-a361-4a9b-984f-0ab46c91a35f", "params":{ "alertDetails":{ }, "customerDomain":"opsgenie", "sendViaMarid":false, "alertId":"2ce35c31-a8e1-4be5-a781-110564a45e75-1568102450408", "source":null, "url":"http://localhost:8088", "integrationName":"Splunk", "alert":{ "alertId":"2ce35c31-a8e1-4be5-a781-110564a45e75-1568102450408" }, "integrationId":"c345dce0-50e3-4498-9433-1fddd3f4f1fd", "token":"4c40855d-a361-4a9b-984f-0ab46c91a35f", "integrationType":"Splunk", "action":"Create", "mappedActionV2":{ "extraField":"", "name":"createEvent" }, "type":"Splunk", "customerId":"5035e5cd-4791-4995-9154-037027f8e0b6" }, "action":"Create", "mappedActionV2":{ "extraField":"", "name":"createEvent" }, "integrationId":"c345dce0-50e3-4498-9433-1fddd3f4f1fd", "integrationType":"Splunk", "integrationName":"Splunk", "customerId":"5035e5cd-4791-4995-9154-037027f8e0b6" }

Sample Alert

Splunk sample alert

Contact

Contact us for your questions and support requests.

Last modified on Apr 9, 2021
Cached at 9:02 PM on Apr 10, 2021 |

Additional Help

Ask the Community