Integrate Opsgenie with Splunk

Splunk alerts can be used to monitor for and respond to specific events. Alerts use a saved search to look for events in real time or on a schedule. Alerts trigger when search results meet specific conditions. Alert actions can be used to respond when alerts trigger.

What does Opsgenie offers Splunk users?

Opsgenie provides a two-way integration with Splunk. Splunk's Searching and Reporting app lets users search their data, create data models and pivots, save searches and pivots as reports, configure alerts, and create dashboards. Through Opsgenie’s Alerts app, forward Splunk alerts to Opsgenie. With the Splunk Integration, Opsgenie acts as a dispatcher for these alerts, determines the right people to notify based on on-call schedules– notifies via email, text messages (SMS), phone calls, iOS & Android push notifications, and escalates alerts until the alert is acknowledged or closed. Through OEC (Opsgenie Edge Connector) you can forward Opsgenie alerts to Splunk as events for additional indexing and search.

Functionality of the integration

• Opsgenie has a Splunk-specific alert app to send Splunk alerts to Opsgenie. Opsgenie also has a specific API for Splunk Integration, Splunk sends alerts through Opsgenie Alerts app to Opsgenie and Opsgenie handles the automatic creation of alerts.

• If Create Splunk Events for Opsgenie Alerts is enabled, alert specific actions (Create Alert, Acknowledge Alert, etc.) will be sent to Splunk as events through OEC.

Add Splunk Integration in Opsgenie

1. Go to Settings > Integrations. Search for Splunk and select Add.

2. Specify who is notified for Splunk alerts using the Responders field. Auto-complete suggestions are provided as you type.

3. Copy the API Key.

4. Select Save Integration.

Configuration in Splunk

1. In Splunk, install the Opsgenie App from Splunkbase.

2. After installation, navigate to "Apps" and click Set Up to configure the Opsgenie App.

3. Paste the API Key copied previously in the chapter: Add Splunk Integration in Opsgenie

4. Run a search in Splunk to create an Alert.

5. Click Save As and select "Alert" from the dropdown list.

6. Populate alert title and specify conditions.

7. Click the +Add Actions button to access the dropdown list and select "Opsgenie".

8. Click Save.

Integration via OEC

Use Opsgenie’s OEC and the Splunk script to update alerts on Splunk. This enables deployment of your own scripts/ability to modify the ones provided and execute customized actions on Splunk.

To use the Splunk integration package, follow the steps in the sections below through the end.

Installation

For Red Hat Based Distributions

• Run following command : rpm -i opsgenie-splunk-<your_version>.rpm

For Debian Based Distributions

• Run following command : dpkg -i opsgenie-splunk-<your_version>.deb

For Windows

• Unzip opsgenie integration zip file which contains OEC package into a directory (C:\opsgenie\oec is the preferred path.) Follow the instructions here to install OEC on Windows.

• In order to learn more about how to run OEC you can refer to the Running OEC documentation.

To be able to execute actions in Splunk, OEC gets the configuration parameters from the configuration file. The configuration file can be found under /home/opsgenie/oec/conf/config.json and for windows C:\opsgenie\oec\conf\config.json

Configuring Splunk Integration for OEC

• In order to use OEC utility for your Splunk integration, Send Via OEC in your integration settings.

• Configurations regarding Splunk are optional if you select send via OEC, you can choose to configure them in your OEC config file.

Configuring Splunk Integration for Splunk Cloud

• If you are using Splunk Cloud, Opsgenie can now deliver events to your Splunk Cloud instance.

• DO NOT select Send Via OEC and enter your Splunk Cloud Url and Splunk Token in the respective fields. Please enter Splunk Cloud Url along with port.

Configuration in Splunk

1. Log in to Splunk as an administrator.

2. From the home page, select Add Data.

3. From bottom, select Monitor.

4. From data input options, select HTTP Event Collector and give your new event collector a name.

5. Complete rest of the steps without modification unless desired. In final step note the token value given by Splunk.

6. To enable data input through HTTP, select Settings > Data Input from top bar, then navigate to HTTP Event Collector. Afterwards make sure All Tokens is set to enabled in Global Settings menu.

Now that collector is configured, Splunk will create events whenever Opsgenie issues alert related actions.

OEC Configuration

• In order to learn more about how to run OEC you can refer to the Running OEC documentation.

To be able to create events in Splunk, OEC gets the configuration parameters from integration settings.

Sample payload sent from Splunk and draggable fields in Opsgenie

The result field of the content below differs according to the fields of the lines that match the search. That's why Opsgenie provides common fields of the result object in available fields. Raw, Index, Serial, Source Type etc. are examples of the common fields.

We also added "Result Object" to the available fields, to make it possible to extract custom fields from the result object.

For example, to put the date_month field of result object to the alert: {{result.date_month}} solves the problem.

Create Alert payload:

JSON

Sample payload sent to Splunk

JSON