Get started with Opsgenie as a user
Learn how to configure your profile, get notifications from Opsgenie and view on-call schedules.
Integrate Opsgenie with Icinga2
Overview
What does the integration offer?
The Icinga2 integration plugin utilizes the full capabilities of Opsgenie and provides bidirectional integration with Icinga2.
How does the integration work?
Icinga2 sends alerts to Opsgenie with detailed information. Opsgenieacts as a dispatcher forIcinga2 alerts, determines the right people to notify based on on-call schedules– notifies via email, phone calls, text messages (SMS), and iPhone & Android push notifications, and escalates alerts until they are acknowledged or closed.
Opsgenie automatically connects to Icinga2, gets performance data from Graphite for the host/service, and attaches it to the alert.
Opsgenie posts alert updates to Icinga2. For example, acknowledging the alert will automatically ack the alert in Icinga2, alert comments are reflected in Icinga2, etc.
Set up the integration
The Icinga2 integration plugin utilizes the full capabilities of Opsgenieand provides bi-directional integration with Icinga2. The steps in the following procedure describe how to integrate Opsgenie and Icinga2 using the Icinga2 integration plugin. Note that slight alteration to these instructions may be necessary depending on the exact Linux distribution and your Icinga2configuration.
Installation prerequisites
The installation packages support the following systems:
RedHat-based Linux distributions
Debian-based Linux distributions
Install the Opsgenie plugin for Icinga 2
Opsgenie Edge Connector (OEC) is a prerequisite for configuring the outgoing authentication of Icinga2 integration. You can combinedly use OEC and Icinga2 scripts to update alerts on Icinga 2. With this setup, you can deploy your script, modify the ones provided, or run customized actions on Icinga2. Download the latest version of the Icinga2 package from this repository.
Instructions for RedHat-based distributions
Run the following command:
rpm -i opsgenie-icinga2-<your_version>.rpm
The rpm package does not overwrite the existing configuration during upgrades. It saves the new default configuration file as integration.conf.rpmnew. Learn more about config file handling for rpm upgrades.
If you're updating the integration from version 1.., update your integration.conf file:
Remove the icinga.alert_histogram_image_url, icinga.trends_image_url, and icinga.command_url properties.
Add the icinga.api_url (Icinga2 API endpoint of your Icinga server) and icinga.graphite_url properties.
Instructions for Debian-based distributions
Run the following command:
dpkg -i opsgenie-icinga2-<your_version>.deb
Instructions for other distributions
Run the following command:
unzip opsgenie-icinga2-<your_version>.zip
Add an Icinga 2 integration
To add an Icinga 2 integration in Opsgenie:
Go to Settings > Integrations.
Select Add integration.
Run a search and select “Icinga 2”.
On the next screen, enter a name for the integration.
Optional: Select a team in Assignee team if you want a specific team to receive alerts from the integration.
Select Continue.
The integration is saved at this point.Expand the Steps to configure the integration section and copy the API key.
You will use this URL while configuring the integration in Icinga 2 later. This key is used by Icinga 2 to authenticate with Opsgenie and specify the integration to process Icinga2 alerts.Select Turn on integration.
The rules you create for the integration will work only if you turn on the integration.
Configure the Opsgenie plugin in Icinga 2
The plugin uses a golang-executable file (included in the plugin as send2opsgenie) to create, acknowledge, and close alerts in Opsgenie. Configure Icinga 2 to execute this file on events to create, acknowledge, and close alerts in Opsgenie. Setting the apiKey is required. Other configuration parameters are set to defaults that work with most Icinga 2 implementations but may also need to be modified. The following table lists the parameters and states if they are mandatory:
Configuration Parameter | Description | Mandatory to fill | Location |
|---|---|---|---|
apiKey | Copy the API key from the Icinga2 integration created previously. send2opsgenie uses this key to authenticate to Opsgenie. API key is also used to identify the right integration configuration that should be used to process alerts. | Yes | /home/opsgenie/oec/conf/config.json |
baseUrl | If you're using Opsgenie from another domain(eg. EU, sandbox), you should update this configuration.Opsgenie Icinga2 integration, Advanced Settings page. | Optional | /home/opsgenie/oec/conf/config.json |
responders | Responders field is used to specify which teams to be notified of Icinga2 alerts. This field is used to set the default responders field value. Modify to route different alerts to different teams in Opsgenie Icinga2 integration via the Advanced Settings page. | Optional |
|
tags | Tags field is used to specify the tags of the alert that created in Opsgenie. | Optional |
|
icinga_server | icinga_server field is used to identify the Icinga2 server in Opsgenie, and only required when there are multiple Icinga2 servers. This field is used by Opsgenie when sending actions executed by users (acknowledge, close, etc.) back to the Icinga2 servers via OEC | Optional |
|
logPath | Specifies the full path of the log file. (Default value is /var/log/opsgenie/send2opsgenie.log) | Optional |
|
icinga2opsgenie.http.proxy.enabled | icinga2opsgenie.http.proxy.enabled field is to enable/disable external proxy configuration. The default value is false. | Optional |
|
icinga2opsgenie.http.proxy.host | It is the host of the proxy. | Optional |
|
icinga2opsgenie.http.proxy.port | It is the port of the proxy. | Optional |
|
icinga2opsgenie.http.proxy.scheme | It is the proxy connection protocol. It may be http or https depending on your proxy servers. Its default value is http. | Optional |
|
icinga2opsgenie.http.proxy.username | It is the Proxy authentication username. | Optional |
|
icinga2opsgenie.http.proxy.password | It is the Proxy authentication password. | Optional |
|
Configure the golang-executable file in any of the following three methods:
Method 1: Configure from conf file
Configure from the /home/opsgenie/oec/conf/integration.conf file. This overwrites any configuration you previously made in the script.
Method 2: Configure by using Golang flags
Configure by entering flags into the command definition in the /etc/icinga2/features-available/opsgenie.conf file. Use -apiKey flag for your apiKey and -is flag for the icinga_server name. If no multiple Icinga 2 servers are in use, you don't have to define the Icinga 2 server. Using flags overwrites all the other configuration methods mentioned earlier.
To send additional custom arguments, define them as key-value pairs in the argumentsdictionary. Since Icinga 2 shuffles the arguments, use the order property to ensure the custom arguments are placed at the end of the list. Example:
1
2
3
4
5
6
7
8
9
"-spd" = "$service.perfdata$"
"customargument1" = {
value = "$customargument1$"
order = 1
}Parse custom arguments by adding {{_payload.customArgName}} to the input fields as needed. Learn more about using raw parameters.
Method 3: Configure from script
Configure apiKey and icinga_server from send2opsgenie.go script. Build the script again and add the new executable to the /home/opsgenie/oec/scripts directory. Find information about the location of the send2opsgenie.go and how to build a go script in the Source section.
Configure the golang-executable to use a proxy for sending HTTP requests by defining the HTTP_PROXY=http://host:port environment variable.
Define Icinga contacts
1. Copy /home/opsgenie/oec/conf/opsgenie.conf as an available Icinga feature:
Shell: cp /home/opsgenie/oec/conf/opsgenie.conf /etc/icinga2/features-available/opsgenie.conf
2. Enable opsgenie and restart Icinga 2:
Shell: icinga2 feature enable opsgenie
3. Restart Icinga:
Shell: systemctl restart icinga2
If everything goes well, alerts are seen in Opsgenie for every notification created in Icinga 2.
Configure Opsgenie to update Icinga 2
This is an optional step.
Select the Send Alert Actions To Icinga 2 checkbox on the integration configuration page. You can combinedly use OEC and Icinga 2 scripts to update alerts on Icinga 2. With this setup, you can deploy your own script, modify the ones provided, or run customized actions on Icinga 2.
To run actions in Icinga 2, OEC gets the configuration parameters from the configuration file config.json (found at /home/opsgenie/oec/conf/oec-config.json).
Configuration parameters
api_url: OEC uses this URL to post alert updates to Icinga 2, such as alert acknowledgment, comments, etc. Replace the "https://localhost:5665" with the API endpoint of your Icinga 2 server.
graphite_url (optional): OEC retrieves performance data graphs from Icinga 2 using this URL. Replace the "http://localhost:5003" with the Graphite endpoint of your Icinga 2 server.
user: The username to authenticate to Icinga 2 API
password: The password to authenticate to Icinga 2 API.
http.timeout: The timeout duration in milli secs to connect to Icinga 2 API.
expire_acknowledgement_after: Removes acknowledgment after the specified time (in minutes.) Disabled by default.
insecure: Ignore SSL verification while accessing the API endpoint of your Icinga 2 server.
The downloaded package includes the OEC utility (found in /usr/local/bin) and the script that OEC needs to run (found in /home/opsgenie/oec/scripts). Be sure to run OEC after configuring it. Learn more about running OEC.
The Icinga 2 integration package does not support SSL v1.0. If your Icinga 2 server has SSL v1.0, upgrade your SSL server.
Source for and recompiling send2opsgenie
The source for the executable send2opsgenie is found in /usr/bin/ and send2opsgenie.go, in /home/opsgenie/oec/scripts, respectively, and is also available in this repository. To change the behavior of the executable, edit send2opsgenie.go and build it by using the following command: go build send2opsgenie.go
For installing go, refer to http://golang.org/doc/install. Note that the executable in the plugin is built for linux/386 systems.
FAQ and troubleshooting
If the integration is not working, review this section and follow the prescribed guidelines.
Icinga alerts are not getting created in Opsgenie
Run the following test command from the shell and check if a test alert is created in Opsgenie:
1
/home/opsgenie/oec/scripts/send2opsgenie -entityType=host -t=PROBLEM -hs=DOWN -hn=test_hostIf you get a "Trace/breakpoint trap" error, the send2opsgenie plugin isn't compatible with the server distribution. Rebuild send2opsgenie.go according to the specific server environment as described in the “Source for and recompiling send2opsgenie” section in this article.
If the alert is created in Opsgenie, the integration is configured correctly. Icinga 2 is probably not notifying the Opsgenie contact for alerts. Check your Icinga 2 alert notifications log.
If the alert is not created in Opsgenie, check the logs at /var/log/oec/send2opsgenie.log.
Look for the following errors in the log file:If you see "RestException[Could not authenticate.]" in the logs, Opsgenie couldn't identify the API key. Check if the API key is set correctly per the steps outlined in the “Configure the Opsgenie plugin in Icinga 2” section of this article.
If a "Could not execute this action with apiKey of [Icinga2] integration" error is seen in the logs, the wrong integration package may have been downloaded. Make sure the downloaded Icinga 2 integration package is correct.
If unsure of the problem, set the plugin's log level to debug and try again. Contact us and share the logs.
If there is no /var/log/oec/send2opsgenie.log file or there are no logs in it, check the following:
Check if the Icinga user has permission to write to /var/log/oec directory. The installation package should automatically do this for you. If you face issues, run the following command:
chown -R icinga:opsgenie /var/log/oecCheck the Icinga 2 server logs at /var/log/icinga2/icinga2.log. See if there are error logs regarding send2opsgenie. Contact us with the logs as needed.
Set the send2opsgenie plugin's log level to DEBUG
Set the send2opsgenie plugin's log level to DEBUG. Open the /home/opsgenie/oec/conf/integration.conf file and change the line send2opsgenie.logger=warning to icinga2opsgenie.logger=debug.
The Icinga 2 alert is not acknowledged when the alert is acknowledged in Opsgenie
Check the alert logs.
If "Posted [Acknowledge] action to Icinga 2.." is not present in the log, Opsgenie didn't send the Acknowledge action to Icinga 2. Check the integration configuration, it might not have a matching alert action.
If only the "Posted [Acknowledge] action to Icinga 2.." log occurs followed by no related logs, it might mean OEC is having connection problems. Check the logs.
Could not open Icinga RPM package
If you figure out while installing the rpm package that the package is obsolete, use rpm -i opsgenie-icinga2-1.0.4-rpm-x86-64.rpm --nodeps instead.
If you get "is already installed" error, use rpm -i opsgenie-icinga2-1.0.4-rpm-x86-64.rpm --force instead.
Error in perf_data.png generation in icinga 2
If you're receiving an error while embedding perfData graphite into HTML, your Icinga 2 version is using perfData.png instead of perf_data.png for the graphite name. To fix the issue, update the python script as follows:
1
2
3
4
From:
buf += """<div class="img"><img src="perf_data.png"></div>"""
To:
buf += """<div class="img"><img src="perfData.png"></div>"""Was this helpful?