• Products
  • Get started
  • Documentation
  • Resources

Integrate Opsgenie with Icinga2

Icinga2 diagram

What does Opsgenie offer to Icinga2 users?

Opsgenie Icinga2 integration plugin utilizes full capabilities of Opsgenie and provides a two-way integration with Icinga2.

Functionality of the integration

  • Icinga2 sends alerts to Opsgenie with detailed information. Opsgenie acts as a dispatcher for Icinga2 alerts, determines the right people to notify based on on-call schedules– notifies via email, phone calls, text messages (SMS) and iPhone & Android push notifications, and escalates alerts until the alert is acknowledged or closed.

  • Opsgenie automatically connects to Icinga2, gets performance data from Graphite for the host/service and attaches it to the alert.

  • Opsgenie posts alert updates to Icinga2, e.g. acknowledging the alert will automatically ack the alert in Icinga2, alert comments are reflected to Icinga2 etc.

Icinga2 alerts

Installation

The steps below describe how to integrate Opsgenie and Icinga2 using Opsgenie Icinga2 integration plugin. Note that you may need to slightly alter these instructions depending on your exact Linux distribution and your Icinga2 configuration.

Prerequisites

Packages provided support the following systems:

  • Red Hat based linux distributions

  • Debian based linux distributions

Installation of OpsGenie Icinga2 Plugin

Download the latest version of OEC

To download the latest version of Icinga2 package, please use readme file of oec-scripts repository.

For Red Hat Based Distributions

  • Run the following command:

Shell

1 rpm -i opsgenie-icinga2-<your_version>.rpm

 

During upgrades, rpm package does not overwrite your existing configurations. It saves the new default configuration file as opsgenie-integration.conf.rpmnew. You can find more information about rpm upgrade config file handling from here.

If you're updating the integration from version 1.., you must update your opsgenie-integration.conf file:

  • Remove icinga.alert_histogram_image_url, icinga.trends_image_url, icinga.command_url properties.

  • Add icinga.api_url (Icinga2 API endpoint of your Icinga server) and icinga.graphite_url properties.


For Debian Based Distributions

  • Run the following command :

Shell

1 dpkg -i opsgenie-icinga2-<your_version>.deb

For Other Distributions

  • Run the following command :

Text

1 unzip opsgenie-icinga2-<your_version>.zip

Add Icinga2 integration in Opsgenie

  1. Go to Opsgenie’s Icinga2 Integration page.

Under the Free and Essentials plans, the tabs under the Team dashboard are limited to Main, Integrations, Members, and Activity Stream.

2. Specify who is notified of Icinga2 alerts using the Responders field. Auto-complete suggestions are provided as you type.

Did you know that you can also add this integration from your team dashboard? 

If you want your team to be the owner of this integration, add it from your team dashboard. Opsgenie will assign the alerts received through this integration to your team only.

To do that;

  1. Go to Teams,

  2. Find your team and select it to go to your team dashboard,

  3. From your team navigation, select Integrations,

  4. Select Add integration.

3. Click Save Integration. An "API Key" is generated for the integration. This key is used by Icinga2 to authenticate with Opsgenie and specify the integration to process Icinga2 alerts.

Icinga2 integration

Opsgenie Plugin Configuration in Icinga2

The plugin uses a golang-executable file (included in the plugin as send2opsgenie) to create, acknowledge, and close alerts in Opsgenie. Icinga2 should be configured to execute this file on events to create, acknowledge, and close alerts in Opsgenie.

Setting the apiKey is required. Other configuration parameters are set to defaults that work with most Icinga2 implementations but may need to be modified as well.

Configuration Parameter

Description

Mandatory to fill

Location

apiKey

Copy the API key from the Icinga2 integration created previously. send2opsgenie uses this key to authenticate to Opsgenie. API key is also used to identify the right integration configuration that should be used to process alerts.

Yes

/home/opsgenie/oec/conf/config.json

baseUrl

If you're using Opsgenie from another domain(eg. EU, sandbox), you should update this configuration.Opsgenie Icinga2 integration, Advanced Settings page.

Optional

/home/opsgenie/oec/conf/config.json

responders

Responders field is used to specify which teams to be notified of Icinga2 alerts. This field is used to set the default responders field value. Modify to route different alerts to different teams in Opsgenie Icinga2 integration via the Advanced Settings page.

Optional

 

tags

Tags field is used to specify the tags of the alert that created in Opsgenie.

Optional

 

icinga_server

icinga_server field is used to identify the Icinga2 server in Opsgenie, and only required when there are multiple Icinga2 servers. This field is used by Opsgenie when sending actions executed by users (acknowledge, close, etc.) back to the Icinga2 servers via OEC

Optional

 

logPath

Specifies the full path of the log file. (Default value is /var/log/opsgenie/send2opsgenie.log)

Optional

 

icinga2opsgenie.http.proxy.enabled

icinga2opsgenie.http.proxy.enabled field is to enable/disable external proxy configuration. The default value is false.

Optional

 

icinga2opsgenie.http.proxy.host

It is the host of the proxy.

Optional

 

icinga2opsgenie.http.proxy.port

It is the port of the proxy.

Optional

 

icinga2opsgenie.http.proxy.scheme

It is the proxy connection protocol. It may be http or https depending on your proxy servers. Its default value is http.

Optional

 

icinga2opsgenie.http.proxy.username

It is the Proxy authentication username.

Optional

 

icinga2opsgenie.http.proxy.password

It is the Proxy authentication password.

Optional

 

There are three ways to configure golang-executable file:

  1. Configuring from conf file: Copy /home/opsgenie/oec/conf/opsgenie-integration.conf.part as /home/opsgenie/oec/conf/opsgenie-integration.conf. Now you can change the configuration parameters as desired in opsgenie-integration.conf. Configuring from conf file overwrites the configurations made in the script.

  2. Configuring by using Golang Flags: Configure by entering flags to the command definition in /etc/icinga2/conf.d/opsgenie.conf file. Use -apiKey flag for the apiKey and -is flag for the icinga_server name. If multiple Icinga2 servers are not in use, it is not necessary to define the Icinga2 server. Using flags overwrites all the other configuration methods mentioned above.


To send additional custom arguments define them as key-value-pairs in the 

argumentsdictionary. Since Icinga2 shuffles the arguments, please use the 

order property to make sure that the custom arguments are placed at the end of the list. Example usage:

"-spd" = "$service.perfdata$"

"customargument1" = {

value = "$customargument1$"

order = 1

}
Parse custom arguments by adding 

{{_payload.customArgName}}

 to wherever is needed on the input fields.
For more information about using raw parameters please visit Dynamic Fields document.

3. Configuring from script: Configure apiKey and icinga_server from send2opsgenie.go script. If this option is used, build the script again and put the new executable to /home/opsgenie/oec/opsgenie-icinga2 directory. Find information about the location of the send2opsgenie.go and how to build a go script in the Source section.

Configure the golang-executable to use a proxy for sending HTTP requests by defining the environment variable HTTP_PROXY=http://host:port

Define Icinga contacts

  1. Copy /home/opsgenie/oec/opsgenie-icinga2/opsgenie.conf as an available Icinga feature:

Shell

1 cp /home/opsgenie/oec/opsgenie-icinga2/opsgenie.conf /etc/icinga2/features-available/opsgenie.conf

2. Enable opsgenie and restart Icinga2

Shell

1 icinga2 feature enable opsgenie

3. Restart Icinga.

Shell

1 systemctl restart icinga2

If everything goes well, alerts are seen in Opsgenie for every notification created in Icinga.

Configure Opsgenie to Icinga2 Integration (Optional)

In order to use this feature "Send Alert Actions To Icinga2" and "Send via OEC" checkboxes should be enabled in Opsgenie Icinga2 Integration.

Use Opsgenie’s OEC and the Icinga2 script to update alerts on Icinga2. This enables deployment of your own scripts/ability to modify the ones provided and execute customized actions on Icinga2.

To be able to execute actions in Icinga2, OEC gets the configuration parameters from the configuration file. The configuration file can be found under /home/opsgenie/oec/conf/config.json.

Configuration parameters

  • api_url – OEC uses this URL to post alert updates to Icinga2, like alert acknowledgement, comments, etc. Replace the "https://localhost:5665" with your Icinga2 server's API endpoint.

  • graphite_url – (Optional) OEC retrieves performance data graphs from Icinga2 using this URL. Replace the "http://localhost:5003" with your Icinga2 server's Graphite endpoint.

  • user – Username to authenticate to Icinga2 API.

  • password – Password to authenticate to Icinga2 API.

  • http.timeout – Timeout duration in msecs to connect to Icinga2 API.

  • expire_acknowledgement_after – Removes acknowledgement after given value (in minutes.) Disabled by default.

  • insecure – Ignore SSL verification while accessing the Icinga2 server's API endpoint.

The package that you downloaded also includes OEC utility which is located under /usr/local/bin and the script that is needed to be run by OEC which is under /home/opsgenie/oec/scripts. After the configuration of OEC is done, you need to run it. In order to learn more about how to run OEC, you can refer to the Running OEC documentation.

Icinga2 alert details

Icinga2 integration package does not support SSL v1.0. If your Icinga2 Server has SSL v1.0, we suggest you to upgrade your SSL server.

FAQ and Troubleshooting

If you're having trouble getting the integration to work, please check if your problem is mentioned below, and follow our advice:

1. Icinga2 alerts are not getting created in Opsgenie:

Run the following test command from the shell. Check if the test alert is created in Opsgenie:

Shell

1 /home/opsgenie/oec/opsgenie-icinga2/send2opsgenie -entityType=host -t=PROBLEM -hs=DOWN -hn=test_host
  • If a "Trace/breakpoint trap" error occurs: It means the send2opsgenie plugin isn't compatible with the server distribution. Follow the "Source and Recompiling send2opsgenie" section below and rebuild send2opsgenie.go according to the specific server environment.

  • If the alert is created in Opsgenie: It means the integration is installed correctly. The problem might be that Icinga2 is not notifying the Opsgenie contact for alerts. Check the Icinga2 alert notifications log.

  • If not: Check the logs at /var/log/opsgenie/send2opsgenie.log. Look for the following errors in the log file:

  • If a "RestException[Could not authenticate.]" error is seen in the logs, it means Opsgenie couldn't identify the api key. Check if the API key correctly is set correctly, as explained in "Opsgenie Plugin Configuration in Icinga" above.

  • If a "Could not execute this action with apiKey of [Icinga2] integration" error is seen in the logs, the wrong integration package may have been downloaded. Make sure the Icinga2 integration package is downloaded, not Icinga or any other.

  • If the problem cannot be identified, set the plugin's log level to debug, try again and send the logs to us with our contact us form.

  • If there is no /var/log/opsgenie/send2opsgenie.log file, or there are no logs in it, check the following:

  • First, make sure the Icinga user has permission to write to /var/log/opsgenie directory. The installation package automatically does this. If there is a problem, execute: chown -R icinga:opsgenie /var/log/opsgenie

  • Now check the Icinga2 server logs at /var/log/icinga2/icinga2.log. See if there are error logs regarding send2opsgenie, and contact us with them.

Setting send2opsgenie plugin's log level to DEBUG:

Change the line icinga2opsgenie.logger=warning to icinga2opsgenie.logger=debug in /home/opsgenie/oec/conf/opsgenie-integration.conf file.

2. The Icinga2 alert is not acknowledged when you ack the alert at Opsgenie:

  • First, check the alert logs.

  • If the "Posted [Acknowledge] action to Icinga2.." action is not in the log, it means Opsgenie didn't send the Acknowledge action to Icinga2. Check the integration configuration, it might not have matched the alert action.

  • If only the "Posted [Acknowledge] action to Icinga2.." is seen in the log and no related log after that, it might mean OEC is having connection problems. Check the logs.

3. Could not open Icinga2 RPM package

If while installing the rpm package you get an error saying that the package is obsolated, you need to use "rpm -i opsgenie-icinga2-1.0.4-rpm-x86-64.rpm --nodeps" instead.

If you get "is already installed" error, use "rpm -i opsgenie-icinga2-1.0.4-rpm-x86-64.rpm --force" instead.

4. Error in perf_data.png generation in icinga2

If you're receiving error while embedding perfData graphite to HTML, probably your Icinga2 version is using perfData.png instead of perf_data.png as naming of graphite. To fix that issue, please update the python script as follows:

Text

1 2 3 4 From: buf += """<div class="img"><img src="perf_data.png"></div>""" To: buf += """<div class="img"><img src="perfData.png"></div>"""

Source and Recompiling send2opsgenie

The source for the executable send2opsgenie is located under /usr/bin/ and send2opsgenie.go is located under //home/opsgenie/oec/conf/icinga2 and is also available at GitHub Opsgenie Integration repository. To change the behavior of the executable, edit send2opsgenie.go and build it using:

Shell

1 go build send2opsgenie.go

For installing go, refer to https://golang.org/doc/install. Note that the executable in the plugin is built for linux/386 systems.

Last modified on Jun 18, 2021
Cached at 9:03 PM on Jul 31, 2021 |

Additional Help

Ask the Community