Get started with Opsgenie as a user
Learn how to configure your profile, get notifications from Opsgenie and view on-call schedules.
Integrate Opsgenie with Icinga
Opsgenie’s Icinga plugin supports a bi-directional integration with Icinga. The integration leverages Opsgenie’s Icinga-specific executable and OEC utility to automatically create rich alerts (status, alert histogram, trends, etc.) and synchronizes alert status between Icinga and Opsgenie.
Opsgenie Icinga Plugin Integration
Opsgenie Icinga integration plugin utilizes full capabilities of Opsgenie and provides bi-directional integration with Icinga. Integration leverages Opsgenie’s Icinga-specific executable and OEC utility to automatically create rich alerts (status, alert histogram, trends, etc.) and synchronizes alert status between Icinga and Opsgenie.
Installation
The steps below describe how to integrate Opsgenie and Icinga using Opsgenie Icinga integration plugin. Note that slight alterations may be needed for these instructions depending on your exact Linux distribution and your Icinga configuration.
If you're using Lamp based Icinga plugin, backup all existing configurations. Uninstall the old plugin, then install the new one.
Prerequisites
Packages provided support the following systems:
Red Hat based linux distributions
Debian based linux distributions
Installation of Opsgenie Icinga Plugin
Download the latest version of OEC
To download the latest version of Icinga package, please use readme file of oec-scripts repository.
For Red Hat Based Distributions
Run the following command :
Shell
1
rpm -i opsgenie-icinga-<your_version>.rpm
During upgrades, rpm package does not overwrite your existing configurations. It saves the new default configuration file as opsgenie-integration.conf.rpmnew. Find more information about rpm upgrades config file handling from here.
If you want to update from version 201X-XX-XX to 2.X.X, you must add --force parameter. E.g.:
rpm -U --force opsgenie-integration-<your_version>.rpm
We suggest that you backup existing configuration files before update!
For Debian Based Distributions
Run the following command :
Shell
1
dpkg -i opsgenie-icinga-<your_version>.deb
Add Icinga integration to Opsgenie
You can add this integration from your team dashboard
If you're using Opsgenie's Free or Essentials plan, or if you’re using Opsgenie with Jira Service Management's Standard plan, you can only add this integration from your team dashboard as the Integrations page under Settings is not available in your plan.
Adding the integration from your team dashboard will make your team the owner of the integration. This means Opsgenie will assign the alerts received through this integration to your team only.
To do that,
Go to your team’s dashboard from Teams,
Select Integrations, and select Add integration.
Follow the rest of the steps to complete the integration.
Go to Settings > Integrations. Search for Icinga and select Add.
Specify who is notified of Icinga alerts using the Responders field. Auto-complete suggestions are provided as you type.
Copy the API key generated for the integration. This key is used by Icinga to authenticate with Opsgenie and specify the integration to be used to process Icinga alerts.
Select Save Integration.
Opsgenie Plugin Configuration in Icinga
The plugin uses a golang-executable file (included in the plugin as send2opsgenie) to create, acknowledge, and close alerts in Opsgenie. Configure Icinga to execute this file on events to create, acknowledge, and close alerts in Opsgenie.
Configuration Parameter | Description | Mandatory to fill | Location |
|---|---|---|---|
apiKey | Copy the API key from the Icinga integration you've created above. send2opsgenie uses this key to authenticate to Opsgenie. API key is also used to identify the right integration configuration that should be used to process alerts. | Yes | /home/opsgenie/oec/conf/config.json |
baseUrl | If you're using Opsgenie from another domain(eg. EU, sandbox), you should update this configuration.Opsgenie Icinga integration, Advanced Settings page. | Optional | /home/opsgenie/oec/conf/config.json |
responders | Responders field is used to specify which teams should be notified for the Icinga alerts. This field is used to set the default teams field value. It can be modified to route different alerts to different teams in Opsgenie Icinga integration, Advanced Settings page. | Optional | /home/opsgenie/oec/conf/opsgenie-integration.conf |
tags | Tags field is used to specify the tags of the alert that created in Opsgenie. | Optional | /home/opsgenie/oec/conf/opsgenie-integration.conf |
icinga_server | icinga_server field is used to identify the Icinga server in Opsgenie, and only required when there are multiple Icinga servers. This field is used by Opsgenie when sending actions executed by users (acknowledge, close, etc.) back to your Icinga servers via OEC | Optional | /home/opsgenie/oec/conf/opsgenie-integration.conf |
logPath | Specifies the full path of the log file. (Default value is /var/log/opsgenie/isend2opsgenie.log) | Optional | /home/opsgenie/oec/conf/opsgenie-integration.conf |
icinga2opsgenie.http.proxy.enabled | icinga2opsgenie.http.proxy.enabled field is to enable/disable external proxy configuration. The default value is false. | Optional | /home/opsgenie/oec/conf/opsgenie-integration.conf |
icinga2opsgenie.http.proxy.host | It is the host of the proxy. | Optional | /home/opsgenie/oec/conf/opsgenie-integration.conf |
icinga2opsgenie.http.proxy.port | It is the port of the proxy. | Optional | /home/opsgenie/oec/conf/opsgenie-integration.conf |
icinga2opsgenie.http.proxy.scheme | It is the proxy connection protocol. It may be http or https depending on your proxy servers. Its default value is http. | Optional | /home/opsgenie/oec/conf/opsgenie-integration.conf |
icinga2opsgenie.http.proxy.username | It is the Proxy authentication username. | Optional | /home/opsgenie/oec/conf/opsgenie-integration.conf |
icinga2opsgenie.http.proxy.password | It is the Proxy authentication password. | Optional | /home/opsgenie/oec/conf/opsgenie-integration.conf |
There are three ways to configure golang-executable file:
Configuring from conf file: Configure from /home/opsgenie/oec/conf/opsgenie-integration.conf file. Configuring from conf file will overwrite the configurations made in the script.
Configuring by using Golang Flags: Configure by entering flags to command in the opsgenie.cfg file. Use -apiKey flag for your apiKey and -is flag for your icinga_server name. If multiple Icinga servers aren't being used, the Icinga server does not need to be defined. Using flags overwrites all the other configuration methods mentioned above.
To send additional custom arguments, add them after the flags as: customArgName1 customArgValue1 customArgName2 customArgValue2
Parse custom arguments by adding
{{_payload.customArgName}}
to wherever is needed in the input fields.
For more information about using raw parameters please visit Dynamic Fields document.
3. Configuring from script: Configure apiKey and icinga_server from send2opsgenie.go script. If this option is used, build the script again and put the new executable to /home/opsgenie/oec/opsgenie-icinga directory. Find information about the location of the send2opsgenie.go and how to build a go script in the Source section.
Configure the golang-executable to use a proxy for sending HTTP requests by defining the environment variable
HTTP_PROXY=http://host:port
Define Icinga contacts
Copy /home/opsgenie/oec/opsgenie-icinga/opsgenie.cfg config file, (configures a contact and its host and service notification commands) to /usr/local/icinga/etc/objects directory.
Shell
1
cp /home/opsgenie/oec/opsgenie-icinga/opsgenie.cfg /usr/local/icinga/etc/objects
2. Add the following line to the main Icinga configuration file (ICINGA_HOME/etc/icinga.cfg)
Shell
1
2
3
...
cfg_file=/usr/local/icinga/etc/objects/opsgenie.cfg
...
3. Add the contact "opsgenie" to the Icinga configuration’s main contact group in ICINGA_HOME/etc/objects/contacts.cfg file. If using the default configuration, contacts.cfg, add "opsgenie" user to the "admins" contact group.
4. Restart Icinga.
If everything goes well, alerts are seen in Opsgenie for every notification created in Icinga.
Configure Opsgenie to Icinga Integration (Optional)
In order to use this feature "Send Alert Actions To Icinga" and "Send via OEC" checkboxes should be enabled in Opsgenie Icinga Integration.
Use Opsgenie’s OEC and the Icinga script to update alerts on Icinga. This enables deployment of your own scripts/ability to modify the ones provided and execute customized actions on Icinga.
To be able to execute actions in Icinga, OEC gets the configuration parameters from the configuration file. The configuration file can be found under /home/opsgenie/oec/conf/config.json.
Configuration parameters
alert_histogram_image_url – OEC retrieves histogram images from Icinga using this URL. Localhost should be replaced with the Icinga server address.
trends_image_url – OEC retrieves trends images from Icinga using this URL. Localhost should be replaced with the Icinga server address.
command_url – URL to update Icinga alerts when alerts get acknowledged, commented, etc.
user – Username to authenticate Icinga web server to get Icinga histogram and trends images.
password – Password to authenticate Icinga web server to get Icinga histogram and trends images.
The package that you downloaded also includes OEC utility which is located under /usr/local/bin and the script that is needed to be run by OEC which is under /home/opsgenie/oec/scripts. After the configuration of OEC is done, you need to run it. In order to learn more about how to run OEC, you can refer to the Running OEC documentation.
Icinga integration package does not support SSL v1.0. If your Icinga Server has SSL v1.0, we suggest you to upgrade your SSL server.
FAQ and Troubleshooting
If the integration is not working, please check if the problem is mentioned below, and follow our advice:
1. Icinga alerts are not getting created in Opsgenie:
Run the following test command from the shell. Check if the test alert is created in Opsgenie:
Shell
1
/home/opsgenie/oec/opsgenie-icinga/send2opsgenie -entityType=host -t=PROBLEM -hs=DOWN -hn=test_host
If a "Trace/breakpoint trap" error occurs: It means the send2opsgenie plugin being used isn't compatible with your server distribution. Follow the "Source and Recompiling send2opsgenie" section below and rebuild your send2opsgenie.go according to your specific server environment.
If the alert is created in Opsgenie: It means the integration is installed correctly. The problem might be that Icinga is not notifying the Opsgenie contact for alerts. Check the Icinga alert notifications log.
If not: Check the logs at /var/log/opsgenie/send2opsgenie.log. Look for the following errors in the log file:
If the error "RestException[Could not authenticate.]" is in the logs, it means Opsgenie couldn't identify the API key. Check if the API key is set correctly, as explained in "Opsgenie Plugin Configuration in Icinga" above.
If the error "Could not execute this action with apiKey of [Icinga] integration" is in the logs, make sure that the correct Icinga integration package was downloaded and used, not Icinga 2 or any other.
If unsure of the problem, set the plugin's log level to debug, try again and send the logs to us with out contact us form, so we can problem solve on our end and attempt to find a solution for you.
If there is no /var/log/opsgenie/send2opsgenie.log file, or there are no logs in it, check the following:
First, make sure the Icinga user haspermission to write to /var/log/opsgenie directory. The installation package automatically does this for you. If an error occurs, execute: chown -R icinga:opsgenie /var/log/opsgenie
Check the Icinga server logs at /usr/local/icinga/var/icinga.log. See if there are error logs regarding send2opsgenie, and contact us with them.
Setting send2opsgenie plugin's log level to DEBUG:
Change the line send2opsgenie.logger=warning to icinga2opsgenie.logger=debug in /etc/opsgenie/conf/opsgenie-integration.conf file.
2. The Icinga alert is not acknowledged when the alert is acked in Opsgenie:
First, check the alert logs.
If the "Posted [Acknowledge] action to Icinga.." action is not in the log, it means Opsgenie didn't send the Acknowledge action to Icinga. Check the integration configuration, as it might not have matched the alert action.
If the "Posted [Acknowledge] action to Icinga.." action is not in the log and no related log after that, it might mean OEC is having connection problems. Check the logs.
3. Could not open Icinga RPM package
If while installing the rpm package you get an error saying that the package is obsolated, you need to use "rpm -i opsgenie-icinga-1.0.4-rpm-x86-64.rpm --nodeps" instead.
If you get "is already installed" error, use "rpm -i opsgenie-icinga-1.0.4-rpm-x86-64.rpm --force" instead.
Source and Recompiling send2opsgenie
The source for the executable send2opsgenie is located under /usr/bin/ and send2opsgenie.go is located under /etc/opsgenie/ and is also available at GitHub Opsgenie Integration repository. To change the behavior of the executable, edit send2opsgenie.go and build it using:
Shell
1
go build send2opsgenie.go
For installing go, refer to https://golang.org/doc/install. Note that the executable in the plugin is built for linux/386 systems.
Was this helpful?