Integrate Opsgenie with ITRS OP5 Monitor

概要

What does the integration offer?

Opsgenie provides a powerful bidirectional integration with OP5. When a new alert - host or service - is created in OP5, a corresponding Opsgenie alert is automatically created, containing rich information about the alert. Opsgenie provides detailed notifications with on-call rotations, scheduling features, and alert escalations. Users select any of the alert actions of Opsgenie to map to the Acknowledge action of OP5.

How does the integration work?

OP5 to Opsgenie

• When a host or service alert is created on OP5, an alert is created in Opsgenie.

• When a host or service alert is closed on OP5, the corresponding alert is closed in Opsgenie.

• When a host or service alert is acknowledged on OP5, the corresponding alert is acknowledged in Opsgenie.

Opsgenie to OP5

If Send alert updates back to OP5 is selected, actions that are run on OP5 alerts are sent to OP5 as acknowledgment actions.

Set up the integration

OP5 is a bidirectional integration. Setting it up involves the following steps:

• Add an OP5 integration in Opsgenie

• Map alert actions

• Configure the integration in OP5

Install the Opsgenie plugin for OP5

Opsgenie Edge Connector (abbreviated as OEC) is a prerequisite for configuring the outgoing authentication of OP5 integration. Download the latest version of the OP5 package from this repository.

For OP5 Monitor's Red Hat-based distributions

Either share the OP5 file to the remote OP5 monitor or copy the link on the opening page.

• 次のコマンドを実行します。
  wget https://linkTo.the.OP5Package

• 次のコマンドを実行します。
  rpm -i opsgenie-op5-<your_version>.rpm

Before rpm upgrades, back up your configuration files.

During upgrades, the rpm package does not overwrite the existing configuration. It saves the new default configuration file as integration.conf.rpmnew. To update from version 201X-XX-XX to 2.X.X, add the --force parameter. For example: rpm -U --force opsgenie-integration-<your_version>.rpm

Learn more about rpm upgrade config file handling.

Add an OP5 integration

1. Go to Settings > Integrations.

2. Select Add integration.

3. Run a search and select “OP5”.

4. 次の画面で、統合の名前を入力します。

5. オプション: 特定のチームが統合からのアラートを受信するようにする場合は、[Assignee team (担当者チーム)] のチームを選択します。

6. [続行] を選択します。
   この時点で、統合が保存されます。

7. Expand the Steps to configure the integration section and copy the integration API key.
   You will use this key while configuring the integration in OP5 later.

8. [統合をオンにする] を選択します。
   統合のために作成したルールは、統合をオンにした場合にのみ機能します。

Configure OP5 package in OP5 Monitor

The plugin uses a golang-executable file (included in the plugin as send2opsgenie) to create, acknowledge, and close alerts in Opsgenie. Configure OP5 to execute this file on events to create, acknowledge, and close alerts in Opsgenie. Setting the apiKey is mandatory. Other configuration parameters are set to defaults that work with OP5.

The following table lists the parameters and their locations:

Configure the golang-executable file in any of the following three methods:

Method 1: Configure from conf file

Configure from the /home/opsgenie/oec/conf/integration.conf file. This overwrites any configuration you previously made in the script.

Method 2: Configure by using Golang flags

Configure by entering flags into the command in the opsgenie.cfg file. Use -apiKey flag for your apiKey and -ns flag for nagios_server name. If multiple OP5 servers are not in use, there is no need to define the nagios server. Using flags overwrites all the other configuration methods discussed earlier in the article.
Configure the apiKey from the cfg file as follows:

apiKey を cfg ファイルに追加すると、config.json ファイル内の apiKey が上書きされます。

To send additional custom arguments, add them after the flags. For example, customArgName1 customArgValue1 customArgName2 customArgValue2. Parse custom arguments by adding {{_payload.customArgName}} to wherever is needed in the input fields. To learn more about using raw parameters, see dynamic fields.

Method 3: Configure from script

Configure apiKey and nagios_server from send2opsgenie.go script. To use this option, build the script again and put the new executable to /usr/bin directory. Find information about the location of the send2opsgenie.go and how to build a go script in the "Source for send2opsgenie and recompiling" section.

Opsgenie を連絡先として定義する

1. Log in to your OP5 monitor.

2. Go to the /opt/monitor/etc directory to find nagios.cfg.
   cd /opt/monitor/etc

3. Add the following line to the main Nagios configuration file (nagios.cfg).

   1 2 3 ... cfg_file=opsgenie.cfg ...

4. Reboot the monitor to see the "opsgenie" contact in the contact list.

5. Add the contact "opsgenie" to the OP5 Monitor’s main contact group.

If everything goes well, alerts are created in Opsgenie for every notification created in OP5.

Map alert actions

You can define mappings between Opsgenie actions and OP5 actions (also when the source of the alert is OP5), which requires additional authentication for your OP5 account.

• Username: The username of your OP5 account

• Password: The password of your OP5 account

• OP5 Monitor URL: This is the domain name of your OP5 Monitor. For example: https://19.167.1.143

OP5 によって作成されたアラートの場合

In the Send alert updates back to OP5 section, map Opsgenie actions to OP5 actions when the source of the alert is OP5 (when the alert is created by OP5 integration itself). Map different Opsgenie actions to different OP5 actions. For example, acknowledge the alert in OP5, when the alert is acknowledged from OP5. To do this, define If alert is acknowledged in Opsgenie, acknowledge in OP5 mapping in Send alert updates back to OP5 section. 

Configure Opsgenie to update OP5

このステップはオプションです。

Use Opsgenie’s OEC and the OP5 script to update alerts on OP5. This enables deployment of your own scripts/ability to modify the ones provided and execute customized actions on OP5. To execute actions in OP5, OEC gets the configuration parameters from the configuration file. The configuration file is found under /home/opsgenie/oec/conf/oec-config.json.

設定パラメーター

• url – OEC は、この URL を使用して構築されたエンドポイントにアラート アクションを投稿します。

• username – OEC uses your OP5 account's username to authenticate.

• password – OEC uses your OP5 account's password to authenticate.

The package that you downloaded also includes OEC utility which is located under /usr/local/bin and the script that is needed to be run by OEC which is under /home/opsgenie/oec/scripts. After the configuration of OEC is done, be sure to run it. Learn more about running OEC.

Source for and recompiling send2opsgenie

The source for the executable send2opsgenie is found in /usr/bin/ and send2opsgenie.go, in /home/opsgenie/oec/scripts respectively and is also available in this repository. To change the behavior of the executable, edit send2opsgenie.go and build it by using the following command: go build send2opsgenie.go

For installing go, refer to http://golang.org/doc/install. Note that the executable in the plugin is built for linux/amd64 systems.

よくある質問とトラブルシューティング

If the integration is not working, review this section and follow the guidance prescribed:

1. OP5 alerts are not getting created in Opsgenie

シェルから次のテスト コマンドを実行します。Opsgenie でアラートが作成されているかどうかを確認します。

/home/opsgenie/oec/scripts/send2opsgenie -entityType=host -t=PROBLEM -hs=DOWN -hn=test_host

If "Trace/breakpoint trap" error occurs: It means the send2opsgenie plugin isn't compatible with the server distribution. Follow the "Source for and recompiling send2opsgenie" section on this page and rebuild send2opsgenie.go according to the specific server environment.

If the alert is created in Opsgenie: It means the integration is configured correctly. The problem might be that OP5 is not notifying the Opsgenie contact for alerts. Check the OP5 alert notifications log.

If not: Check the log files under /var/log/oec. Look for the following errors in the log file:

• If a "RestException[Could not authenticate.]" error appears in the logs, it means Opsgenie couldn't identify your API key. Check if the API key is set correctly, as explained in "Configure OP5 Package in OP5 Monitor" on this page.

• If "Could not execute this action with apiKey of [OP5] integration" appears in the logs, the wrong integration package may have been downloaded. Make sure to download the OP5 integration package.

• If unsure of the problem, set the plugin's log level to debug, try again, and contact us and send the logs. If there is no log files under /var/log/oec file, or there are no logs in it, check the following:

  • First, make sure the nagios user has permission to write to /var/log/oec directory. The installation package automatically does this for you. If you encounter problems, execute chown -R nagios:nagios:opsgenie /var/log/oec.

  • Now check the OP5 server logs under /var/log/oec. See if there are error logs regarding send2opsgenie, and contact us with them.

Set log level of send2opsgenie plugin to DEBUG

Change the line send2opsgenie.logger=warning to nagios2opsgenie.logger = debug in /home/opsgenie/oec/conf/integration.conf file.

Configure Opsgenie to update Zenoss

Set the send2opsgenie plugin's log level to DEBUG. Open the /home/opsgenie/oec/conf/integration.conf file and change the line zenoss2opsgenie.logger=warning to zenoss2opsgenie.logger=debug.

2. The OP5 alert is not acknowledged when acknowledged in Opsgenie

1. まず、アラート ログを確認します。

• If a "Posted [Acknowledge] action to OP5.." error does not appear in the log, it means Opsgenie didn't send the Acknowledge action to OP5. Check the integration configuration, it might not have matched the alert action.

• "Executed [Acknowledge] action via OEC with errors."エラーがログに表示された場合、OEC の op5ActionExecutor.Groovyスクリプトでエラーが発生したことを意味します。/var/log/opsgenie/ の下のログファイルをチェックして、エラーログを確認します。

• "Posted [Acknowledge] action to OP5.." エラーのみがログに表示されており、それ以降関連ログがない場合、OEC の接続に問題が発生している可能性があります。/var/log/opsgenie/ の下のログ ファイルをチェックして、エラー ログを確認します。

3. Could not open OP5 RPM package

• If you figure out while installing the rpm package that the package is obsolete, use rpm -i opsgenie-op5-1.0.4-rpm-x86-64.rpm --nodeps instead.

• If you get "is already installed" error, use rpm -i opsgenie-op5-1.0.4-rpm-x86-64.rpm --force instead.

If unsure of the problem, set the OEC's script log level to “debug” and try again. Contact us with the log files found under /var/log/opsgenie/.