Integrate Opsgenie with OpenNMS

What does Opsgenie offer OpenNMS users?

Opsgenie enables OpenNMS users to receive OpenNMS events (referred to as alerts in Opsgenie) through email, SMS, phone call, and iPhone/Android push notifications and works with OpenNMS events from mobile phones. Opsgenie alerts contain not only the event but also supporting information such as managed object details, performance charts, etc. enabling users to make better decisions when notified.

Opsgenie provides various integration tools and APIs to integrate monitoring tools with Opsgenie so that users are notified when something important occurs in their environment. Use the following integration scenarios and packages provided by Opsgenie as a starting point to integrate OpenNMS with Opsgenie alerting system.

Scenario 1: Email Integration

OpenNMS has the capability to send emails to notify users for specified events. Using Opsgenie’s email integration capabilities, Opsgenie notifies users via SMS, iPhone or Android push notifications, or phone calls. Instructions below describe how to configure OpenNMS and Opsgenie to send push notifications to iPhone/Android apps using Opsgenie email integration for OpenNMS authentication failed events as an example.

Add OpenNMS Email method in Opsgenie

1. Set up Opsgenie email rules which create alerts from emails sent from OpenNMS. When your account is created Opsgenie provides a default email address and a default email rule for you. Modify your email address and rules in the Settings page. For this setup the default rules are used, which creates alerts for every incoming mail. Find more information on email integration here.

2. Add an Opsgenie Email Integration.

3. Copy the email address from the Opsgenie Email Integration.

4. Click Save Integration.

Add Opsgenie in OpenNMS

1. Create a user account for Opsgenie integration in OpenNMS web UI. Click the Admin link in the menu bar.

2. Go to "Configure Users, Groups and On-Call Roles".

3. Click Configure Users.

4. Click Add New User, type "OpsGenie" in the username field and provide a password in the form.

5. Click OK.

6. Paste the email copied previously to the Email field in the OpenNMS user form.

7. Click Finish.

To receive a notification from OpenNMS, your user or a group that your user is in needs to be a part of a destination path.

1. Go to the "Admin" page.

2. Click Configure Notifications in the "Operations" section, then "Configure Destination Paths," then click New Path.

3. Give it a name, like "OpsGenieEmailPath", then click Edit.

4. Add "OpsGenie" user by selecting it, then click Next >>>.

5. On the next page, leave the defaults ("javaEmail" and "on") and click Next >>> again.

6. Click Finish and you should see your new destination path in the "Existing Paths" section.

Next, associate OpenNMS events with the new destination path.

1. Go to the "Admin" page,

2. Click Configure Notifications in the "Operations" section, then Configure Event Notifications.

3. Click Add New Event Notification at the top of the "Event Notifications" page.

4. Select "OpenNMS-defined internal event: an authentication failure has occurred in WebUI" event for the time being to test the setup easily. This configuration can be changed later if desired.

5. Click Next >>>.

6. Click Skip results validation >>> on the next page.

Finally, configure the information used to send the notification. Fill in the following values:

Name: Authentication Failed
Choose A Path: OpsGenieEmailPath
Text Message: The OpenNMS Web UI had a failed login attempt, by user '%parm[user]%', from IP address %parm[ip]%
Short Message: Authentication failed by user '%parm[user]%'
Email Subject: Authentication failed by user '%parm[user]%'

7. Click Finish. Now notification events appear in the list, but are disabled.

8. Set the radio button to On to enable it. In the main OpenNMS Admin page, there is a radio selector labeled, "Notification Status." Change it to On.

9. Click Update.

10. Configure the mail server in $OPENNMS_HOME/etc/javamail-configuration.properties file and restart OpenNMS server.

Now the setup is completed to create an OpenNMS event and receive an Opsgenie push notification.

Logout from OpenNMS web UI and try to login with an invalid account. A push notification is sent, if not delivered, take a look at "$OPENNMS_HOME/logs/daemon/notifd.log" and see if an error message occurs. If no problem exists, look at Opsgenie alerts page and check whether the new alert exists in the alert list.

Access more information on OpenNMS notifications in the following pages:

OpenNMS Configuring Notifications

OpenNMS Notification Tutorial

Scenario 2: Rich Alert Integration

OpenNMS notification system allows executing arbitrary external programs, which opens endless possibilities to integrate with Opsgenie alerting system through Opsgenie client tools like Lamp. Opsgenie provides a downloadable OpenNMS integration package which shows rich alerting capabilities of our system. Integration files include example groovy script that forwards node events, script also retrieves most recent node outages to the alert from OpenNMS and attaches to the alert to make this information available to the recipients.

Installation

• Download OpsGenie OpenNMS Integration files that include:

• Lamp - Opsgenie command line utility to create alerts

• OpenNMS Integration scripts

• Unzip OpenNMS Integration

• Lamp installation:

• OpenNMS Integration Files contains a customized version of Lamp , install this version.

• Install lamp by referring to Lamp Documentation - Installation steps. Do not download lamp again.

• Make sure the OpenNMS shell scripts have correct LAMP_HOME, the lamp installation directory.

• Create an OpsGenie API Integration, copy the integration apiKey, put into the opsgenie-integration/conf/opsgenie-integration.conf file.

• Verify Lamp is working: Create a test alert, using the lamp utility from the command line, make sure your alert is visible on Opsgenie dashboard. Please refer to Lamp Documentation for more information.

• Groovy script configuration: Instead of implementing all logic in the shell script, the integration package provides a script (ogCreateAlert.groovy) written in groovy - a powerful Java-based scripting language- which can be executed by Lamp. Briefly, the script creates an Opsgenie alert, retrieves node outages through OpenNMS rest API, creates an HTML file showing the outages and attaches it to the alert.

• RECIPIENTS: Users or group of users who will be notified by Opsgenie. Defaults to "all".

• SOURCE: Alert source

• OPENNMS_USER and OPENNMS_PASSWORD: Credentials to authenticate OpenNMS rest API.

• OPENNMS_HOST and OPENNMS_PORT: OpenNMS web server host and port which is used for REST API calls.

Text

• Prepare opennmsShellScripts/opsgenie shell script :

• Copy the shell script anywhere in the same system where OpenNMS installed. This path is pointed to when OpenNMS notification commands are configured.

• Configure LAMP_HOME to point your lamp installation directory.

• Make the script executable in linux (chmod +x)

• Prepare Opsgenie Groups & Users

• Create groups or users that you want to notify from Smarts

• Make sure groups have at least one user

OpenNMS Configuration

1. Add a custom notification command by adding the following section to the $OPENNMS_HOME/etc/notificationCommands.xml configuration file. Check OpenNMS Notification Configuration document for details.

XML

2. Modify the tag so that it points the shell script provided with OpsGenie integration package.

3. Restart OpenNMS server.

4. Add an OpenNMS user and give it a name like "Opsgenie".

5. The user added should be a part of a destination path so that the shell script provided can be executed with OpenNMS.

6. Go to the Admin page, click Configure Notifications in the "Operations" section.

7. Click Configure Destination Paths,

8. Click New Path. Give it a name, like "OpsGenieScriptPath", then click Edit.

9. Add the user by selecting it, then click Next >>>.

10. On the next page, check the "OpsGenie" custom command and on.

11. Click Next >>> again.

12. Finally, click Finish. Now the new destination path appears in the "Existing Paths" section.

Associate OpenNMS events with the new destination path

1. Go to the Admin page.

2. Click Configure Notifications in the "Operations" section.

3. Click Configure Event Notifications.

4. Click *Add New Event Notification at the top of the Event Notifications page and select "OpenNMS-defined node event: nodeLostService" event.

5. Click Next >>>.

6. Click Skip results validation >>> on the next page. Configure the information used to send the notification. Fill in the following values:

   • Name: Node Lost Service

   • Choose A Path: OpsGenieShellPath

   • Text Message: The %service% service poll on interface %interfaceresolve% (%interface%) on node %nodelabel% failed at %time%.

   • Short Message: %service% down on %interfaceresolve% (%interface%) on node %nodelabel%.

   • Email Subject: %service% down on %interfaceresolve% (%interface%) on node %nodelabel%.

7. Click "Finish".

• The notification event in the list will be disabled. Set the radio button to "On" to enable it.

• In the main OpenNMS Admin page, there is a radio selector labeled, "Notification Status." Change it to "On" and click update.

Test
To test the integration package, make the OpenNMS server to scan the local PC which runs an HTTP server on port 8080, with the following command:

Text

After the scan completed OpenNMS creates the node, its interfaces, and services including the HTTP server.

When the HTTP server is shut down, OpenNMS detects it in the next service poll and creates a "nodeLostService" event for it.

The event triggers the notification system which executes the shell script installed. As a result a new alert is created in Opsgenie system and most recent outages of my PC are displayed in the Opsgenie console.