Configuring rules and alerts in Prometheus

Once you set up Prometheus for monitoring purposes, you need to configure the required rules for sending alerts and notifications to the intended recipients. Prometheus needs to work in tandem with the Alertmanager that manages the alerts, including silencing, inhibition, aggregation and sending out notifications via methods such as email, on-call notification systems, and chat platforms.

To make this happen, you need to perform the following activities:

Configure the Alertmanager
Configure Prometheus to talk to the Alertmanager
Create alerting rules in Prometheus

Configuring Alertmanager

You can configure the alerts in alertmanager.yml file that comes with Alert Manager distribution and can be edited to add required configuration.

Given below is an example of a simple Alertmanager yaml.

Email Alert Configuration

route:
  receiver: 'Email-Me'
receivers:
  - name: 'Email-Me'
    email_configs:
      - to: ranjit.singh@adeptia.com
        from: ranjit92singh@gmail.com
        smarthost: smtp.gmail.com:587
        auth_username: ranjit92singh@gmail.com
        auth_password: *****************

Configuring Prometheus to talk to the Alertmanager

You can configure Prometheus to communicate and send alerts to the Alertmanager in prometheus.yml. Alertmanagers can be made to work with Prometheus statically by using the static_configs parameter in the prometheus.yml file. It can also be dynamically discovered using one of the supported service-discovery mechanisms. See configuration details at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alertmanager_config

The following code snippet and the screenshot illustrates the configuration in discussion.

Alert Manager

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
           - localhost:9093

Create alerting rules in Prometheus

To have rules set in Prometheus, you need to follow the steps given below:

Create a new YAML file, say rules.yaml, containing the alerting rules based on your requirement.
Alerting rule example: The following code snippet contains a rule that checks if the configured instance is up or down.
Example Rules file Expand source
```
groups:
  - name: example
    rules:
      - alert: Instance Down
        expr: up == 0
        for: 1m
```
Place path of rules yaml file in prometheus.yml as shown in the following screenshot.

The following description gives you a glimpse of how Prometheus and Altermanager look like and work together to generate an alert, manage it, and send out notifications via email.

Based on the rules you set, Prometheus sends an alert to the Alertmanager, which in turn, based on its configuration, sends out the notification to the intended recipient.

The following screenshot illustrates an alert rule in Prometheus. This rule generates and alert to be sent to the Alertmanager in case an instance goes down.

The following screenshot illustrates the alert received by the Alertmanager.

After the Alertmanager receives and alert, it sends a notification to the intended recipient.

The following screenshot illustrates a sample email notification.