Skip to main content

Incidents

This module creates incidents from external alert webhooks and allows manual incident management.

Currently supported provider:

  • Grafana Alertmanager

Installation​

composer require qore/incidents

Publish and run the migrations:

php artisan vendor:publish --tag=qore.incidents.db --force
php artisan tenants:migrate

Publish the config:

php artisan vendor:publish --tag=qore.incidents.config

After installing, enable the module in your Qore application.

Grafana webhook​

Endpoint​

POST /api/incidents

Authentication​

The endpoint uses the normal Qore confirmed API middleware.

In practice this means Grafana must send a valid authenticated request, for example with a bearer token from a Qore user that belongs to the target tenant.

Payload​

The module reads the normal Grafana Alertmanager payload.

How to create a Grafana alert​

Keep the alert rule simple.

Only these fields are meant to be overridden:

  • annotations.incident_title
  • annotations.description
  • annotations.summary
  • labels.priority

You can use these source tags/fields in your templates:

  • monitor_name
  • instance
  • nodename
  • alertname
  • values
  • valueString

Example:

annotations.incident_title=CPU langdurig te hoog op {{ $labels.instance }}
labels.priority=warning
annotations.description=CPU alert op {{ $labels.monitor_name }} met waarde {{ $values.A }}
annotations.summary=CPU staat al 15 minuten boven 85% op {{ $labels.instance }}

Dependencies​

This module depends on:

  • qore/environments
  • qore/glue
  • qore/tasks
  • qore/timeline

How to test locally​

  1. Start Grafana locally, for example with Docker:
services:
  grafana:
    image: grafana/grafana-enterprise
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana

volumes:
  grafana_data:
  1. Open http://localhost:3000 and log in with:
  • username: admin
  • password: admin
  1. In Grafana, go to Connections -> Data sources -> Add new data source -> Prometheus.

Use this Prometheus URL:

http://stats.qlic.nl:9090

Make sure you are on the office network or connected through VPN, otherwise this endpoint may be blocked.

  1. Export the dashboard from stats.qlic.nl into your local Grafana instance.

  2. Create a notification configuration.

For example, use the standard Webhook integration and set the webhook URL to:

http://host.docker.internal:8000/api/incidents

Optional webhook settings:

  • HTTP Method: POST
  • Authorization Header Scheme: Bearer
  • Authorization Header Credentials: YOUR-API-TOKEN-FROM-QORE
  1. Run a test notification.

If everything is configured correctly, a new incident should be added to the incidents table.

If not, check:

  • the bearer token
  • the webhook URL
  • the Grafana webhook settings
  • whether Grafana can reach the local backend