Alert Engine¶

Unified alerts across all monitoring sources. Webhook and Discord channels included. Silence rules for planned maintenance. Exponential backoff retry on delivery failures.

Alert Sources¶

maintenant generates alerts from every monitoring subsystem:

Source	Events	Default Severity
Container	`restart_loop`, `health_unhealthy`, `container_down`	Warning
Endpoint	`consecutive_failure`	Critical
Heartbeat	`deadline_missed`, `exit_code_failure`	Critical
Certificate	`expiring`, `expired`, `chain_invalid`	Critical
Resource	`cpu_threshold`, `memory_threshold`	Warning
Update	`available`	Info
Agent	`disconnected`	Warning

Container down¶

A container that stops and stays stopped raises no alert on its own: restart_loop needs it to come back, and health_unhealthy needs a HEALTHCHECK. Set MAINTENANT_CONTAINER_DOWN_AFTER to a Go duration (5m, 30s, 1h) and container_down fires once a container has been in exited or dead for that long, at the severity configured on the container. It resolves on its own when the container runs again.

The threshold is unset by default, because switching it on alerts retroactively on every container already stopped. A container that exited with code 0 is recorded as completed and never counts as down, so a finished job stays quiet. The sweep runs every 30 seconds, which is the alert's resolution, not its threshold.

An agent alert fires when a remote agent stops reporting, whether its stream dropped or it never came back after a restart, and resolves on reconnection. Revoking or deleting an agent clears it instead of raising one.

Deleting a monitored entity (container, agent, heartbeat, endpoint, certificate) resolves its active alerts. On every startup, maintenant also resolves any active alert whose entity no longer exists, so alerts left behind by an earlier version cannot linger.

Notification Channels¶

maintenant delivers alerts to Discord, Telegram, email, Slack, Teams, and any HTTP webhook. Each channel type formats the payload natively for the target platform.

Discord¶

maintenant sends Discord embeds with severity-colored borders.

POST /api/v1/channels
{
  "name": "ops-discord",
  "type": "discord",
  "url": "https://discord.com/api/webhooks/..."
}

Generic HTTP Webhook¶

For Slack, Teams, or any other service, use a generic webhook:

POST /api/v1/channels
{
  "name": "custom-webhook",
  "type": "webhook",
  "url": "https://your-service.example.com/alert"
}

maintenant sends a JSON payload with alert details to the configured URL.

Email (SMTP) ¶

Native email delivery over your own SMTP server. Available from the Personal edition.

Telegram ¶

A native Telegram channel: you supply a bot token and a chat id, never a URL. The destination is fixed, so nothing has to be worked around — neither the payload format a generic webhook gets wrong, nor the SSRF guard that refuses a local relay. Available from the Personal edition.

1. Create the bot. Message @BotFather, send /newbot, follow the questions. It hands back a token shaped like 8123456789:AAF-…. Treat it as a password: it is the bot.

2. Find the chat id.

Private chat: message the bot once, then read message.chat.id from https://api.telegram.org/bot<token>/getUpdates.
Group or channel: add the bot to it, post a message, same call. The id is negative, often prefixed -100. Paste it exactly as it appears.

3. Create the channel.

POST /api/v1/channels
{
  "name": "oncall-telegram",
  "type": "telegram",
  "url": "-1001234567890",
  "secret": "8123456789:AAF-...",
  "config": { "thread_id": "42" }
}

config.thread_id is optional and only applies to groups organised in topics; leave it out and messages land in the general thread.

The token is write-only. It is never returned by the API, never written to a log, and never appears in an error message; responses carry has_secret so the interface can say a token is on file without holding it. An update that omits secret keeps the stored one.

Messages use Telegram's HTML formatting: a severity emoji and the entity on the first line, so a phone notification says what happened before you open it. A recovery arrives as a separate message, marked ✅. Anything over Telegram's 4096-character limit is truncated with a visible marker rather than rejected.

When a send fails, the delivery log carries Telegram's own words — chat not found, bot was kicked from the supergroup chat, bot can't initiate conversation with a user — because those name the fix, where an HTTP code does not. On a rate limit, the retry waits at least as long as Telegram asked.

What an expired licence changes

A Telegram channel already created keeps delivering in any edition. What Community closes is creating, testing, and editing it. Disabling and deleting stay open: an expired licence must never leave you unable to silence a channel.

Slack & Teams ¶

Native Slack (Block Kit) and Microsoft Teams (MessageCard) channels, with platform-specific formatting.

POST /api/v1/channels
{
  "name": "ops-slack",
  "type": "slack",
  "url": "https://hooks.slack.com/services/..."
}

Channels are silent by default¶

A notification_channel represents where to send (an URL/email/webhook target). It does not decide when to send. After creating a channel, it stays silent until referenced by an Alert Trigger or by an Escalation Policy.

This decoupling enables the reserved-escalation pattern: a channel that only fires through an escalation policy at a delayed level (e.g. CTO email at T+1h), without receiving the initial alert.

Alert Triggers¶

Triggers are the routing layer. Each trigger combines a filter and a list of channel destinations: when an alert matches a trigger's filter, the alert is dispatched to all of its channels.

POST /api/v1/alert-triggers
{
  "name": "Critical containers → ops",
  "filter_severities": "critical",
  "filter_sources": "container",
  "filter_scopes": "",
  "filter_tags": "",
  "enabled": true,
  "notify_on_resolve": true,
  "channel_ids": [3, 7]
}

Filters are CSV strings, combined in AND between fields and OR within a field. An empty filter matches everything. Multiple triggers can share the same channel without duplicating deliveries (the engine de-dupes per alert).

A trigger relays both the initial alert and its recovery. Set notify_on_resolve to false (default true) for a channel that should only receive failures.

CE vs Pro filters :

Filter	CE	Pro
`filter_severities` (e.g. `critical,warning`)	✅	✅
`filter_sources` (e.g. `container,endpoint`)	✅	✅
`filter_scopes` (e.g. `container:42,endpoint:7`)	—	✅
`filter_tags` (e.g. `prod,payments`)	—	✅

Trigger CRUD endpoints :

Method	Path
`GET`	`/api/v1/alert-triggers`
`POST`	`/api/v1/alert-triggers`
`GET`	`/api/v1/alert-triggers/{id}`
`PUT`	`/api/v1/alert-triggers/{id}`
`DELETE`	`/api/v1/alert-triggers/{id}`

Triggers can also be managed via MCP tools: list_triggers, get_trigger, create_trigger, update_trigger, delete_trigger. Channels have the matching set: list_channels, get_channel, create_channel, update_channel, delete_channel, test_channel.

Migration note: previous versions used routing_rules attached to channels. On upgrade, those rules are auto-converted to AlertTriggers (one trigger per rule). Channels without any rule receive a generated Default — all alerts → {channel name} trigger to preserve the legacy broadcast behavior. The legacy /api/v1/channels/{id}/rules* endpoints have been removed.

Testing Channels¶

Send a test alert to verify your channel configuration:

POST /api/v1/channels/{id}/test

Silence Rules¶

Suppress alerts during planned maintenance windows. Silence rules prevent alert delivery without discarding the events.

# Create a silence rule
POST /api/v1/silence
{
  "reason": "Scheduled database maintenance",
  "starts_at": "2026-03-01T02:00:00Z",
  "ends_at": "2026-03-01T04:00:00Z",
  "matchers": {
    "source": "endpoint",
    "entity_id": "postgres-health"
  }
}

# List active silence rules
GET /api/v1/silence

# Cancel a silence rule
DELETE /api/v1/silence/{id}

Use silence rules for deployments

Create a silence rule before deploying to avoid alerting on expected container restarts and brief endpoint downtime.

Retry and Backoff¶

When a webhook delivery fails, maintenant retries with exponential backoff:

Attempt 1: immediate
Attempt 2: 30 seconds
Attempt 3: 1 minute
Attempt 4: 2 minutes
Attempt 5: 4 minutes

This ensures alerts are delivered even when the receiving service is temporarily unavailable.

Viewing Alerts¶

Active Alerts¶

GET /api/v1/alerts/active

Returns only currently active (unresolved) alerts.

Alert History¶

GET /api/v1/alerts

Returns all alerts, including resolved ones.

Single Alert¶

GET /api/v1/alerts/{id}

API Endpoints¶

Method	Endpoint	Description
`GET`	`/api/v1/alerts`	List all alerts
`GET`	`/api/v1/alerts/active`	List active alerts
`GET`	`/api/v1/alerts/{id}`	Get alert details
`GET`	`/api/v1/channels`	List notification channels
`POST`	`/api/v1/channels`	Create a channel
`PUT`	`/api/v1/channels/{id}`	Update a channel
`DELETE`	`/api/v1/channels/{id}`	Delete a channel
`POST`	`/api/v1/channels/{id}/test`	Send test alert
`GET POST`	`/api/v1/alert-triggers`	List / create triggers
`GET PUT DELETE`	`/api/v1/alert-triggers/{id}`	Manage a trigger
`GET POST`	`/api/v1/escalation-policies`	List / create policies (Pro)
`GET PUT PATCH DELETE`	`/api/v1/escalation-policies/{id}`	Manage a policy (Pro)
`GET`	`/api/v1/alerts/{id}/escalation-runs`	List runs for an alert (Pro)
`GET`	`/api/v1/silence`	List silence rules
`POST`	`/api/v1/silence`	Create silence rule
`DELETE`	`/api/v1/silence/{id}`	Cancel silence rule

Alert Escalation — Pro: multi-level escalation chains
Container Monitoring — Restart loop and health check alerts
Endpoint Monitoring — Consecutive failure alerts
Heartbeat Monitoring — Deadline missed alerts
Certificate Monitoring — Expiry alerts
Resource Metrics — Threshold alerts