PostgreSQL storage

By default, maintenant keeps everything in a SQLite file next to the binary. That is the right choice while one instance watches one host: nothing to install, nothing to administer, and a backup is a file copy.

It becomes a ceiling once the instance watches a fleet. The server holds one class of data the agents cannot rebuild: their identity and their enrolment. If the machine carrying the file is lost, so is the link between the server and every monitored host. Bringing up a replacement would mean re-enrolling the fleet by hand, host by host, while monitoring is blind.

Pointing the server at a PostgreSQL database you already operate removes that: the instance stops being attached to its machine.

What the product does, and what stays yours

The product provides one property: it keeps nothing essential on the machine that runs it. Redundancy, backups, restores and promotion of the database are yours, as is detecting a dead instance and restarting it elsewhere. maintenant does not detect failures, elect a leader, or move anything.

Agents are never affected. An agent stores its state in SQLite, always, and refuses a connection string outright.

Requirements

• PostgreSQL 14 or newer, reachable from the instance, with an empty database and a role that may create tables in it.
• A connection string written as a postgres:// or postgresql:// URL.
• An instance in server or embedded mode. In agent mode the setting is refused, with a message naming the mode.

PostgreSQL storage is open in every edition. --mode=server and enrolling agents need the Personal edition or above, not the database.

New install

Point the instance at the database with MAINTENANT_DATABASE_URL. The connection string is written once, as a YAML anchor, so the instance and the copy command further down can never drift onto two different databases:

# compose.yml
x-database-url: &database-url
  "postgres://maintenant:secret@db.internal:5432/maintenant?sslmode=require"

services:
  maintenant:
    image: ghcr.io/kolapsis/maintenant:latest
    restart: unless-stopped
    read_only: true
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp:noexec,nosuid,size=64m
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc:/host/proc:ro
      - /etc/os-release:/host/etc/os-release:ro
      - maintenant-data:/data
    environment:
      MAINTENANT_ADDR: "0.0.0.0:8080"
      MAINTENANT_DATABASE_URL: *database-url

volumes:
  maintenant-data:

docker compose up -d

The database is yours and stays out of this file: the guide assumes a server you already operate, not one the stack brings up.

The schema is created on first start, with no manual step. Check it:

curl -s localhost:8080/api/v1/health | jq .storage
# { "engine": "postgres", "connected": true, "peers": 0 }

When the database does not answer, refuses the credentials, runs a PostgreSQL older than 14, or holds a schema written by a newer release, the instance stops at startup with a message naming the cause. It never falls back to the local file.

Without MAINTENANT_DATABASE_URL, nothing changes: SQLite, exactly as before.

Keep the password out of the file you commit

The connection string carries a password, and compose.yml usually ends up in a repository. Put it in a .env file next to it and reference it from the anchor instead:

x-database-url: &database-url "${MAINTENANT_DATABASE_URL:?}"

MAINTENANT_DATABASE_URL has no _FILE variant: the value is read from the environment only, so a Compose or Docker secret cannot be pointed to directly. The product itself never writes the string anywhere, so the file that carries it is the only place it can leak from.

Transport encryption

When the connection string carries no explicit sslmode and the host is not local, the product adds sslmode=require: an external database is reached over TLS by default. Any explicit value wins, disable included, so a database on the same host or a private link can be relaxed deliberately. For a database crossing a network you do not control, prefer sslmode=verify-full, which also checks the server's certificate against its hostname. When that certificate comes from a private CA, add sslrootcert=/path/to/ca.pem to the connection string: MAINTENANT_CA_CERT covers the HTTPS and gRPC connections, not the database one.

"Local" means a host that is empty, localhost, 127.0.0.1 or ::1, or a Unix socket (a host parameter starting with /). Anything else, a Compose service name such as db included, is not local and gets sslmode=require.

A PostgreSQL that does not offer TLS then stops the instance at startup with the database server does not accept TLS. When the string carries no sslmode, the message says that sslmode=require was added by default and names the two ways out: enable TLS on the PostgreSQL server, or set sslmode=disable explicitly in MAINTENANT_DATABASE_URL if the network to the database is trusted (a container on the same Docker network, for instance). When the string already sets a mode that requires TLS, the message gives the same two ways out, the second one being to lower the sslmode you set.

The connection string is never written to the logs, the API, the interface or the telemetry. Where a target has to be named, it appears redacted: postgres://maintenant@db.internal:5432/maintenant.

What must follow the instance

A few files live in the data directory rather than in the database. That directory is the one holding MAINTENANT_DB, which stays meaningful with PostgreSQL: it no longer holds the data, but it still says where these files go. The image sets it to /data/maintenant.db, so the directory is /data; on a native install, set MAINTENANT_DB (or --db) to a path in a directory that follows the instance.

State	File	If the directory does not follow
Signed licence cache	<dataDir>/.maintenant-license	Re-verified online at startup. Offline: Community until the network returns.
Update window record	<dataDir>/.maintenant-update-window	A fresh grace window opens, which plays in your favour.
Embedded agent identity (only with MAINTENANT_EMBEDDED_AGENT)	<dataDir>/embedded-agent/	The embedded agent enrols again as a new host. The previous one stays in the list, stale, until you delete it.

Anonymous telemetry keeps its own state in a shm directory next to them (/data/shm in the image). There is no setting for it. Losing it only breaks the continuity of anonymous counters; nothing about the fleet depends on it.

So keep the /data volume with the instance when it moves. It is the only case where a move is not fully transparent, and it is settled by carrying the volume, which any cluster manager can do.

On Kubernetes (Helm)

The chart takes the connection string from a Secret you create, so it never appears in the values:

# values.yaml
database:
  existingSecret: maintenant-database   # Secret holding the connection string
  urlKey: url                           # key inside that Secret (default: url)

kubectl create secret generic maintenant-database \
  --from-literal=url='postgres://maintenant:secret@db.internal:5432/maintenant?sslmode=require'

Leave database.existingSecret empty and the instance keeps using SQLite on its volume. The volume (persistence.*) stays in use with PostgreSQL: it carries the licence cache and the update window record described above.

Migrating an existing install

An install already running on the local file can move to PostgreSQL without touching a single monitored machine. One command carries what the fleet cannot rebuild; everything else is re-reported or recomputed.

Add the connection string and a one-shot copy-store service next to the instance you already run:

# compose.yml
x-database-url: &database-url
  "postgres://maintenant:secret@db.internal:5432/maintenant?sslmode=require"

services:
  maintenant:
    # … your existing service, unchanged, plus:
    environment:
      MAINTENANT_DATABASE_URL: *database-url

  copy-store:
    # The profile keeps it out of `up`; `run` enables it by itself.
    profiles: ["migrate"]
    image: ghcr.io/kolapsis/maintenant:latest
    read_only: true
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp:noexec,nosuid,size=64m
    volumes:
      # The volume the instance writes to. The copy opens it read-only.
      - maintenant-data:/data
    # No MAINTENANT_DATABASE_URL here on purpose: the source is the local file,
    # the target is the argument. Giving it both would blur which is which.
    command:
      - "--db"
      - "/data/maintenant.db"
      - "--copy-store-to"
      - *database-url

Then, in order:

# 1. Stop the instance: the source must not move during the copy.
docker compose stop maintenant

# 2. Carry what does not rebuild itself.
docker compose run --rm copy-store

# 3. Restart on the database, which the service now points at.
docker compose up -d maintenant

The agents reconnect on their own, with the identity they already had.

docker compose run keeps a terminal attached, so the command says what it is about to do and waits for you. For an unattended run, add --yes to the command: list.

What travels

Agent identities and enrolment tokens, container rows, declared endpoint, heartbeat and certificate monitors, outbound heartbeat targets, notification channels and their secrets, alert triggers, silences, escalation policies, webhook subscriptions, per-container thresholds, the whole status page (settings, components, assets, FAQ, footer, subscribers), published incidents, planned maintenance windows, and your operator decisions: update exclusions, version pins, acknowledged findings.

What does not, and what you will notice

Everything the fleet rebuilds by itself: agent inventories (re-sent within 30 s), check and resource history (heartbeat pause periods included), state transitions, active alerts and their deliveries, CVE and image-update intelligence, ephemeral OAuth tokens.

Two effects are worth knowing before you start, and the command says them before writing anything:

• An alert acknowledged before the copy comes back unacknowledged if it is still active. The acknowledgement lives on the alert row, which does not travel. One click to redo, but it should not surprise you.
• Curves start from zero: resource, uptime and check history, daily uptime aggregates included (the 365-day history of endpoints, heartbeats and containers). Aggregates rebuild at their usual intervals.

Guarantees

Situation	What happens
The target is not empty	Refused before writing anything. Merging would need conflict rules nothing can settle correctly.
The copy fails half-way	The whole transaction rolls back, schema included. The target is empty again and you can retry.
Whatever happens	The source is opened read-only and never written to. Your original install stays usable.
The copy succeeds	Row counts are compared table by table, source against target, and any difference is an error.

Exit codes: 0 copied and verified, 1 refused (non-empty target, unreadable or out-of-date source, unreachable target, or you declined), 2 failed mid-copy and rolled back. A source written by an older version is refused too: start the current version once on it so that it migrates, then copy.

The copy does not switch your configuration: you set MAINTENANT_DATABASE_URL and restart yourself, deliberately. It does not run in the other direction either: going back means removing the variable, which returns to the local file as you left it.

Latency

Every write now costs a round trip to the database, where a local file cost none. Put the database in the same region, ideally the same network, as the instance. The measure that matters is the one you feel: the delay between an event reported by an agent and its appearance on screen should stay in the same class as before. A database several tens of milliseconds away is fine; one on another continent is not.

The probe must not kill the pod

/api/v1/health answers 200 even when the database is momentarily unreachable. The outage is reported in storage.connected, not in the HTTP status. This is deliberate: the endpoint is the target of the Kubernetes liveness and startup probes, and a probe failing on a ten-second blip would restart the instance exactly when the database needs to be left alone.

Do not replace it with a check that fails on that case.

Upgrading

The first instance that starts on a new version applies the schema changes. Instances started together queue on a lock held in the database, so only one of them migrates. An older version refuses a schema written by a newer one and does not write into it: to run an older version again, restore a backup of the database taken before the upgrade.

Two instances on one database

If a second instance beats on the same database, peers is non-zero and a warning is logged naming their count, hosts and versions. The product does not arbitrate: exclusion belongs to your cluster manager. Saying it is the only action it takes.

Testing against PostgreSQL

Two levels, both able to run on either engine.

The Go suites. They run on SQLite by default, and on PostgreSQL when a test server is configured. make test-pg starts a disposable PostgreSQL 14, points MAINTENANT_TEST_DATABASE_URL at it and runs the store suite:

make test-pg

Without that variable, the PostgreSQL cases are skipped, never failed, so go test ./... still works without Docker.

That skip is convenient locally and dangerous in CI: a typo in the variable name, or a database service that never came up, would skip every PostgreSQL case and report success. MAINTENANT_REQUIRE_POSTGRES=1 turns each of those skips into a failure, and CI sets it on every step that is supposed to exercise PostgreSQL. Use it whenever a green run is meant to prove PostgreSQL coverage:

MAINTENANT_TEST_DATABASE_URL=... MAINTENANT_REQUIRE_POSTGRES=1 go test ./internal/store/...

The whole product. A test stack builds the image and runs it with the services a real install talks to (SMTP catcher, webhook receiver, a container to watch), then drives it through its HTTP surface:

make e2e-sqlite     # the stack on the default local file
make e2e-postgres   # the same stack, on PostgreSQL 14
make e2e-both       # both in turn, on clean state: this is the one that matters

e2e-both is the check worth running before shipping anything that touches storage: identical results on both engines is the claim, and a difference shows up as a failed line rather than as a surprise in production.

To poke at a running stack instead: make e2e-up-postgres, then http://127.0.0.1:18090 (mail on :18091, webhooks on :18092). make e2e-logs follows the instance, make e2e-down removes everything including the volumes. Ports are overridable (E2E_HTTP_PORT, E2E_DB_PORT, …) if something already listens there.

The migration, end to end. make e2e-migrate takes a running SQLite stack, stops it, copies it into the PostgreSQL one the way an operator would, restarts on the database and re-runs the checks, so the path from an existing install to an external database is exercised, not just described. It runs the very copy-store service shown above, on the released image and entrypoint, so what CI proves is the command you type.

Going back

Remove MAINTENANT_DATABASE_URL from the service and docker compose up -d maintenant: the instance finds its SQLite database in the state it left it. Whatever was written to PostgreSQL in the meantime is not there, and the migration command does not run in that direction.