mipo
Menu
ProductFeaturesArchitectureSecurityComplianceDocsGitHub
Guide

Operations: Scanner CA Rollover

An evaluator-facing overview of how mipo rotates the CA that signs scanner client certificates, with no scanner ever losing the ability to authenticate. This is not the operational procedure.

Operators: use the runbook in your release bundle

The step-by-step procedure, every verb with its real flags, and the recovery paths ship with every mipo release at docs/operations/scanner-ca-rollover.md, alongside the incident runbooks ops/runbooks/scanner-ca-rollover-stall.md, ops/runbooks/scanner-ca-rollover-failed.md and ops/runbooks/scanner-ca-emergency-transition.md. Follow those instead of this page.

The rollover flow

There is no GUI for a rollover. The manager's background driver executes the state machine one phase at a time (a tick every 5 seconds); operators observe and gate it with runRolloverCli.js inside the manager container or the admin API (/api/config/scanner-mtls/rollover, step-up gated; starting one requires the body literal confirm: 'rollover-ca'). At most one rollover is non-terminal at a time, and a non-terminal rollover also blocks the fleet policy flip to required.

  1. Stage generation N+1 (start --operation-id <id>). Abandonable until the next step.
  2. Widen the served trust bundle to N and N+1 — DB revision (dual_bundle_db_active), watchdog publish (dual_bundle_published), dedicated mTLS Traefik reload (dual_bundle_reloaded), real handshake (dual_served_verified). From here the rollover is forward-only.
  3. Re-issue every scanner and resolver identity a leaf signed by N+1 (n1_issuing) and wait until each scanner and resolver has actually presented it (fleet_reissued). Usually the longest phase; an unreachable scanner stalls it.
  4. Narrow the bundle to N+1 only — single_bundle_db_active, single_bundle_published, single_bundle_reloaded, single_served_verified. The driver stops here.
  5. An operator confirms the drop (confirm-drop, echoing the revision and snapshot digest from status). The driver destroys N’s private key on a later tick (n_destroyed).

Terminal phases are n_destroyed, emergency_n_destroyed and abandoned. Every bundle transition is four separate phases (*_db_active, *_published, *_reloaded, *_served_verified) because each can fail and is retried independently. The emergency path (emergency_quarantined onward) never serves a dual bundle and needs no confirmation: the compromised key is destroyed as soon as the replacement is served-verified and the fleet re-issued.

The verbs

  • start --operation-id <id> — Record intent; first advance to n1_staged.
  • status [--operation-id <id>] — Phase, blockers, and the drop-confirmation preview.
  • advance / run --until <phase> [--timeout-ms N] — Operator path when the background driver is unhealthy: one phase, or up to a named phase.
  • confirm-drop --operation-id <id> --single-bundle-revision <int> --snapshot-digest <hex> --actor-user-id <int> --actor-ip <ip> — Authorize the destruction of N. Deferred to a later driver tick; irreversible once done.
  • abandon --operation-id <id> --reason <text> — Legal only from idle or n1_staged.
  • emergency --compromised-generation <int> [--operation-id <id>] — CA compromise: quarantine + revoke first, no dual overlap, forced re-issue, destroy. No confirmation step.
  • reconcile-bundles [--watchdog-deadline-ms N] / verify-bundles [--require-proof-receipt N] — Republish the DB’s active bundle to the served bundle and record a proof receipt; read-only verification (both run by restore.sh).

Two irreversible edges

Forward-only after the dual bundle. abandon is legal only from idle or n1_staged; once dual_bundle_db_active is reached the runtime already trusts the staged CA and the only way out is to finish the single-bundle phases. Destruction is deferred and final. confirm-drop records the operator's authorization against a persisted readiness snapshot; the driver wipes generation N on a later tick, and a scanner still holding only an N leaf after that must be re-enrolled.

Alarms and runbooks

  • scanner-ca-rollover-stall — no phase advance and no blocker change for the stall window → ops/runbooks/scanner-ca-rollover-stall.md
  • scanner-ca-rollover-failed — a step of an orderly rollover threw → ops/runbooks/scanner-ca-rollover-failed.md
  • scanner-ca-emergency-transition-failed — a step of the emergency transition threw → ops/runbooks/scanner-ca-emergency-transition.md
  • scanner-certificate-expiry — a scanner leaf is close to expiry (renewal failing) → check the scanner host and status blockers

Scanners: provisioning, enrollment, fleet policy → Alarm rules Scanner install