mipo

Alarm Rules

Built-in alarm rules ship pre-configured with mipo and cover scanner health, infrastructure liveness, and core-service availability. Each rule has a trigger event, severity, and default routing — operators can override severity, mute, or change notification channels per rule from the Alarm Rules admin page. Rules cannot be created or deleted — they are code-defined and seeded automatically on every boot.

Fields & columns

NameDescription
EnabledWhether this rule creates alarms. Disabled rules ignore matching events.
NameUnique rule identifier (e.g., scanner-down, tls-expiring)
DescriptionHuman-readable explanation of what the rule detects
SeverityCurrent severity for new alarms. Can be overridden from the default.
Auto-ResolveEvent type that auto-resolves matching alarms (null = manual only)
scanner-downAvailability: fires on scanner_down (default: high), auto-resolves on scanner_up. Scanner missed heartbeat timeout (2 minutes).
ingest-downAvailability: fires on ingest_down (default: critical), auto-resolves on ingest_up. Ingest API node stopped heartbeating.
db-downAvailability: fires on db_down (default: critical), auto-resolves on db_up. Database connection pool test failed.
proxy-downAvailability: fires on proxy_down (default: critical), auto-resolves on proxy_up. Traefik reverse proxy health check failed.
dns-downAvailability: fires on dns_down (default: high), auto-resolves on dns_up. Public FQDN DNS resolution failed.
backup-failedOperations: fires on backup_failed (default: high), manual resolve only. Manual or scheduled backup completed with errors.
backup-schedule-downOperations: fires on backup_schedule_down (default: high), auto-resolves on backup_schedule_up. A backup schedule is enabled but the newest bundle is older than 26 hours (or an install older than that has never produced one) — checked against the backup HTTP API bundle list every evaluator tick.
alarm-stormOperations: fires on alarm_storm_detected (default: high), auto-resolves on alarm_storm_normal. More than 500 alarm rows were opened in the last hour — a runaway signal or a stuck evaluator cursor (see ops/runbooks/alarm-storm.md).
dns-resolver-group-unavailableReverse DNS: fires on dns_resolver_group_unavailable_detected (default: high), auto-resolves on dns_resolver_group_unavailable_normal. Lookups for a scan have waited more than 120 seconds with no online resolver in the scan's resolver group — checked per resolver group by the dispatcher (see ops/runbooks/reverse-dns.md).
dns-enrichment-backlogReverse DNS: fires on dns_enrichment_backlog_detected (default: warning), auto-resolves on dns_enrichment_backlog_normal. A reverse-DNS job in the resolver group has waited unclaimed for more than 30 minutes (twice the enrichment window), which means the resolver group is not keeping up with its work (see ops/runbooks/reverse-dns.md).
db-disk-highThreshold: fires on db_disk_high (default: warning), auto-resolves on db_disk_normal. Database disk usage exceeded threshold.
db-connections-highThreshold: fires on db_connections_high (default: warning), auto-resolves on db_connections_normal. Database connection pool has sustained waiting queries.
tls-expiringDeadline: fires on tls_expiring (default: warning), auto-resolves on tls_renewed. TLS certificate expires within 30 days.
scanner-certificate-expiryDeadline: fires on scanner_certificate_expiry_high (high), auto-resolves on scanner_certificate_expiry_normal. Scanner mTLS certificate expires within 5 days.
scanner-ca-rollover-stallStall: fires on scanner_ca_rollover_stall_high (default: high), auto-resolves on scanner_ca_rollover_stall_normal. A CA rollover has made no phase advance and no change to its outstanding blocker set for 24 hours. A nonterminal rollover holds the one-operation slot that blocks every future rollover and the migration-to-required mTLS policy flip, so a stalled one is a blocked platform, not a paused job.
scanner-ca-rollover-failedOperations: fires on scanner_ca_rollover_failed (default: high), manual resolve only. A step of an orderly CA rollover threw. A retryable lock timeout deliberately does NOT raise this — the background driver retries contended locks every 5 seconds on its own.
scanner-ca-emergency-transition-failedOperations: fires on scanner_ca_emergency_transition_failed (default: critical), manual resolve only. A step of the emergency CA-compromise transition threw. The emergency path leaves the fleet with no trusted issuer between containment and promotion, so this alarm means the fleet may be sitting in that gap.
scanner-load-highResource: fires on scanner_load_high (default: warning), auto-resolves on scanner_load_normal. Scanner load average exceeds 80% of CPU capacity.
scanner-memory-highResource: fires on scanner_memory_high (default: warning), auto-resolves on scanner_memory_normal. Scanner available memory below 10%.
db-sessions-highDatabase: fires on db_sessions_high (default: warning), auto-resolves on db_sessions_normal. Active database sessions exceed 80% of max_connections.
db-long-queriesDatabase: fires on db_long_queries_high (default: warning), auto-resolves on db_long_queries_normal. Database query running longer than 60 seconds.
auth-failures-highSecurity: fires on auth_failures_high (default: high), auto-resolves on auth_failures_normal. More than 10 authentication failures in 5 minutes.
session-ip-spreadSecurity: fires on session_ip_spread_high (default: warning), manual resolve only. One user account has active sessions from too many distinct IP addresses.
scan-stuckStall: fires on scan_stuck (default: warning), manual resolve only. Scan still running but all jobs are finished.
scanner-heartbeat-failedSecurity: fires on scanner_heartbeat_failed (default: warning), manual resolve only. Scanner heartbeat authentication failed — invalid API key or IP binding violation. Requires manual acknowledgement; a scanner can resume heartbeating successfully while prior auth failures remain in scope.
dispatcher-downAvailability: fires on dispatcher_down (default: critical), auto-resolves on dispatcher_up. Dispatcher service stopped heartbeating for more than 2 minutes. When the dispatcher is down, no new scan jobs are dispatched and active scans stall.
backup-downAvailability: fires on backup_down (default: warning), auto-resolves on backup_up. Backup container is unreachable via DNS resolution. Typically indicates the backup container has stopped or failed to start.
manager-memory-highResource: fires on manager_memory_high (default: warning), auto-resolves on manager_memory_normal. Manager heap usage exceeded 80% of the 512 MB container memory limit. Sustained high memory may precede an OOM termination; consider restarting the manager or investigating large in-flight requests.
container-recovery-failedInfrastructure: fires on container_recovery_failed (default: critical), manual resolve only. A container did not recover within 2 minutes after the watchdog attempted a restart. Indicates a persistent crash loop or configuration error that automatic healing cannot fix.
container-restart-stormInfrastructure: fires on container_restart_storm (default: high), manual resolve only. A container restarted more than 3 times within a 15-minute window. Suggests a crash loop; investigate container logs before allowing further restarts.
container-downInfrastructure: fires on container_down (default: high), auto-resolves on container_up. Container is running but not reachable over the Docker internal network — health check connections are refused or timing out.
container-restartedInfrastructure: fires on container_restarted (default: info), manual resolve only. Container was restarted by the watchdog autoheal mechanism. Informational — the restart itself is the resolution of the underlying fault. Review logs if restarts become frequent.

How to

Disable an alarm type

  1. Find the rule in the table
  2. Click the enabled toggle to turn it off
  3. New events of this type will be ignored (existing alarms remain)

Override severity

  1. Find the rule in the table
  2. Use the severity dropdown to change the level
  3. New alarms will use the overridden severity

Reset severity to default

  1. Find the rule with a non-default severity (shown in parentheses)
  2. Click the "Reset" button
  3. Severity reverts to the code-defined default

Gotchas

  • Disabling a rule does not resolve existing alarms — it only prevents new ones.
  • Severity overrides apply to new alarms only. Existing alarms keep their original severity.
  • Rules are re-seeded on boot. New rules appear automatically after a code update.
  • The default severity is immutable — it reflects the code-defined importance of the fault.

API calls (3)

MethodPathDescription
GET /api/admin/alerting/alarm-rules List all built-in alarm rules
PATCH /api/admin/alerting/alarm-rules/:id Toggle enabled or override severity
POST /api/admin/alerting/alarm-rules/:id/reset Reset severity to default

Related

  • Scanner CA Rollover — The rollover the stall / failed / emergency-transition alarms watch
  • Alarms — Rules create alarms when matching events arrive
  • Events — Events are matched against rules to create alarms
  • Notification Policies — Policies can be scoped to specific alarm rule names