Enabled | Whether this rule creates alarms. Disabled rules ignore matching events. |
Name | Unique rule identifier (e.g., scanner-down, tls-expiring) |
Description | Human-readable explanation of what the rule detects |
Severity | Current severity for new alarms. Can be overridden from the default. |
Auto-Resolve | Event type that auto-resolves matching alarms (null = manual only) |
scanner-down | Availability: fires on scanner_down (default: high), auto-resolves on scanner_up. Scanner missed heartbeat timeout (2 minutes). |
ingest-down | Availability: fires on ingest_down (default: critical), auto-resolves on ingest_up. Ingest API node stopped heartbeating. |
db-down | Availability: fires on db_down (default: critical), auto-resolves on db_up. Database connection pool test failed. |
proxy-down | Availability: fires on proxy_down (default: critical), auto-resolves on proxy_up. Traefik reverse proxy health check failed. |
dns-down | Availability: fires on dns_down (default: high), auto-resolves on dns_up. Public FQDN DNS resolution failed. |
backup-failed | Operations: fires on backup_failed (default: high), manual resolve only. Manual or scheduled backup completed with errors. |
backup-schedule-down | Operations: fires on backup_schedule_down (default: high), auto-resolves on backup_schedule_up. A backup schedule is enabled but the newest bundle is older than 26 hours (or an install older than that has never produced one) — checked against the backup HTTP API bundle list every evaluator tick. |
alarm-storm | Operations: fires on alarm_storm_detected (default: high), auto-resolves on alarm_storm_normal. More than 500 alarm rows were opened in the last hour — a runaway signal or a stuck evaluator cursor (see ops/runbooks/alarm-storm.md). |
dns-resolver-group-unavailable | Reverse DNS: fires on dns_resolver_group_unavailable_detected (default: high), auto-resolves on dns_resolver_group_unavailable_normal. Lookups for a scan have waited more than 120 seconds with no online resolver in the scan's resolver group — checked per resolver group by the dispatcher (see ops/runbooks/reverse-dns.md). |
dns-enrichment-backlog | Reverse DNS: fires on dns_enrichment_backlog_detected (default: warning), auto-resolves on dns_enrichment_backlog_normal. A reverse-DNS job in the resolver group has waited unclaimed for more than 30 minutes (twice the enrichment window), which means the resolver group is not keeping up with its work (see ops/runbooks/reverse-dns.md). |
db-disk-high | Threshold: fires on db_disk_high (default: warning), auto-resolves on db_disk_normal. Database disk usage exceeded threshold. |
db-connections-high | Threshold: fires on db_connections_high (default: warning), auto-resolves on db_connections_normal. Database connection pool has sustained waiting queries. |
tls-expiring | Deadline: fires on tls_expiring (default: warning), auto-resolves on tls_renewed. TLS certificate expires within 30 days. |
scanner-certificate-expiry | Deadline: fires on scanner_certificate_expiry_high (high), auto-resolves on scanner_certificate_expiry_normal. Scanner mTLS certificate expires within 5 days. |
scanner-ca-rollover-stall | Stall: fires on scanner_ca_rollover_stall_high (default: high), auto-resolves on scanner_ca_rollover_stall_normal. A CA rollover has made no phase advance and no change to its outstanding blocker set for 24 hours. A nonterminal rollover holds the one-operation slot that blocks every future rollover and the migration-to-required mTLS policy flip, so a stalled one is a blocked platform, not a paused job. |
scanner-ca-rollover-failed | Operations: fires on scanner_ca_rollover_failed (default: high), manual resolve only. A step of an orderly CA rollover threw. A retryable lock timeout deliberately does NOT raise this — the background driver retries contended locks every 5 seconds on its own. |
scanner-ca-emergency-transition-failed | Operations: fires on scanner_ca_emergency_transition_failed (default: critical), manual resolve only. A step of the emergency CA-compromise transition threw. The emergency path leaves the fleet with no trusted issuer between containment and promotion, so this alarm means the fleet may be sitting in that gap. |
scanner-load-high | Resource: fires on scanner_load_high (default: warning), auto-resolves on scanner_load_normal. Scanner load average exceeds 80% of CPU capacity. |
scanner-memory-high | Resource: fires on scanner_memory_high (default: warning), auto-resolves on scanner_memory_normal. Scanner available memory below 10%. |
db-sessions-high | Database: fires on db_sessions_high (default: warning), auto-resolves on db_sessions_normal. Active database sessions exceed 80% of max_connections. |
db-long-queries | Database: fires on db_long_queries_high (default: warning), auto-resolves on db_long_queries_normal. Database query running longer than 60 seconds. |
auth-failures-high | Security: fires on auth_failures_high (default: high), auto-resolves on auth_failures_normal. More than 10 authentication failures in 5 minutes. |
session-ip-spread | Security: fires on session_ip_spread_high (default: warning), manual resolve only. One user account has active sessions from too many distinct IP addresses. |
scan-stuck | Stall: fires on scan_stuck (default: warning), manual resolve only. Scan still running but all jobs are finished. |
scanner-heartbeat-failed | Security: fires on scanner_heartbeat_failed (default: warning), manual resolve only. Scanner heartbeat authentication failed — invalid API key or IP binding violation. Requires manual acknowledgement; a scanner can resume heartbeating successfully while prior auth failures remain in scope. |
dispatcher-down | Availability: fires on dispatcher_down (default: critical), auto-resolves on dispatcher_up. Dispatcher service stopped heartbeating for more than 2 minutes. When the dispatcher is down, no new scan jobs are dispatched and active scans stall. |
backup-down | Availability: fires on backup_down (default: warning), auto-resolves on backup_up. Backup container is unreachable via DNS resolution. Typically indicates the backup container has stopped or failed to start. |
manager-memory-high | Resource: fires on manager_memory_high (default: warning), auto-resolves on manager_memory_normal. Manager heap usage exceeded 80% of the 512 MB container memory limit. Sustained high memory may precede an OOM termination; consider restarting the manager or investigating large in-flight requests. |
container-recovery-failed | Infrastructure: fires on container_recovery_failed (default: critical), manual resolve only. A container did not recover within 2 minutes after the watchdog attempted a restart. Indicates a persistent crash loop or configuration error that automatic healing cannot fix. |
container-restart-storm | Infrastructure: fires on container_restart_storm (default: high), manual resolve only. A container restarted more than 3 times within a 15-minute window. Suggests a crash loop; investigate container logs before allowing further restarts. |
container-down | Infrastructure: fires on container_down (default: high), auto-resolves on container_up. Container is running but not reachable over the Docker internal network — health check connections are refused or timing out. |
container-restarted | Infrastructure: fires on container_restarted (default: info), manual resolve only. Container was restarted by the watchdog autoheal mechanism. Informational — the restart itself is the resolution of the underlying fault. Review logs if restarts become frequent. |
tls-storage-unwritable | TLS: fires on tls_storage_unwritable (default: high), auto-resolves on tls_storage_recovered. TLS cert/config storage or the watchdog pointer directory is not writable — TLS mutations are refused (503) until recovered. |
tls-activation-unverifiable | TLS: fires on tls_activation_unverifiable (default: critical), manual resolve only. TLS cert activation failed AND the rollback could not be verified served — TLS is in an unknown state, manual intervention required. |
cancelling-scanner-dead | Custody: fires on cancelling_scanner_dead (default: warning), manual resolve only. A cancelling scan had jobs on a heartbeat-dead scanner; the reconciler force-cancelled them to converge the cancel. |
reconciler-lag-high | Custody: fires on reconciler_lag_high (default: high), auto-resolves on reconciler_lag_normal. State-change reconciler has a backlog of completed scans whose diffs have not been computed within the threshold. |
audit-chain-divergent | Audit: fires on audit_chain_divergent (default: critical), auto-resolves on audit_chain_consistent. The audit log hash chain no longer verifies — at least one row does not match its recorded hash or its link to the previous row; forensic evidence may have been altered. |
audit-write-failing | Audit: fires on audit_write_failing (default: critical), auto-resolves on audit_write_recovered. Audit-write circuit breaker is OPEN — ordinary mutations are rejected (503) until the audit DB path recovers. |
results-on-ineligible-parent | Custody: fires on results_on_ineligible_parent (default: high), manual resolve only. Computed diffs found on a scan that is not completed — the diffing atomic boundary may have been breached. |
never-started-job-has-evidence | Custody: fires on never_started_job_has_evidence (default: high), manual resolve only. A job assigned but never server-started has accepted result/receipt evidence — a chain-of-custody invariant violation (likely a pre-custody legacy row); auto-redispatch is withheld. |
never-started-redispatch-exhausted | Custody: fires on never_started_redispatch_exhausted (default: warning), manual resolve only. A job was auto-redispatched the maximum number of times and still never started — left for operator repair. |
stale-scanner-evidence-detected | Custody: fires on stale_scanner_evidence_detected (default: high), manual resolve only. Accepted scanner evidence exists for a job the server never recorded as started — a chain-of-custody invariant violation; evidence is preserved for operator investigation. |
container-restart-failing | Infrastructure: fires on container_restart_failing (default: high), auto-resolves on container_restart_recovered. Watchdog autoheal keeps failing to restart an exited/unhealthy container (the Docker error is in the event details). |
backup-container-restarted | Operations: fires on backup_container_restarted (default: high), auto-resolves on backup_container_recovered. Watchdog autoheal restarted the backup container; scheduled backups may be stalled or masked by self-healing. |
resolver-down | Availability: fires on resolver_down (default: high), auto-resolves on resolver_up. Reverse-DNS resolver missed heartbeat timeout (2 minutes). |
dns-nameserver-failing | Reverse DNS: fires on dns_nameserver_failing_detected (default: warning), auto-resolves on dns_nameserver_failing_normal. Over half the reverse-DNS lookups against a nameserver group failed (nameserver error or timeout) in the last 5 minutes. |
tls-scanner-ingress-divergent | TLS: fires on tls_scanner_ingress_divergent (default: high), auto-resolves on tls_scanner_ingress_consistent. The scanner ingress (:8443) does not serve the published scanner anchor, or its anchor, pointer, hostname or Public URL is inconsistent; scanners may be unable to connect. |
tls-activation-rolled-back | TLS: fires on tls_activation_rolled_back (default: high), auto-resolves on tls_cert_installed. A TLS certificate activation failed served verification and was rolled back to the previous generation. |
dns-zone-divergent | TLS: fires on dns_zone_divergent (default: high), auto-resolves on dns_zone_consistent. A DNS provider zone write was lossy — compensation restored the pre-write records; confirm the zone. |
tls-renewal-failing | TLS: fires on tls_renewal_failed (default: warning), auto-resolves on tls_renewal_succeeded. Automatic renewal of a certificate profile is failing — the bound certificate will eventually expire. |