mipo
Menu
ProductFeaturesArchitectureSecurityComplianceDocs

Scanner Resource Exhausted

A probe failed because the SCANNER HOST ran out of a local resource — file descriptors, socket buffers, local ports, or the kernel neighbour (ARP/NDP) table — or was blocked locally. Such a probe says nothing about the target but is indistinguishable from a closed port, so the scanner fails the whole job instead of recording results that would write false open-to-closed changes into the audit trail.

Fields & columns

NameDescription
kindWhat ran out: fd, buffers, local_ports, local_firewall, interface_down, neighbour_table, or unknown (an errno the scanner does not classify — the raw text is in the message)
messageThe scanner's full report: SCANNER_RESOURCE_EXHAUSTED(kind), the cause and address, how many of the planned probes had run, and the remedy

How to

Diagnose

  1. Read the job error's message — it names the resource, the errno and the remedy
  2. On the scanner host, read the startup self-check lines: journalctl -u mipo-scanner | grep selfcheck
  3. For kind=neighbour_table, read the kernel counters on the HOST: cat /proc/net/stat/arp_cache (and ndisc_cache) — the table_fulls and unresolved_discards columns
  4. For kind=fd, compare the process limit with the worker count: grep "Max open files" /proc/$(pgrep -f mipo-scanner)/limits

Fix on the scanner host

  1. Raise the limit that matches the kind: LimitNOFILE via a systemd drop-in (fd), net.ipv4.neigh.default.gc_thresh3 and its ipv6 twin (neighbour_table), net.ipv4.ip_local_port_range (local_ports), or the host socket-buffer limits (buffers), OR
  2. Lower MIPO_SCAN_CONCURRENCY / raise MIPO_SCAN_TIMEOUT in /etc/mipo-scanner.env — the load is the probe RATE, roughly concurrency ÷ timeout
  3. Restart: sudo systemctl restart mipo-scanner, then confirm the selfcheck lines report enough headroom

Clear the error

  1. Acknowledge the job error — the job is terminal and re-running it on an unfixed host would fail again
  2. The alarm resolves itself once that scanner completes its next job

Gotchas

  • The scan results from a failed job are NOT used as a baseline and cannot close a port — the reconciler excludes failed jobs from participation. A raw job still keeps the genuine open ports it had already reported.
  • A delta (compacted) job sends NO receipt when it fails this way, so the next scan compares against the previous baseline.
  • The neighbour-table counters are HOST-WIDE: another workload overflowing the table also fails this job. That is deliberate — if the table overflowed at all, this scan's accuracy was at risk.
  • Neighbour-table overflow is only detectable from the host's own network namespace. A scanner in a container namespace logs "not verifiable here" at startup and cannot catch it — run such a scanner with --network host.
  • Conntrack is NOT one of these kinds. The conntrack table is checked only at STARTUP, where a table too small for the configured probe rate makes the scanner refuse to start. A conntrack table that fills DURING a scan drops the SYN before it leaves the host, so the probe just times out like a dead host — which is exactly why the startup check is sized for the worst case rather than relying on per-probe detection. See docs/operations/scanner-install.md "Kernel limits for fast scans".
  • kind=unknown means the scanner met an errno it does not classify. It fails the job on purpose: an unexplained error must never be recorded as a closed port. The raw errno is in the message.
  • The scanner REFUSES TO START when its configured probe rate cannot fit a kernel limit, so a host that trips this at startup never scans at all — check journalctl for the refusal.

API calls (2)

MethodPathDescription
GET /api/health/job-errors List active job errors, including this class
POST /api/health/job-errors/:stateId/repair action=acknowledge

Related