mipo
Menu
ProductFeaturesArchitectureSecurityComplianceDocs

Scanner Job Dispatch

Scanner job dispatch is the pull-based protocol that transfers scan assignments from the dispatcher to each scanner. Scanners poll for pending jobs, claim them, report progress, and submit completion — the ingest nodes coordinate this lifecycle. Understanding the job protocol is essential when debugging stalled scans, investigating scanner connectivity issues, or integrating a custom scanner implementation.

Fields & columns

NameDescription
jobIdUnique integer identifying a scan job. Stable across the job lifecycle.
scanIdThe parent scan this job belongs to. Multiple jobs may share the same scanId when a scan spans multiple scanners.
targetsArray of IP ranges and port combinations assigned to this job.
maxProbesPerSecondEffective rate limit for this job, computed from the scanner, subnet, and port list rate limits using slowest-wins logic.
progressCountNumber of probes (one IP + one port) completed so far across the whole job (sent in progress updates). The server divides it by the job's planned probe count for the percentage shown on the scan.
resultCountNumber of results recorded so far (sent in progress updates and completion).
statusFinal job outcome: "completed" (scanner finished successfully), "failed" (scanner reported an error) or "cancelled" (the scanner acknowledges an operator cancel; accepted only while the parent scan is cancelling).
errorMessageOptional human-readable error description sent with a "failed" completion.

How to

Understand the job lifecycle

  1. Dispatcher creates pending jobs in the jobs database for each scanner assigned to a scan.
  2. Scanner calls GET /scanner/jobs to claim and receive pending assignments. Uses FOR UPDATE SKIP LOCKED for concurrent-safe claiming across multiple ingest nodes.
  3. Scanner calls POST /scanner/jobs/:jobId/start to mark the job as running.
  4. Scanner calls POST /scanner/jobs/:jobId/progress periodically with progressCount and resultCount as it scans.
  5. Scanner calls POST /scanner/jobs/:jobId/complete with status "completed", "failed" or "cancelled" (only while the parent scan is cancelling) and a final resultCount.
  6. When all jobs for a scan finish, the ingest node marks the parent scan as completed or failed.

Understand the reverse-DNS job lifecycle (resolvers)

  1. When a scan that requests reverse DNS discovers hosts, the dispatcher bundles their PTR lookups into DNS jobs for the scan's resolver group (up to 64 lookups per job).
  2. A resolver calls GET /scanner/dns/jobs to claim up to 4 jobs. Claiming IS starting: each job comes back assigned to that resolver with a lease running, and the resolver's heartbeat (activeDnsJobIds) renews the lease.
  3. As lookups finish, the resolver calls POST /scanner/dns/jobs/:dnsJobId/observations with numbered batches of up to 64 observations. The server files every item as accepted, duplicate, stale, conflicting or rejected; a replayed batch is answered "duplicate" rather than stored twice.
  4. When the job is done the resolver calls POST /scanner/dns/jobs/:dnsJobId/receipt with a signed receipt. The signature is mandatory: the server recomputes every hash and count before storing it, and the receipt is what the Execution Proof page lists for the job.
  5. A job whose lease lapses (the resolver stopped heartbeating) is marked expired, and its unfinished lookups return to the queue to be bundled into a new job for any online member of the group; lookups still open when the scan's 15-minute window ends are expired and the scan's reverse DNS closes incomplete.

Debug a stalled scan

  1. Check the scan status via GET /api/scans/:id — if it shows "running" for an extended period, jobs may be stuck.
  2. Check scanner heartbeat status in Config > Scanners to confirm the scanner is online.
  3. Verify the scanner process is running and can reach the ingest endpoint (port 443).
  4. A scan marked stuck will trigger the scan-stuck alarm after all jobs are finished but the scan remains in "running" state.

Gotchas

  • Job ownership is enforced: a scanner can only start, update, or complete jobs assigned to it. Attempting to modify another scanner's job returns 403.
  • Scanners authenticate using the X-API-Key header (not session cookies). A missing or invalid key returns 401.
  • IP binding is checked on job fetches (GET /scanner/jobs). A scanner connecting from an unexpected IP will receive 403 with code IP_BINDING_VIOLATION.
  • The limit query parameter on GET /scanner/jobs defaults to 1 and caps at 10. Scanners should claim one job at a time unless they are designed to parallelize.
  • If a scanner disappears mid-job (heartbeat timeout), the job remains in "running" state. The dispatcher may re-assign it after a configurable timeout.
  • All job endpoints require a valid API key. Disabled scanners receive 403 on all job operations.
  • Job routes are scoped to the identity kind. A resolver identity calling GET /scanner/jobs is refused with 403 WRONG_IDENTITY_KIND before any claim is made, and a port scanner is refused the same way on the /scanner/dns/jobs routes — the two kinds share one identity table and one authentication path, but never each other's work.

API calls (7)

MethodPathDescription
GET /scanner/jobs Claim and receive pending jobs for this scanner. Uses X-API-Key header. Optional ?limit=1 parameter (max 10). Returns { jobs: [...] }.
POST /scanner/jobs/:jobId/start Transition a claimed job to running state. Called immediately after the scanner begins work. Returns { ok: true }.
POST /scanner/jobs/:jobId/progress Stream incremental progress. Body: { progressCount, resultCount }. Call periodically during scanning. Returns { ok: true }.
POST /scanner/jobs/:jobId/complete Finalize a job. Body: { status: "completed"|"failed"|"cancelled", resultCount?, errorMessage? }. "cancelled" is accepted only while the parent scan is cancelling. Triggers scan completion check when all jobs finish.
GET /scanner/dns/jobs Resolver identities only: claim reverse-DNS jobs for this resolver. Optional ?limit=1 (max 4). Claiming is starting — each returned job is assigned with a lease the heartbeat renews. Returns { jobs: [...] }.
POST /scanner/dns/jobs/:dnsJobId/observations Resolver identities only: report lookup results for one claimed job. Body: { batchSeq, observations: [...] } (max 64 per batch). Returns each observation index under accepted, duplicate, stale, conflicting or rejected; a 200 is final even when every item was refused.
POST /scanner/dns/jobs/:dnsJobId/receipt Resolver identities only: submit the signed execution receipt that closes the job. Body: { scanId, dnsJobId, taskSpecHash, receipt, signature } — the signature is mandatory and the server recomputes every hash and count before storing. Returns { ok, outcome, receiptId, jobStatus }.

Related

  • Scanner Heartbeat — Heartbeat and job protocol are the two core scanner-to-ingest communication channels
  • Scanners — Scanner provisioning and configuration governs which jobs each scanner receives
  • Recent Scans — Scan status reflects the aggregate outcome of all its component jobs
  • Resolvers — The identity kind that claims the reverse-DNS job routes described here