Skip to content

Make the YARA scanner asynchronous to avoid blocking the event loop #704

Description

@rabbitstack

Motivation

The YARA scanner currently runs Scan() synchronously on the caller's goroutine, which means every scan, including the underlying libyara call executes inline on the ETW event processing path. This couples scan latency directly to event throughput: a slow or backed-up scan stalls the event loop rather than just delaying the scan result, which is a problem under high event volume or when scanning larger payloads (e.g. ADS reads).

Expectation

Scanning should no longer block event processing.

  • Scan() returns immediately after enqueuing the work; scan results reach consumers only through the existing alert-sender pipeline
  • Under load, once the pending-scan queue is full, new scan requests are dropped (not blocked), and the drop is observable via a metric
  • The number of concurrent scan workers is bounded and configurable
  • A Wait()/Flush() mechanism allows draining in-flight scans deterministically (used by tests, and by shutdown)

Proposal

  • Introduce a bounded queue (buffered, size controlled by a new AsyncQueueSize config option, default 1024). Scan() does a non-blocking send. A full channel drops the event and increments a yara.dropped.scans metric
  • Add a semaphore (sem chan struct{}, capacity MaxWorkers, default 32, also configurable), gating how many scan goroutines run concurrently; a single dispatch() goroutine acquires a slot before launching each worker
  • Track every launched goroutine with a drainWg sync.WaitGroup so Close() can drain all in-flight work before destroying the YARA compiler, avoiding calls into libyara after Destroy()
  • Extract the eligibility/setup logic currently inline in Scan() into a function that runs synchronously on the caller's goroutine (fast path); only the actual yara.Scanner invocation moves into the worker pool. This includes making the ADS sys.ReadFile read synchronous (capped at 1MB with a 1s timeout) so scan bytes are captured before the event may be recycled
  • Protect the rwxs/mmaps maps with a mutex, since worker goroutines now write to them concurrently
  • Update the Scan() interface contract: it currently returns (bool, error) with the bool indicating a match, which is incompatible with async execution. It should always return (false, nil), with match results delivered exclusively via the alert-sender pipeline
  • Add new config fields for AsyncQueueSize and MaxWorkers

Here is the proposed architecture diagram:

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs: configIndicates the issue requires changes in the config file/flagsscope: yaraAnything related to libyara and pattern matchingtype: enhancement

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions