Automatic Linux ISF resolution, YARA scan scope + rule de-duplication, plugin catalog maintenance, and dumpfile fix (#45)

[imb0ru]

Feature: automatic Linux kernel symbol (ISF) resolution

Linux memory analysis in Volatility3 requires an Intermediate Symbol File (ISF) matching the exact kernel banner of the image. Until now the analyst had to build that ISF and upload it manually before any Linux plugin could run. This adds end to end automatic resolution.

How it works

When a Linux evidence is added, a background task:

1. detects the kernel banner (banners.Banners),
2. resolves a matching ISF from the community index (Abyss-W4tcher/volatility3-symbols, banners_plain.json) and downloads it,
3. verifies it against the image by actually running linux.pslist,
4. registers it as a Symbol (so it shows up in the Symbols page) and marks the evidence ready.

Plugin extraction and YARA scanning for Linux evidences are gated until symbols are ready. If no ISF can be imported, the UI shows the detected banner plus precise manual build guidance (the exact dbgsym package and dwarf2json steps, with an Ubuntu specific package/URL derived from the banner). Deleting the ISF re-validates and re-closes the gate, and a re-fetch button plus manual upload re-verification are available.

Changes

Backend

• backend/volatility_engine/models.py: new LinuxSymbolResolution model (status detecting / resolving / verifying / ready / failed_banner / failed_isf, plus banner, method and manual guidance).
• backend/volatility_engine/isf.py: remote index lookup (exact match, then kernel version + build fallback), download into the Volatility symbol path, and manual build guidance builder.
• backend/volatility_engine/engine.py: detect_linux_banner() and verify_linux_symbols().
• backend/volatility_engine/tasks.py: generate_linux_symbols (auto on evidence creation) and reverify_linux_symbols (re-validates on manual ISF upload or deletion, opening or closing the gate).
• backend/volatility_engine/views.py: SelectiveExtractionTask and YaraScanTask return 409 for Linux until the ISF is ready; new IsfResolutionView (GET/POST /api/evidence/<id>/isf/) for status and on demand re-fetch. Triggers wired in evidences/signals.py (creation) and symbols/signals.py (upload/deletion).
• New versioned migrations for volatility_engine (0002_volatilityplugin_error_message, 0003_linuxsymbolresolution).

Frontend

• frontend/src/components/Investigate/PluginSelector.tsx: live ISF status panel (banner, guidance, Retry), a success toast when symbols are ready, and the "Start Analysis" button disabled until ready (Linux only).

---

Maintenance: Volatility3 plugin catalog updates + admin diff tool

The curated plugin list (volweb_plugins.json / volweb_misc.json) had drifted from the pinned volatility3 version, which had reorganized many plugins into <os>.malware.* and emitted deprecation FutureWarnings.

• Moved 11 Linux and 12 Windows plugins to their malware namespace, fixing two class renames along the way (linux.tty_check.tty_check to linux.malware.tty_check.Tty_Check, windows.unhooked_system_calls.unhooked_system_calls to windows.malware.unhooked_system_calls.UnhookedSystemCalls). This clears the deprecation warnings.
• Added new curated plugins: Windows vadinfo, mftscan, etwpatch, registry.printkey.
• New staff only, read only admin view "Plugin catalog diff" (backend/volatility_engine/admin.py): compares the curated list against the plugins actually exposed by the installed volatility3 (framework.list_plugins()), automatically hiding deprecated aliases and YARA scanning plugins (handled separately). It has zero runtime impact and does not touch the shared context optimization; it just makes updating the curated list deliberate when the pinned volatility3 version is bumped.

---

Bug fix: dump_file task 'int' object is not iterable (ISSUE #45)

The error occurred when clicking the "Dump" button on a file found by the FileScan plugin.

In Volatility3, the virtaddr and physaddr parameters of the DumpFiles plugin are defined as ListRequirement(element_type=int), meaning they expect a list of integers. The code was passing a plain int(offset) instead, causing the 'int' object is not iterable error during the automagic execution phase.

Fix: In backend/volatility_engine/engine.py, in both dump_file_windows and dump_file_linux methods, the values passed to the configuration are now wrapped in a list:

Before

self.context.config["plugins.DumpFiles.virtaddr"] = int(offset)
self.context.config["plugins.DumpFiles.physaddr"] = int(offset)

After

self.context.config["plugins.DumpFiles.virtaddr"] = [int(offset)]
self.context.config["plugins.DumpFiles.physaddr"] = [int(offset)]

---

Feature: YARA scan scope selection (process memory vs kernel)

Adds an explicit choice between two complementary Volatility plugins when running a YARA scan, fixing a silent zero matches bug on user space malware (e.g. ransomware payloads).

Problem

The original implementation invoked volatility3.plugins.yarascan.YaraScan, which operates only on the primary (kernel) layer of the dump. As a result, scans against memory dumps containing well known user space malware returned 0 matches even when the payload was known to be present. Every artefact that lives in user space (mutex strings, command lines, registry path strings, encryption routine constants, PE images of injected modules) was outside the scanned region.

Fix

Introduces a scan_scope parameter end to end that selects the appropriate Volatility plugin, per OS:

Scope	Plugin	Use case
"vad" (default)	windows.vadyarascan.VadYaraScan (Windows, VADs) / linux.vmayarascan.VmaYaraScan (Linux, VMAs)	Walks every process's memory. Recommended for malware/ransomware.
"kernel"	yarascan.YaraScan	Scans the primary kernel layer. For rootkits / kernel mode threats.

Windows regions are VADs, Linux regions are VMAs, so the two OSes expose the same per process concept under different plugin names.

Changes

Backend

• backend/volatility_engine/engine.py: run_yara_scan(..., scan_scope="vad") picks the plugin class and config prefix based on scope and evidence OS. The yara_file config is set under both the per process prefix and plugins.YaraScan to cover the embedded scanner it delegates to.
• backend/volatility_engine/tasks.py: start_yarascan(..., scan_scope="vad") threads the parameter through all invocation paths and logs the active scope.
• backend/volatility_engine/views.py: YaraScanTask reads and validates scan_scope against {"vad", "kernel"}.

Frontend

• frontend/src/components/Investigate/YaraScan.tsx: scanScope state (default "vad"), a RadioGroup above the rulesets/rules panels. The "Process memory" option is OS aware and shows the correct plugin id and region (VAD on Windows, VMA on Linux). The scope is sent as scan_scope in the scan payload and the control is disabled while a scan runs.

Backwards compatibility

The parameter is optional with a sensible default ("vad"), so existing clients keep working.

---

Bug fix: Linux process memory YARA scan referenced a non existent plugin

The engine selected linux.vadyarascan.VadYaraScan for the Linux "vad" scope, but Linux has no vadyarascan (it uses VMAs, linux.vmayarascan.VmaYaraScan), so a Linux process memory YARA scan raised an ImportError. Windows was unaffected. Fixed to use VmaYaraScan on Linux, and the frontend scope label was made OS aware accordingly.

---

Bug fix: YARA scan error handling + duplicate-identifier crash

Duplicate rule identifiers aborted the whole scan

When several rules (typically from overlapping community rulesets) declare the same identifier, yara.compile raises a duplicated identifier SyntaxError and the entire scan fails. backend/volatility_engine/engine.py now de-duplicates the combined rules before compilation (_dedupe_yara_rules): it keeps the first occurrence of each rule identifier, collapses duplicate import/include statements, and logs the dropped identifiers. The scan then runs on the union of the distinct rules instead of failing outright.

Failed scans were reported as success

run_yara_scan now has a clear contract (True = matches, False = no matches, None = could not run) and raises on real failures instead of swallowing them. start_yarascan treats None/failure as an error and notifies the UI accordingly, so a broken scan no longer shows a phantom "completed successfully".

---

Feature: content-based de-duplication of YARA rules

Previously rule de-duplication happened only at scan time (by identifier, in the engine). Nothing prevented the same rule from being stored many times. This adds global, content based de-duplication across the whole database.

How it works

• New content_hash field on YaraRule (backend/yararules/models.py): a normalized SHA-256 of rule_content (stable against line ending / whitespace / blank line differences), kept in sync automatically in save(). Indexed, not unique, so uniqueness is enforced in application code and legacy data with pre existing duplicates migrates cleanly.
• backend/yararules/utils.py: compute_content_hash, backfill_content_hashes (idempotent) and get_existing_content_hashes.

Behaviour by entry point

Action	Result
Import (file upload / GitHub) of a duplicate	skipped and reported (response includes skipped_duplicates)
Manual create with identical content	400 with a clear message naming the existing rule
"Save as Copy" without changing the content	400 (it is a duplicate)
Editing name / ruleset / description without touching the content	allowed
Editing content so it matches another rule	400

Bulk imports skip and report (so one duplicate does not fail an import of hundreds of rules), while a single manual submission returns an explicit error.

Changes

Backend

• backend/yararulesets/views.py: the chunked upload (CompleteUploadView) and GitHub import (GitHubImportView) check each rule's content hash against the DB and against rules seen earlier in the same batch, skip duplicates, count them, and return skipped_duplicates. An etag IntegrityError is now caught and treated as a duplicate.
• backend/yararules/serializers.py: YaraRuleSerializer.validate rejects manual create/update whose content duplicates an existing rule (excluding the instance being edited).

Frontend

• frontend/src/components/Dialogs/YaraRuleCreationDialog.tsx + frontend/src/components/Lists/YaraRuleList.tsx: import dialogs report e.g. "Upload complete: N created, M duplicate(s) skipped".
• frontend/src/components/Dialogs/YaraRuleEditDialog.tsx: the save error handler surfaces DRF field errors (the duplicate content message) instead of a generic "Request failed with status code 400".

---

Infrastructure fix: versioned migrations + single migrator

Adding schema changes (the content_hash column, then the LinuxSymbolResolution model) exposed two pre existing problems.

Versioned migrations

yararules, yararulesets and volatility_engine relied on the entrypoint regenerating 0001_initial on every boot. Django tracks applied migrations by name, so once 0001_initial was recorded, any newly added field or model was never materialized and the schema silently drifted (DuplicateColumn on rebuild). This PR commits real, versioned migrations:

• backend/yararulesets/migrations/0001_initial.py
• backend/yararules/migrations/0001_initial.py + 0002_yararule_content_hash.py
• backend/volatility_engine/migrations/0002_volatilityplugin_error_message.py + 0003_linuxsymbolresolution.py

The initial migrations reflect the original models (matching what existing databases already have applied) and the incremental ones add the new schema, so changes apply correctly on both fresh and existing deployments.

Single migrator (race condition)

The entrypoint ran makemigrations + migrate in every service (backend + both workers) at startup, so concurrent migrate runs raced on the same DDL.

• backend/entrypoint.sh: migrations / collectstatic / initadmin now run only when RUN_MIGRATIONS=true.
• docker-compose.yaml / docker-compose-prod.yaml: only volweb-backend sets RUN_MIGRATIONS=true; the workers depends_on the backend so the schema is migrated first.

[forensicxlab]

Hi @imb0ru thanks for the PR !
Will test and merge in the coming days 👍

Cheers !