build(nix): modular flake.nix + ./nix static-analysis harness

[daveseddon-runpod]

What

Adds a small root flake.nix that wires up modular ./nix/ modules, giving a reproducible, pinned static-analysis surface for this repo — no global tool installs needed.

Opened as a draft so we can watch the new Nix static analysis workflow run and see what the harness surfaces.

Layout

• flake.nix — thin entry point (flake-utils.eachSystem), src filter, formatter = nixfmt-rfc-style.
• nix/lib{,/*} — shared builders: mk-check (gated derivations), mk-lint (per-tool apps), mk-all-app (composites), maybe-tool (graceful skip when a tool is absent from the pin).
• nix/checks.nix — sandbox-safe gates for nix flake check.
• nix/lints/* — per-tool apps + domain composites (lint-shell/-yaml/-json/-py/-nix-files/-docker/-md/-secrets/-sast/-supply/-spelling) + lint-extreme + everything.
• nix/devshell/ — nix develop toolchain.
• Configs: .yamllint.yaml, .markdownlint.yaml, pyproject.toml; docs in nix/README.md.
• .github/actionlint.yaml — declares the repo's Blacksmith self-hosted runner labels.

CI

.github/workflows/nix.yml installs Nix and runs nix flake check --keep-going — the gated, sandbox-safe subset:
nixfmt / statix / deadnix / nil, shellcheck, yamllint, actionlint, hadolint, markdown, taplo, editorconfig-checker, gitleaks.

Network / dependency-heavy tools (trivy, osv-scanner, semgrep, lychee, trufflehog, pyright, pylint) are exposed as nix run .#lint-* apps rather than CI gates.

Usage

nix develop            # shell with every tool
nix flake check        # gated subset (what CI runs)
nix run .#lint-py      # a whole domain
nix run .#everything   # flake check + every manual lint

Heads-up

This runs the linters against the existing tree with pragmatic defaults, so the first CI run is expected to surface pre-existing findings (e.g. hadolint DL3008, editorconfig/whitespace, shellcheck style). That output is the point — it shows coverage. We can then decide what to tune (per-tool configs) vs. fix before making any of it a required check.

Known coverage gaps

HCL (docker-bake.hcl), INI (grafana.ini), and CSV have no dedicated linter wired in (documented in nix/README.md); large generated JSON is validated for well-formedness only.

[daveseddon-runpod]

First CI runs — results

The Nix static analysis workflow runs nix flake check --keep-going and completes in ~40s (Nix install + all gated checks, uncached). It surfaced findings immediately, and I've pushed fixes for the two that were harness bugs / false positives. Current state:

✅ 10 gated checks passing: nixfmt, statix, deadnix, nil, shellcheck, taplo, editorconfig, actionlint, gitleaks, hadolint

❌ 2 remaining — pre-existing style findings (team decision to fix vs. relax):

What got fixed along the way

• actionlint — was failing with no project was found: the mkCheck sandbox stages the source without .git, which actionlint needs to locate workflows + read config. Fixed by git init-ing a throwaway repo in the check (nix/checks.nix).
• gitleaks — flagged .github/actions/smoke-test/action.yml:205, which is a -----BEGIN RSA PRIVATE KEY----- string literal in a case statement that validates a user-supplied SSH key — not a committed secret. Added .gitleaks.toml allowlisting that file + PEM-header pattern (matchCondition = AND, so a real key committed elsewhere still trips it).
• hadolint — aligned with the repo's existing policy via .hadolint.yaml ignoring DL3006/3008/3013/3022/3042 (the same codes hadolint-pr.yml / hadolint-push.yml already ignore per-Dockerfile).

Remaining findings

yamllint — 19 issues, all trivial hygiene:

• 13× trailing spaces
• 5× missing newline at end of file
• 1× CRLF line ending

markdownlint — 130 issues, style: top rules MD060 table-column-style (40), MD012 multiple-blanks (28), MD032 blanks-around-lists (14), MD022 blanks-around-headings (11), MD040 fenced-code-language (8), …

Recommendation: the yamllint set is a cheap cleanup pass (whitespace only). The markdown set is larger — either a markdownlint-cli2 --fix pass or relax a few more rules in .markdownlint.yaml. Both are pre-existing (not introduced here), so I've left them visible rather than papering over them; happy to do a cleanup commit or dial the configs back, whichever you prefer. Until then these checks stay red by design — the workflow shouldn't be a required gate yet.

[daveseddon-runpod]

Experimental: Nix-built container image (build-only)

Added a proof-of-concept for building the images with Nix (pkgs.dockerTools.streamLayeredImage) instead of Dockerfiles, purely to measure the win. Nothing is published — a new build-only image-size CI job builds the image and prints this comparison (reproduced on every run):

image	compressed (download)	uncompressed	layers
nix base-cpu (Python 3.14)	295 MB	921 MB	~98
runpod/base:1.0.7-ubuntu2404 (Dockerfile)	714 MB	—	20

≈ 59% smaller compressed download, and the stream is byte-identical across two builds (deterministic — CI asserts it via sha256 ×2).

What's in it

nix/images/base-cpu.nix → Python 3.14 (jupyterlab/notebook/ipywidgets/hf_transfer) + uv/filebrowser/nginx/openssh + the common CLI tooling + the repo's start.sh. Exposed as packages.<system>.base-cpu (Linux only); nix flake check does not build it, so the gated lint job stays at ~35s.

Fair-comparison caveat

The published base bundles Python 3.9–3.13 + a full apt userland; this PoC ships a single Python (3.14). Some of the delta is "fewer Pythons" — but the determinism, per-store-path layer dedup, and no-apt/no-compilers-in-the-image properties are structural and hold regardless. Adding multi-Python parity would grow it (est. +150–250 MB) and still beat the Dockerfile build.

Bumps

• Tracks latest nixos-unstable; PoC image uses Python 3.14.

GPU parity (not done here)

Documented in nix/images/README.md. Recommended next step: the hybrid approach — keep NVIDIA's tested CUDA base as fromImage and layer the deterministic Nix userland on top — then re-run compare-size.sh against a CUDA PyTorch tag.

Full write-up, tradeoffs, and local repro in nix/images/README.md.

[daveseddon-runpod]

Hybrid CUDA image (build-only) — result

Pinned nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04 as fromImage and layered the same Nix userland on top. Compared to runpod/base:…-cuda1281 (torch pip-installs identically on both, so excluded — this compares the userland layer). Built + measured in CI (manual-dispatch cuda-image-size job, 15m):

image	compressed
nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04 (shared base)	5593 MB
hybrid base-cuda (nvidia + Nix userland)	5846 MB — Nix layer ≈ 253 MB
runpod/base:1.0.7-cuda1281-ubuntu2404 (nvidia + apt)	6359 MB — apt layer ≈ 765 MB
runpod/pytorch:1.0.7-cu1281-torch280 (base + torch)	11211 MB

The Nix userland layer is ~67% smaller than the apt one (253 vs 765 MB). But the shared CUDA base + torch wheels dominate, so the whole-image win is only ~8%. Honest takeaway: for GPU images the big levers are a runtime (non-devel) base and a Nix-built torch, not the userland.

The cuda-image-size job is manual-dispatch only (if: github.event_name == 'workflow_dispatch') — it pulls a 5.6 GB base and streams ~11 GB, so it must not run on every push. Trigger with gh workflow run "Nix static analysis" --ref <branch>.

Cross-image layer sharing (fleet caching)

Since these co-deploy on the same hosts, cross-image Docker layer reuse is the real win. dockerTools layers are keyed by store path → identical paths give byte-identical layers/digests → a host caches shared deps once (base-cpu and base-cuda already share their userland layers). Caveat: streamLayeredImage assigns layers per-image (popularity heuristic, maxLayers cap), so a shared path can land in a dedicated layer in one image and the customisation bundle in another. To force maximal overlap: tiered fromImage bases (common → +CUDA → +torch), raise maxLayers, or use nix2container (purpose-built for family-wide sharing) — plus a "unique-vs-naïve cached bytes" footprint metric as a guardrail. Full write-up in nix/images/README.md.

[daveseddon-runpod]

Prototype: shared-userland image family + layer-sharing footprint

Built the tiered/shared-base idea and a metric for the fleet-caching win. A family of images all reuse the exact same Nix userland store paths and only add their own Python stack, so those layers dedupe on any host running more than one:

• family-base — plain userland
• family-data — + numpy/pandas/scikit-learn/matplotlib/scipy
• family-serve — + fastapi/uvicorn/pydantic/pillow/requests

New CI job family-footprint (footprint.sh) measures what a host actually caches once shared layers dedupe vs the naïve per-image sum:

	store closure (NAR)
family-base	875 MB
family-data	1551 MB
family-serve	962 MB
naïve sum (no sharing)	3388 MB
unique cached on a host (deduped)	1575 MB
saved by sharing	1813 MB — 53%

A box running all three flavors caches ~1.6 GB instead of ~3.4 GB. This is automatic in Nix: layers are keyed by store path, so the shared userland + tools are the same layer digests across every image — Dockerfile images only share their exact common prefix. Metric is uncompressed NAR bytes (what the Nix store + Docker dedup key on); compressed is proportional.

Ways to push it further (in nix/images/README.md): tiered fromImage bases (common → +CUDA → +torch), higher maxLayers, or nix2container; and footprint.sh doubles as a CI guardrail so a change that breaks sharing shows up as a regression.

CI status: flake-check (34s), image-size (~3m), family-footprint (41s) all green; cuda-image-size is manual-dispatch (skips on push).

[daveseddon-runpod]

Correction: real OCI layer sharing (not store-path closure)

Good catch on my earlier family-footprint number — it measured Nix closure (store paths), which overstates Docker layer sharing. streamLayeredImage only gives the top maxLayers (default 100) store paths their own layer and bundles the rest into a per-image customisation layer with a distinct digest. So identical store paths do not all become shared OCI layers.

footprint.sh now measures the real OCI layers (stream docker-archive → skopeo inspect → dedupe by blob digest — exactly what a host's Docker cache shares):

	real OCI layers
family-base	99 layers, 921 MB
family-data	99 layers, 1623 MB
family-serve	99 layers, 1010 MB
naïve sum	3555 MB
unique on a host (deduped by digest)	2312 MB
saved by sharing	1243 MB — 34% (was mis-reported as 53%)

And the maxLayers lever, measured:

maxLayers	real OCI shared	layers/image
100 (default)	34%	99
200	38%	199
400	53%	275

So the 53% store-path figure is only a ceiling, reached when every store path is its own layer (~275 layers/image — impractical). Default dockerTools gives 34% real OCI sharing; raising maxLayers trades layer count for sharing; nix2container reaches the ceiling without the layer-count blow-up.

Note: the fleet number (42%, published images) was already real OCI — it used skopeo LayersData digests all along. Only the family metric needed this fix. README updated; family-footprint CI job now green with the corrected metric.

[daveseddon-runpod]

nix2container as separate targets — head-to-head

Added nix2container as a flake input and mirrored the family as separate *-n2c targets (dockerTools work untouched, easy to compare). footprint.sh now measures both, real OCI layers, side by side. The result corrects a common assumption:

builder	maxLayers	real OCI shared	layers/img
dockerTools	100 (default)	34%	99
dockerTools	200	38%	199
dockerTools	400	53%	275
nix2container	100	16%	100
nix2container	300	48%	283
nix2container	600	53%	288

Two findings:

1. The sharing ceiling is set by the closure, not the tool — both converge at ~53% (the store-path ceiling), and only when maxLayers ≥ closure size (~275) so every path is its own layer. Neither exceeds it.
2. nix2container does not raise the ceiling; at its default (100) it shares less (16%) than dockerTools, because it bundles the non-dedicated closure into fewer/larger per-image layers. Its real edge is that many layers are cheap to build (lazy store-path refs, no per-layer tarball), so maxLayers=600 is practical where dockerTools-400 must materialise 275 tarballs.

The catch either way: 53% needs ~275+ layers/image — overlay2 and some registries discourage that many. So per-path auto-layering isn't the answer.

The actual lever is deliberate tiering: put the whole shared userland into one explicit layer every image reuses (nix2container buildLayer, or fromImage chaining) → near-ceiling sharing at a low layer count. That's the natural next step if we pursue this.

CI green: family-footprint now reports both builders (dockerTools 34% / nix2container 53%); everything else stays green; heavy cuda/fleet jobs remain manual-dispatch.

[daveseddon-runpod]

Explicit-tiering prototype — the practical answer

Added family-*-tiered targets: one explicit nix2container buildLayer holds the whole shared userland (~900 MB) that all three images reuse byte-for-byte, with only a small per-flavor delta on top. footprint.sh now compares all three approaches on real OCI layers (CI job family-footprint, green):

approach	real OCI shared	layers / image
dockerTools (auto, default)	34%	99
nix2container (auto, maxLayers 600)	53%	288
nix2container (tiered buildLayer)	51%	~15–21

Tiering reaches ~the sharing ceiling (51% vs 53%) at ~19 layers/image instead of ~288. The shared userland is one cached blob; family-data / family-serve add only their extra Python stack. That's near-maximal fleet sharing at a layer count overlay2/registries are happy with — the practical win the auto-layering couldn't deliver.

It extends naturally to the GPU tiers: common userland → +CUDA (per version) → +torch (per combo), each an explicit shared layer, so the whole fleet dedupes down to unique deltas.

Where the image experiment stands (all build-only, nothing published)

• base-cpu (dockerTools): 295 MB vs runpod/base 714 MB compressed, deterministic.
• base-cuda hybrid: nvidia base + Nix userland; Nix userland layer ~67% smaller than apt.
• fleet footprint: current published images already share ~42%.
• family layer-sharing: dockerTools 34% → n2c auto 53%@288 layers → n2c tiered 51%@~19 layers.

CI: flake-check, image-size, family-footprint green on every push; cuda-image-size + fleet-footprint are manual-dispatch.

[daveseddon-runpod]

Native-parity rebuild: base + pytorch entirely from Nix

New nix/prod-images/ tree (separate from the earlier PoC in nix/images/) rebuilds the containers with no apt/pip/uv installs and no vendor CUDA base — everything source-built from nixpkgs, hash-pinned. All four targets build; torch is genuinely compiled against Nix cudaPackages (verified torch.version.cuda == 12.9, cuDNN 9.22 — no download.pytorch.org wheel).

target	gzip (≈ download)	vendor equivalent	delta
prod-base-cpu	850 MB	runpod/base:*-ubuntu2404 714 MB	+19% (bundles full gcc/gfortran/ffmpeg toolchain; 1 python vs 5)
prod-base-cuda	3698 MB	runpod/base:*-cuda1281 6359 MB	−42%
prod-pytorch-cpu	1250 MB	—	—
prod-pytorch (CUDA)	7791 MB	runpod/pytorch:*-cu1281-torch280 11211 MB	−30%

What's Nix-native now: the jupyter-archive dep missing from nixpkgs is packaged from the PyPI sdist (source, hash-pinned); CUDA comes from nixpkgs redistributables (hash-pinned fetches, not the ~5.6 GB nvidia/cuda:*-devel base); torch/torchvision/torchaudio are source-compiled with cudaSupport. A Nix-adapted /start.sh reproduces the runtime contract (nginx/sshd/jupyter) without Debian service.

Honest caveats:

• Version drift — native nixpkgs gives one torch (2.12.0) + CUDA 12.9, newer than and not matching the repo's 2.6–2.9.1 × cu126–130 matrix. Inherent to source-from-nixpkgs.
• CUDA torch is a heavy from-source compile (unfree ⇒ absent from cache.nixos.org) — built locally here (~24-core/125 GB box); GH-hosted CI can't, so no CI job for it (mirrors the manual-dispatch cuda-image-size).
• Python 3.14 vs 3.9–3.13; jupyterlab 4.5.8 vs pinned 4.5.9.
• Runtime not container-tested in the sandbox (image builds + contents/imports verified; full nginx/sshd/jupyter startup needs a real pod).
• Out of scope: autoresearch, nvidia-pytorch (NGC), rocm.

Verified: userland imports (jupyterlab/hf_transfer/jupyter_archive), torch CUDA build, flake-check green, all committed on nix. Details in nix/prod-images/README.md.