refactor(core): introduce staged pipeline

[sigilmakes]

Summary

Introduces the staged Goldilocks Core pipeline with composable stage backends and a first-class Kmesh stage, making it the shared execution path for Python callers, the staged CLI, and future HTTP wrappers.

The Core graph is explicit and inspectable:

Load → Analyze → Advise → Kmesh → Select → Generate → Bundle

Every stage backend is swappable through the Pipeline object — no registries, no base classes, no string resolution in Core. Backends are plain functions composed into a dataclass. CoreJobRequest stays data-only and serializable.

Why

The refactor prevents Python, CLI, and future HTTP entry points from growing separate recommendation logic. It also makes the k-point resolution path a first-class backend seam so ML models can participate in the pipeline without hacks.

Composable Pipeline

Pipeline is a dataclass with one callable per stage. default_pipeline() returns the built-in composition. Callers customize with dataclasses.replace():

from dataclasses import replace
from goldilocks_core import default_pipeline, run_core_job, CoreJobRequest
from goldilocks_core.advisors import ml_kmesh_advisor

pipeline = replace(default_pipeline(), kmesh=ml_kmesh_advisor(spec))
result = run_core_job(CoreJobRequest(structure="Si.cif"), pipeline=pipeline)

CoreJobRequest carries what to compute. Pipeline carries how to compute it. This separation keeps the request JSON-safe while letting Python callers swap backends directly.

Kmesh stage

K-point resolution was previously split across Advise (intent) and Select (concrete grid). ML k-point prediction returns a KPointSelection, not KPointAdvice, so there was no clean seam to plug it in.

The new Kmesh stage sits between Advise and Select:

Analyze → Advise → Kmesh → Select

• Default backend (resolve_kpoints_from_advice): hint precedence (k_grid > k_spacing > advice spacing), then conversion to mesh.
• ML backend (ml_kmesh_advisor(spec)): hints still win, then model prediction, provenance source="model".
• Custom backends: any function satisfying Callable[[Structure, CalculationHints, KPointAdvice], KPointSelection].

Select now receives the Kmesh result instead of deriving k-points internally.

CLI --model

goldilocks-core recommend Si.cif --model model.joblib --json

The CLI resolves --model into a ModelSpec → ml_kmesh_advisor(spec) → Pipeline. The model path is never on CoreJobRequest.

Staged changes

Stage 1 — Contracts and provenance

• Per-field Attributes: docstrings on all 22 dataclasses + 13 type aliases in contracts.py.
• PseudoMetadata and PseudoPolicy use slots=True.
• ProvenanceSource literal type with per-value documentation.
• Serialization round-trip tests.

Stage 2 — Pipeline implementation

• StructureAnalysisRecord with composition, element classification, symmetry, electronic character, disorder warnings.
• ParameterAdvice with provenance-backed k-points, smearing, magnetism, SOC, pseudopotential intent, convergence.
• SelectionRecord with deterministic pseudo ranking and cutoff extraction.
• QE SCF input generation from completed advice/selection records.
• Portable bundle directory output with manifest.json.
• Staged CLI: goldilocks-core recommend|generate|bundle.

Stage 3 — Composable Pipeline (this PR, layered on stages 1–2)

• Pipeline dataclass with AnalyzeStage, AdviseStage, KMeshAdvisor, SelectStage, GenerateStage, BundleStage callable fields.
• default_pipeline() factory.
• pipeline parameter on run_core_job(), recommend(), generate(), write_bundle().
• resolve_kpoints_from_advice() extracted from Select into Kmesh stage.
• ml_kmesh_advisor(spec) factory for model-backed Kmesh backends.
• --model / --model-name / --model-version CLI flags.
• Kmesh StageRecord in job results.
• K-point conflict warnings now propagate through Kmesh provenance.
• Extensive documentation: docs/pipeline.md, docs/backends.md, docs/contracts.md, docs/manifest.md, docs/stages/kmesh.md.

Acceptance criteria

• Pipeline dataclass with one callable per stage
• run_core_job(request) with no pipeline argument produces identical output to previous implementation
• run_core_job(request, pipeline=custom_pipeline) uses provided backends
• Kmesh is its own pipeline step between Advise and Select
• ml_kmesh_advisor(spec) produces source="model" provenance
• Hint precedence enforced: k_grid > k_spacing > strategy fallback
• Select no longer resolves k-points internally
• CoreJobRequest remains fully data-only and serializable
• No new external dependencies
• All existing tests pass + new tests for Pipeline composition, Kmesh, and ML integration

Open design note

Kmesh is currently a hardcoded layer in the pipeline graph. Longer term, the advise→select stages should also be individually pluggable — not just through Pipeline field replacement, but potentially through a more dynamic backend system that supports multiple prediction backends for different areas (k-points, smearing, pseudopotentials, etc.). The current Pipeline design is a solid foundation for this: each stage is already a standalone callable with a typed signature. The next evolution would make backend selection more granular and composable without adding registries or string resolution to Core.

Closes #8, #12, #13, #14, #15, #16, #17, #19, #20.

---

Written by an agent on behalf of Willow.

[sigilmakes]

Review: refactor/staged-core-pipeline

Files changed: 37 in the PR diff against test/baseline-coverage.
Verdict: ready as a draft stacked PR.

Target spec alignment

The branch matches the #8 staged Core target closely:

• Implements the fixed Load → Analyze → Advise → Select → Generate → Bundle graph.
• Adds CoreJobRequest, StageRecord, CoreJobResult, and run_core_job().
• Keeps CLI and future HTTP mapping around the shared job request/result surface.
• Adds QE SCF generation from completed advice/selection records without generator-side scientific defaults.
• Adds deterministic bundle writing and manifest.json content.
• Expands analysis/advice and pseudopotential ranking without adding new external dependencies.
• Removes the legacy goldilocks_core.shared path instead of adding compatibility shims.
• Adds portable tests for contracts, stages, jobs, generation, bundle output, CLI flow, pseudo ranking, and regression paths.

Findings

Severity	File	Issue
Low	src/goldilocks_core/jobs.py:38, src/goldilocks_core/cli/core.py:165	Advice provenance warnings are preserved in nested advice records but are not aggregated into CoreJobResult.warnings / CLI human summaries. This is not blocking because JSON callers can still inspect provenance, but if top-level warnings are intended to be comprehensive, aggregate advice-stage warnings into the job result and StageRecord(name="advise").

No critical, high, or medium issues found.

Verification

uv run ruff check src tests              -> passed
uv run ruff format --check src tests     -> passed
uv run pytest -q                         -> 67 passed
uv run pre-commit run --all-files        -> passed

Stack / CI notes

• Draft PR: refactor(core): introduce staged pipeline #18, based on test/baseline-coverage.
• No GitHub checks are reported for this branch; local verification is the current gate.
• Do not merge until the stacked baseline PR is resolved or this PR is rebased/base-updated.

---

Written by an agent on behalf of Willow.

[sigilmakes]

Pushed 4ec6789 — composable Pipeline with swappable stage backends and Kmesh stage.

What changed:

• Pipeline dataclass with typed callable fields for each stage
• default_pipeline() factory
• pipeline= injection on run_core_job(), recommend(), generate(), write_bundle()
• Kmesh extracted as a first-class stage between Advise and Select
• ml_kmesh_advisor(spec) factory for model-backed k-point backends
• --model / --model-name / --model-version CLI flags
• K-point conflict warnings now propagate through Kmesh provenance
• CoreJobRequest stays data-only; backend choice lives in Pipeline
• Extensive docs: pipeline.md, backends.md, contracts.md, stages/kmesh.md, plus updates to README, architecture, tutorial, CLI, extension, migration, provenance, serialization, decisions, changelog

Tests: 78 passed, ruff clean, pre-commit clean.

Design note for future work: Kmesh is currently a hardcoded pipeline layer. The next evolution should make advise→select stages individually pluggable so that different prediction backends (k-points, smearing, pseudopotentials, etc.) can be swapped independently. The Pipeline dataclass is the foundation — each stage is already a standalone callable — but the granularity may need to increase beyond the current six fields.

---

Written by an agent on behalf of Willow.