DEV-1450 principled redesign of syntax (WIP)#139
Conversation
Add `slayer/core/keys.py` with the typed-key family that the new resolution pipeline uses for structural identity (P2): - `ColumnKey(path, leaf)` — row-level base column ref. Local and cross-model share this shape; only `path` (empty vs non-empty) distinguishes (P3). - `ColumnSqlKey(model, column_name)` — derived column ref. - `StarKey` — sentinel source for `*:count`. - `SqlExprKey(canonical_sql)` — identity for Mode-A fragments (used as `AggregateKey.column_filter_key`). - `AggregateKey(source, agg, args, kwargs, column_filter_key)` — unified local + cross-model aggregate identity. Kwargs sorted by validator; scalars normalised; `column_filter_key` distinguishes same-column-different-filter aggregates. - `TransformKey(op, input, args, kwargs, partition_keys, time_key)` — window / temporal operator over a value. `partition_keys` is a frozenset (order-independent). - `ArithmeticKey(op, operands)` — operand order matters. - `ScalarCallKey(name, args)` — closed-allowlist scalar call (C12). - `Phase` IntEnum (ROW=0, AGGREGATE=1, POST=2) — `.phase` property on every key; ArithmeticKey/ScalarCallKey take max over operands. - `SCALAR_FUNCTIONS` frozenset — the closed allowlist. - `normalize_scalar(value)` — bool/None passthrough; int→Decimal; float→Decimal(str(...)) to avoid binary imprecision; Decimal/str passthrough; anything else raises TypeError. All keys are frozen Pydantic models; hashable; usable as dict keys for slot interning. 60 tests cover identity, hashing, kwarg canonicalisation, phase composition, and the C12 allowlist contents. Dormant: no engine code routes through these yet (stages 7a/7b wire them up). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add the typed scope kinds (P5) and the resolved-source bundle (P11): - `slayer/core/scope.py` — `StageColumn` (typed projection element, P6), `StageSchema` (flat-namespace projection of one stage that downstream stages bind against), `ModelScope` (join-graph-bearing scope used during binding). - `slayer/engine/source_bundle.py` — `ResolvedSourceBundle`: eagerly resolved query inputs (source_model + referenced_models + inline_extensions + named_queries + query_variables + datasource_hint). Built once by the orchestrator; the binder reads purely (P11 — no ContextVar machinery). I2 (anchor-optional, extension injection): `ModelScope.source_model` and `ResolvedSourceBundle.source_model` are `Optional[SlayerModel]` from day one. DEV-1450's binder asserts `source_model is not None` at use sites so behavior is unchanged. The type-level optionality is the extension point for a future anchor-less mode (where the global join graph is auto-discovered from `model.column` refs). 25 tests cover: hidden vs visible slots, flat namespace contract (__-bearing names are flat in StageSchema), lookup helpers, construction with/without source_model. Dormant: no engine code routes through these yet. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the consolidated DEV-1369 join walker out of `SlayerQueryEngine` into `slayer/engine/path_resolution.py` so the new binder modules can import it directly without dragging the engine in. - New `walk_join_chain(*, source_model, hop_names, resolve_model, named_queries, strict_missing_join)` — free function that takes the engine's `_resolve_model` as a callback. - `NoJoinError` sentinel moves with it (the lenient-missing-join signal used by dimension-resolution callers). - `SlayerQueryEngine._walk_join_chain` becomes a thin shim that passes `self._resolve_model` to the new function. All existing call sites keep their signature. - `_NoJoinError` is re-imported into the engine module under the same name so internal usages don't change. 13 tests pin the contract: cycle detection, strict vs lenient missing-join, multi-hop returns (terminal_model, first_join), prefer_data_source threading, named_queries default. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lift agg-name collection and parameter resolution out of enrichment.py and generator.py into `slayer/engine/agg_registry.py` so the new binder modules don't have to reach into those tangles. Helpers are pure: given a model + a resolve_join_target callback, they produce structured results without touching storage or spawning side maps. - `collect_reachable_agg_names(source_model, resolve_join_target, named_queries)` — BFS the join graph for custom aggregation names (lifted from `enrichment.py:_collect_reachable_agg_names`). Cycle-safe via visited set. - `is_known_aggregation_name(name, custom_names)` — built-in or in the custom set. - `resolve_aggregation(name, available_aggs)` — find the `Aggregation` definition for a name, returning None when only a built-in default should apply. - `required_params_for(agg_name)` — required built-in params (e.g. `weighted_avg` requires `weight`). - `merge_agg_params(agg_def, query_kwargs)` — combine defaults with query-time overrides. 25 tests cover the BFS, custom-name lookup, builtin-override resolution, and param merging. Dormant: existing enrichment / generator code still inlines its own logic; stages 7a/7b switch over. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add the typed error / warning classes for the new pipeline with a
stable str() format so snapshot tests can pin the contract:
<ErrorName>: <one-line summary>
at <location>
scope: <short scope summary>
suggestion: <did-you-mean>
Errors (subclass `SlayerError`):
- `UnknownReferenceError(name, scope_kind, scope_summary, suggestion)`
- `AmbiguousReferenceError(name, candidates)` — sorts candidates
- `IllegalScopeReferenceError(name, scope_kind, reason)` — covers
C8 (`__` in ModelScope) and DEV-1449 (dotted refs against
StageSchema)
- `IllegalWindowInFilterError(filter_expr, source, suggestion)` —
raw `OVER(...)` in DSL filter OR filter naming windowed
`Column.sql` (predicate promotion was removed in DEV-1336)
- `AggregationNotAllowedError(column, agg, reason)` — type-bucket
/ PK / allowed_aggregations violations
- `UnknownFunctionError(name, location, suggestion)` — Mode-B call
outside SCALAR_FUNCTIONS / transforms / aggregations (C12)
- `MeasureRecursionLimitError(chain, limit)` /
`MeasureCycleError(chain)` — named-measure expansion guards
- `DuplicateMeasureNameError(name, occurrences)` /
`MeasureNameCollidesWithColumnError(name, model)` /
`CanonicalAliasShadowsColumnError(formula, canonical, model)` —
DEV-1443 alias-collision validations
- `UnreachableFilterDroppedWarning(filter_text, reason)` — warning,
emitted by the cross-model planner when it drops an unreachable
host filter from a sub-query CTE
Slack normalization payload + carrier (`slayer/core/warnings.py`):
- `NormalizationWarning` — Pydantic payload with `rule_id`,
`original`, `normalized`, `location`, `rule_doc_url`.
- `SlayerNormalizationWarning` — UserWarning carrier so callers
can route via `warnings.catch_warnings()` AND via the structured
`SlayerResponse.warnings` list (stage 6 wiring).
36 tests snapshot str(error) for every class and verify the
warning carrier roundtrips through `warnings.warn(...)`.
Dormant: no code raises these yet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ED_MEASURE) Add `slayer/engine/normalization.py` and wire it into the engine. The layer rewrites tolerant-but-unambiguous agent input to canonical form before the typed pipeline sees it (P0), and emits structured `NormalizationWarning` payloads alongside the existing in-tree UserWarnings (which still fire from `_rewrite_funcstyle_aggregations` until stage 7b removes them). Active rules in stage 6: - `FUNC_STYLE_AGG` (Mode B): `sum(revenue)` → `revenue:sum`; `count(*)` → `*:count`; `percentile(amount, p=0.5)` → `amount:percentile(p=0.5)`. Applied to `ModelMeasure.formula`, `SlayerQuery.measures[].formula`, and `SlayerQuery.filters` entries. Custom model-level aggregation names are recognised in `normalize_model` via the model's `aggregations` list. - `MISPLACED_MEASURE` (query shape): bare entries in `SlayerQuery.measures` that resolve as a column on the model (and not as a `ModelMeasure` name) move to `SlayerQuery.dimensions`. Mirrors the existing `_auto_move_fields_to_dimensions` heuristic but emits the structured warning. - `DOT_PATH_IN_SQL` (Mode A): wired but stubbed — returns input unchanged. Full sqlglot-AST, scope-aware rewrite lands in a follow-up. The slot is wired so downstream activation needs no plumbing changes. Engine wiring (`slayer/engine/query_engine.py`): - `SlayerResponse.warnings: List[NormalizationWarning]` additive field (default empty). - `_execute_pipeline` runs `normalize_query` immediately after the source model is resolved (so MISPLACED_MEASURE has model context); rewrites flow into the existing `_auto_move_fields_to_dimensions` + enrichment, which see already-canonical input and silently no-op. - All three `SlayerResponse(...)` construction sites (dry_run, explain, normal execute) propagate `warnings=slack_warnings`. - `save_model` runs `normalize_model` before persistence so stored formulas land in canonical form. 17 tests cover FUNC_STYLE_AGG rewrites + warning emission, MISPLACED_MEASURE detection + move, custom-agg recognition, canonical-input no-op behavior, engine wiring through `engine.execute(..., dry_run=True)` and `engine.save_model(...)`. Total: 3056 unit tests passing, no regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…threading
Codex review of stages 1-6 surfaced two important bugs. Fix both
plus pin the contract with new tests.
1. **bool vs Decimal interning collision in typed keys** (stage 1).
Python collapses ``True == 1 == Decimal('1')`` and the same for
``False`` / ``0`` at both equality and hashing. So
``AggregateKey(args=(True,))`` previously interned with
``AggregateKey(args=(Decimal('1'),))`` — silently sharing a slot
for two semantically different aggregations
(``percentile(p=True)`` vs ``percentile(p=1)``).
Add `_typed_leaf` / `_typed_args` / `_typed_kwargs` helpers in
`slayer/core/keys.py` that wrap scalar leaves as
``(type_tag, value)`` at hash/eq time without changing the
stored representation. Override `__hash__` and `__eq__` on
`AggregateKey`, `TransformKey`, and `ScalarCallKey` to use them.
`ColumnKey` / `ColumnSqlKey` / `StarKey` / `SqlExprKey` are
unaffected (they don't take scalar args).
4 new tests in `tests/test_keys.py::TestBoolDecimalDisambiguation`
pin the contract across all three key classes.
2. **custom_agg_names not threaded to normalize_query** (stage 6).
The engine called `normalize_query(query, model=model)` without
custom aggregation names, so a slack `custom_sum(revenue)` in a
query measure or filter was left to the legacy
`_rewrite_funcstyle_aggregations` rewrite (which still
functions correctly) but didn't surface a structured warning in
`SlayerResponse.warnings`.
Collect custom names from `model.aggregations` in
`_execute_pipeline` and pass them through. Joined-model custom
aggs are still out of scope (need the full reachable-agg-names
walk) — that arrives with stage 7a's binder; the legacy rewrite
continues to catch them until then.
1 new test in `tests/test_slack_normalization.py` verifies the
structured warning surfaces for a model-defined custom aggregation.
Codex finding 3 (FUNC_STYLE_AGG on Mode-A filters) is a false
positive: `SlayerQuery.filters` is Mode B per CLAUDE.md / DEV-1378.
Mode-A filters live on `Column.filter` and `SlayerModel.filters`,
which the normalization layer does not touch.
Codex finding 4 (backslash-escaped quote handling in literal-span
detection) is a minor edge case for SQL dialects that allow ``\\'``
inside literals; noted as a follow-up.
Net: 3061 unit tests passing (was 3056), no regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `slayer/engine/planned.py` with the typed shapes consumed by the SQL generator after stage 7b cutover. Dormant — no engine code routes through them yet. - `ValueSlot(id, key, declared_name, public_name, public_aliases, hidden, phase, label, type, expression)` — one materialised slot. `key` is structural identity; rendering metadata lives here. P4 / C13 multi-alias supported. Hidden invariant enforced by a model validator: hidden slots must not carry public_name / public_aliases. - `JoinRequirement(source_model, target_model, join_pairs, join_type)` — one hop in a cross-model chain. `join_type` defaults to LEFT to match `ModelJoin`. Non-empty + well-formed `join_pairs` validated. - `CrossModelAggregatePlan(aggregate_slot_id, target_model, datasource, join_chain, join_back_pairs, cte_stage_schema, shared_grain_slots, applied_filter_ids, dropped_filter_warnings, hidden, public_alias)` — plan for one cross-model aggregate slot (P3). Strategy-agnostic: the I1 CrossModelPlanner Protocol (stage 7a.2) populates this struct; the struct doesn't lock in isolated-CTE. - `TransformLayer(op, slot_ids)` — one transform layer; generator picks the render strategy per op (window function vs self-join CTE). - `FilterPhase(id, phase, text, expression)` — bound filter routed by phase (P8). `expression` carries the renderable payload so the generator never re-parses text. - `OrderEntry(slot_id, direction)` — ORDER BY entry; strict lowercase `asc`/`desc` since OrderEntry is planner-produced. - `BoundExpr(value_key, sql_text)` — minimum scaffold for the bound-expression payload that stage 7a.5's binder will fill in. Carried by ValueSlot.expression and FilterPhase.expression. - `PlannedQuery(source_relation, join_plan, row_slots, aggregate_slots, cross_model_aggregate_plans, combined_expression_slots, transform_layers, filters_by_phase, projection, order, limit, offset, stage_schema)` — the fully typed plan that the SQL generator consumes. Codex review (round 1) on the new module surfaced three important items: FilterPhase lacked a bound-expression payload (fixed via BoundExpr scaffold), JoinRequirement was missing join_type (fixed), ValueSlot hidden invariant was not enforced (fixed via model validator). All addressed. 32 unit tests pin the contract, including the hidden invariant, join_type defaults, OrderEntry case strictness, and BoundExpr propagation through slots. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the no-op stub of `_apply_dot_path_in_sql` with an AST-based, scope-aware Mode-A rewrite over `Column.sql`, `Column.filter`, and `SlayerModel.filters`. Root-scope `<join>.<intermediate>.<leaf>` refs collapse to `<join>__<intermediate>.<leaf>`; first-segment shadowing by CTE name, AS alias, Subquery/CTE source, or FROM-table schema/catalog qualifier emits an ambiguity warning without rewriting. Refs inside subqueries / CTE bodies / set-op branches are left alone via lexical ancestor walking. Multi-statement slack input is a no-op. Wired through `normalize_model`. The legacy regex `slayer.core.models._fix_multidot_sql` still runs at construction time and pre-empts most real-world inputs; Stage 7b deletes the regex and hands full ownership to this rule. 32 new tests in `tests/test_dot_path_in_sql.py`. Full unit suite green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… (I1) Adds slayer/engine/cross_model_planner.py with: - CrossModelPlanner Protocol (substitutable strategy). - IsolatedCteCrossModelPlanner — default impl encoding the inherited_filter_policy decision table. - HostFilterRouting input + classify_host_filter() helper. - FilterRoute enum: DROP_HOST_LOCAL / PROPAGATE_WHERE / PROPAGATE_HAVING / DROP_UNREACHABLE / STAY_AT_HOST_POST. Extends CrossModelAggregatePlan in planned.py with route-explicit fields the SQL generator (7b) needs without re-classifying: - where_filter_ids / having_filter_ids / target_model_filters. Multi-hop semantics: join_back_pairs and CTE schema reference the FIRST hop's target grain (e.g. customers.id for orders→customers→regions), while the aggregate output column's type is sourced from the terminal model. ColumnSqlKey routing uses host_model_name to distinguish host-local derived columns (DROP_HOST_LOCAL), derived columns on the target path (PROPAGATE_WHERE), and derived columns on other branches (DROP_UNREACHABLE). 32 new tests covering Protocol substitution, single + multi-hop join chains, every row of the decision table, ColumnSqlKey routing, edge cases (unknown slot id, empty refs). Full unit suite green. Dormant — no engine wiring. Stage 7a.6's ProjectionPlanner is the first consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds slayer/engine/syntax.py with parse_expr() — pure syntax for the Mode-B DSL (no scope resolution, no named-measure expansion, no slack rewriting; those are upstream). ParsedExpr family: Ref / DottedRef / StarSource / Literal / AggCall / TransformCall / ScalarCall / Arith / UnaryOp / Cmp / BoolOp. All frozen Pydantic models with value-based equality. Pipeline: colon preprocess (skipping string-literal spans) → Python ast.parse → AST walk to typed nodes. Closed allowlist for scalar functions (SCALAR_FUNCTIONS in keys.py); transforms from ALL_TRANSFORMS; aggregations via placeholder lookup. AST-level dunder rejection across Name / Attribute / keyword.arg positions — robust to string literals. Rejections (raises): - Unknown function call → UnknownFunctionError. - Raw OVER(...) → IllegalWindowInFilterError. - `__` in user identifier → ValueError. - Chained comparisons (`1 < x < 10`) → ValueError. - Function-style aggregations (`sum(revenue)`) → UnknownFunctionError (slack normalization owns the rewrite; if it reaches the parser, that's a bypass). - Scalar-function kwargs (`lower(name=...)`) → ValueError. 62 tests in tests/test_syntax.py. Full unit suite green: 3219 passed. Dormant — no engine wiring. Stage 7a.5 binder is the first consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds slayer/sql/sql_expr.py with: - parse_sql_expr(text, *, dialect=None) → SqlExprKey - canonicalize_sql(text, *, dialect=None) → str - has_window_function(text) → bool (re-export) - assert_no_window_in_filter(text, *, source) → None / raises parse_sql_expr wraps as `SELECT (<text>) AS _` before sqlglot parsing, then strips the wrapper paren. Mirrors the generator's predicate-parse trick — necessary because sqlglot's SQLite/MySQL parser otherwise falls back to a Command node for top-level `replace(...)` and other keyword conflicts. Dialect-specific rewrites preserved through the canonical key: - SQLite json_extract function form (vs `->` operator) - log10 / log2 preservation for dialects with native single-arg aliases (allowlist duplicated from slayer/sql/generator.py — comments in both call out the sync point; consolidation deferred). 23 tests in tests/test_sql_expr.py. Full unit suite green. Dormant — no engine wiring. Stage 7a.5 binder is the first consumer (AggregateKey.column_filter_key). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds slayer/engine/binding.py with: - bind_expr(parsed, *, scope, bundle) → BoundExpr - bind_filter(parsed, *, scope, bundle) → BoundFilter - walk_value_keys(key) — traverse helper for referenced-key collection Two scope kinds (P5): - ModelScope: dotted refs walk the join graph via bundle. C14 self- prefix stripped before walk. `__` rejected unless it exact-matches a column on the model. - StageSchema: flat namespace; dotted refs raise IllegalScopeReferenceError. FilterBinder layers phase classification (Phase.ROW / AGGREGATE / POST = max of referenced slot phases) plus IllegalWindowInFilterError when a filter references a ColumnSqlKey whose Column.sql contains a window function (DEV-1369: no auto-promotion). keys.py: add LiteralKey to ValueKey union for arithmetic over scalars (`amount + 1`); add `path: Tuple[str, ...]` to ColumnSqlKey so joined derived columns carry the same path-bearing identity ColumnKey has. Defence-in-depth: _bind_scalar re-checks SCALAR_FUNCTIONS in case the parser was bypassed via direct ParsedExpr construction. 31 tests in tests/test_binding.py. Full unit suite green: 3273 passed. Dormant — no engine wiring. Stage 7a.6 planner is the first consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds slayer/engine/planning.py with three composable concerns: 1. ValueRegistry — interns ValueKeys by structural identity. Two structurally-equal keys share one ValueSlot (P2). Same key declared with multiple `name`s accumulates multiple public_aliases on one slot (P4 / C13). Alias-collision validations preserved from DEV-1443: DuplicateMeasureNameError, MeasureNameCollidesWithColumnError, CanonicalAliasShadowsColumnError. 2. desugar_change / desugar_change_pct — lower sugar transforms to `x - time_shift(x)` / `(x - time_shift(x)) / time_shift(x)`. The inner aggregate keeps the same ValueKey instance across both operands so the registry interns it once (DEV-1446). `partition_by` from the sugar form threads through to the underlying time_shift (C6) — the binder lifts it onto TransformKey.partition_keys, the lowerer passes it through. 3. ProjectionPlanner — allocates slots for declared measures + creates hidden slots for refs that appear ONLY in order/filter. Slot dependency selection (`_iter_slot_deps`) only materialises ColumnKey, ColumnSqlKey, AggregateKey, TransformKey — composite nodes (ArithmeticKey, ScalarCallKey) and literals stay inlined. binding.py: _bind_transform now lifts the `partition_by` kwarg onto TransformKey.partition_keys; non-column partition_by values are rejected with a clear error. 26 tests in tests/test_value_registry.py / test_transform_lowerer.py / test_projection_planner.py. Full unit suite green: 3299 passed. Dormant — no engine wiring. Stage 7a.7 stage_planner is the first consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds slayer/engine/stage_planner.py with:
- plan_query(*, query, bundle, scope=None, ...) → PlannedQuery
- Parses + binds measures / dimensions / filters / order against
the supplied scope (ModelScope by default, StageSchema for
downstream stages).
- Runs ProjectionPlanner + assembles a PlannedQuery with stage_schema.
- plan_stages(*, queries, bundle, ...) → List[PlannedQuery]
- Topologically sorts via Kahn's algorithm (rejects duplicate stage
names and cycles).
- Per-stage StageSchema becomes the binding scope for downstream
stages (DEV-1449: dotted refs in downstream stages raise
IllegalScopeReferenceError).
- User-supplied `name` on a measure becomes the StageSchema column
alias (DEV-1448). Multi-alias declarations (same key, two names)
emit one StageSchema column per alias.
ValueRegistry: exempts self-named dimensions (ColumnKey(leaf=X)
declared as X) from MeasureNameCollidesWithColumnError — the
declaration is the column itself, not a rename.
10 tests in tests/test_stage_planner.py covering:
- single-stage smoke (measure + dimension + filter),
- DEV-1448 (named measure → schema column alias),
- DEV-1449 (downstream flat refs; dotted refs rejected),
- topo sort (duplicate name and cycle rejection),
- multi-alias StageSchema columns.
Full unit suite green: 3309 passed.
Dormant — no engine wiring. cross_model_planner parameter is built
but not yet invoked from PlannedQuery construction; Stage 7b wires
the full cross-model-aggregate plan emission. Same for BoundExpr
payload propagation onto ValueSlot / FilterPhase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move {var} placeholder substitution out of the legacy enrichment path
(slayer/engine/enrichment.py:1162) into a small, pipeline-friendly
slayer/engine/variables.py module. Dormant in this commit — wired in by
stage 7b.15 (engine cutover).
Public surface:
- merge_query_variables(*, runtime, stage, outer, model_defaults):
4-layer merge replacing the legacy 3-layer _merge_query_variables.
Precedence runtime > stage > outer > model_defaults; None / empty
layers act as identities; inputs unmutated.
- apply_variables_to_query(*, query, variables=None, dry_run_placeholders=False):
returns a fresh SlayerQuery copy with {var} substituted in `filters`.
Always copies (no shared empty-list aliasing). variables=None is
normalized to empty dict. dry_run_placeholders=True fills missing
valid placeholders with "0" but does NOT mask invalid names.
Scope deliberately matches legacy: only SlayerQuery.filters is
substituted. Formula text, Column.sql, Column.filter, and
SlayerModel.filters are not variable-substituted today and this module
preserves that contract.
39 new tests in tests/test_variables_planner.py cover precedence,
escape (`{{` / `}}`), invalid names, unmatched braces, dry-run +
invalid-name interaction, idempotence, copy semantics, and re-export
identity. Codex review of tests caught HIGH#1 (copy vs identity) and
HIGH#2 (explicit idempotence test); both folded in. Codex review of
impl caught two LOW findings (empty-list aliasing and Optional
variables); both folded in.
Full unit suite green (3348 passed, 2 skipped); ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New slayer/engine/measure_expansion.py: AST-to-AST rewrite of a
ParsedExpr tree. Every Ref(name=X) whose X resolves to a ModelMeasure
(on the model or in extra_measures) is replaced with the recursively-
expanded ParsedExpr of that measure's formula.
Per Codex F4 fold-in on the DEV-1450 plan: expansion lives PRE-BIND, not
planner-layer. The binder (slayer/engine/binding.py:341-354) currently
raises UnknownReferenceError for bare measure names; this module runs
ahead of the binder and rewrites those refs into binder-resolvable AST
shapes.
Public API:
expand_model_measures(
*,
expr: ParsedExpr,
model: SlayerModel,
extra_measures: Sequence[ModelMeasure] = (),
depth_limit: Optional[int] = None,
) -> ParsedExpr
Eligible positions: root; Arith / UnaryOp / Cmp / BoolOp operands;
ScalarCall.args; TransformCall.input / args / kwarg values.
Not eligible: DottedRef (cross-model paths), AggCall in any position
(source / args / kwargs are column-level by contract), function-name
slots on TransformCall.op / ScalarCall.name, Literal, StarSource,
SlayerQuery.order entries.
Depth limit: SLAYER_MEASURE_EXPANSION_DEPTH env var (default 32);
explicit kwarg wins; <1 raises ValueError. Exceeded chain raises
MeasureRecursionLimitError. Per-chain cycle detection raises
MeasureCycleError with the full traversal chain.
Per-call memoization of parsed measure formulas (Codex impl review
MEDIUM#1). _PARSED_EXPR_TYPES tuple derived from get_args(ParsedExpr) so
future syntax additions are auto-walked (Codex LOW#5).
41 new tests in tests/test_model_measure_expansion.py cover the
eligibility matrix, recursion / cycle semantics, depth-limit boundaries
(default 32 succeeds at 32, fails at 33), env-var override + explicit
override precedence, extra_measures shadowing model measures, purity
under re-expansion, and pre-bind "does not validate columns" contract.
Full unit suite green (3389 passed, 2 skipped); ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add slayer/core/keys.py::TimeTruncKey — row-phase ValueKey identifying a time-truncated column by (column: ColumnKey, granularity: str). The underlying column is recoverable via key.column so date-range filters (stage 7b.3c) can bind against the raw column independently of the truncation. Identity is structural: same (column, granularity) interns to one slot; different granularities on the same column are distinct slots. The ValueRegistry can keep month / day / raw uses of the same column as separate materialised values without special-casing — Codex's recommendation (Option A) over a granularity-on-slot or TransformKey(op='date_trunc') encoding. Granularity stored as the str value of a TimeGranularity StrEnum so the key stays pure data without an enum import at the keys layer. TimeTruncKey added to the ValueKey union so binders and planners that dispatch on isinstance(key, ValueKey) see it. Dormant — no engine wiring yet. Stage 7b.3b adds bind_time_dimension + planner integration; 7b.3c adds main-TD resolution and date_range -> filter conversion. 12 new tests in tests/test_time_trunc_key.py cover construction, identity (same / different grain / different column / cross-model path), phase=ROW, frozen-immutability, and ValueKey union membership. Full unit suite green (3401 passed, 2 skipped); ruff clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires SlayerQuery.time_dimensions through to TimeTruncKey slots in the
new pipeline.
New: bind_time_dimension(td, scope, bundle) in slayer/engine/binding.py.
Resolves the underlying column via the existing identifier / dotted-ref
binders, validates it's a base ColumnKey with a temporal (DATE/TIMESTAMP)
type, and returns a BoundExpr whose value_key is
TimeTruncKey(column, granularity). Derived Column.sql temporal columns
(which would resolve to ColumnSqlKey) raise NotImplementedError with a
clear message — TimeTruncKey.column is typed as ColumnKey and widening
it is deferred to a follow-up. StageSchema scopes raise
IllegalScopeReferenceError (downstream stages see the upstream's
already-truncated column as a flat name).
Planner: TimeTruncKey added to _SLOTTABLE_KIND, _iter_slot_deps, and
_canonical_name. The TimeTruncKey itself is the materialized slot
(generator emits DATE_TRUNC at SELECT time); the inner ColumnKey is NOT
yielded as a separate dep — adding a TD must not auto-add the raw
column as a separate output (matches legacy). Canonical name drops the
granularity suffix to match legacy alias contract
(EnrichedTimeDimension.alias = f"{model}.{td.dimension.full_name}").
ValueRegistry: extended the self-named-dimension exemption to cover a
local TimeTruncKey over a same-named column, so declaring a TD on
created_at doesn't trip MeasureNameCollidesWithColumnError. Joined TDs
flatten through __ paths and so don't collide with host source columns.
stage_planner.plan_query: _declared_measures_from_query now iterates
query.time_dimensions between dimensions and measures (legacy
user_projection order: dims → time dims → measures). Each TD becomes a
DeclaredMeasure with flat declared_name / public_name; label propagates.
Codex review of tests caught 4 findings: added joined-TD label
propagation, multi-hop joined TD coverage (orders → customers →
regions), tightened stage schema membership assertions to exact-match
shape, and explicit derived-Column.sql rejection coverage. Codex review
of impl returned 1 MEDIUM (cross_model_planner TimeTruncKey
classification, deferred to 7b.5 when cross-model TD routing surfaces)
and 2 LOW findings deferred (data_type_bucket reuse would introduce an
import cycle through schema_drift → ingestion → query_engine;
NotImplementedError is acceptable for a documented temporary limit).
26 new tests in tests/test_time_dimensions_planner.py.
3427 unit tests passing, 0 regressions, ruff clean. Integration tests
not run locally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two pieces wired into stage_planner.plan_query:
1. ``_build_date_range_filter(td, scope, bundle)`` converts a
TimeDimension's ``date_range=[start, end]`` into a row-phase
BoundFilter whose predicate is ``ArithmeticKey("and", (>=, <=))``
over the BARE underlying ColumnKey. Bound literals normalize via
``normalize_scalar`` (strings pass through). The filter binds to
the raw column, not the TimeTruncKey, so generator slice 7b.11 can
apply it on the outer projection while the shifted self-join CTE
reads unfiltered raw data (legacy edge-period semantics for
time_shift / change / change_pct).
FilterPhase.expression is populated with PlannedBoundExpr for
auto-generated filters (date_range has no user text). User-filter
expression population stays deferred to stage 7b.6
(BoundExpr type unification).
Malformed date_range entries (``[]``, ``[single]``) silently no-op,
matching legacy ``slayer/sql/generator.py:2517``.
2. ``_resolve_main_time_dimension(query, model)`` resolves the active
TD for transform / windowing semantics. 0 TDs → None; 1 TD →
that TD (ignores main_time_dimension, matches legacy); 2+ TDs with
main_time_dimension → match by full_name, fall back to leaf,
ambiguous leaf raises AmbiguousReferenceError, unknown raises
UnknownReferenceError. 2+ TDs with model.default_time_dimension →
leaf match restricted to host-local TDs (legacy
``_resolve_time_alias`` returns f"{model}.{default}" so the joined
case is never selected through that path). Neither → None.
Codex review of impl caught two MEDIUM findings folded in here:
ambiguous-leaf matches now raise AmbiguousReferenceError listing both
candidates rather than returning by query order; default_time_dimension
matching restricts to host-local TDs to preserve legacy alias
semantics.
snap_to_whole_periods ownership stays with SlayerQuery's existing
method called pre-normalization in query_engine._execute_pipeline:595;
the planner consumes already-snapped queries and never re-snaps.
25 new tests in tests/test_time_dimensions_filters.py covering:
helper resolution (0/1/multi TDs, full-name vs leaf precedence,
ambiguity, default-host-vs-joined, unknown default no-op);
date_range filter shape (AND-of-comparisons, row-phase, expression
populated, binds raw ColumnKey not TimeTruncKey, hidden raw-column
slot materialized, joined TD path-bearing); malformed date_range
emits no filter; multiple TDs each emit their own filter; user
filters + date_range coexist with date_range last.
3452 unit tests passing, 0 regressions, ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes:
1. Per-op binder validation in slayer/engine/binding.py:_bind_transform.
New _TRANSFORM_KWARG_RULES whitelist per op (partition_by is
special-cased and accepted everywhere it makes sense). ntile
requires positive integer n (rejects bool, float-with-fractional,
negative, zero); time_shift requires periods; lag/lead default
periods=normalize_scalar(1) when missing. Transforms take exactly
one positional argument (the value); anything else raises forcing
the kwarg form (e.g. ``lag(value, periods=2)`` instead of
``lag(value, 2)``). New _fold_to_scalar helper folds Literal and
UnaryOp(-, Literal) into normalised scalars; non-scalar shapes raise
(TransformKey.kwargs only accepts Scalar, not arbitrary ValueKey).
2. _iter_slot_deps in slayer/engine/planning.py now walks
TransformKey.partition_keys and TransformKey.time_key so partition
columns and time-key columns surface as slot deps. The
ProjectionPlanner now walks measure deps too and interns them as
hidden slots, materialising inner aggregates / partition columns /
time-key columns for the generator to consume.
3. New lower_sugar_transforms(value_key) -> value_key recursive walker
in planning.py. Replaces TransformKey(op="change"|"change_pct") with
the desugared arithmetic form, preserving inner aggregate identity
(DEV-1446). Wired into stage_planner._declared_measures_from_query.
desugar_change / desugar_change_pct now set
kwargs=(("periods", normalize_scalar(-1)),) on the produced
time_shift TransformKey, matching the new typed invariant that
time_shift requires explicit periods.
4. New _emit_transform_layers in stage_planner — one TransformLayer
per TransformKey slot in topological dependency order (Kahn's
algorithm). Nested transforms like ``cumsum(change(amount:sum))``
emit the inner time_shift layer before the outer cumsum layer; the
generator slices can render windows / self-joins in the right order
without re-walking. Per-slot transform metadata (partition_keys /
time_key / args / kwargs) lives on the slot's TransformKey so
TransformLayer stays minimal.
Codex review caught: positional-args ambiguity (now rejected), missing
periods in desugar output (now -1 explicitly), and op-grouping
collapsing nested same-op layers (now per-slot topological).
Deferred: partition_by=[list, of, cols] parser syntax (Mode-B parser
extension scope); consecutive_periods.period value validation (defer
until generator slice 7b.11 consumes it).
29 new tests in tests/test_transforms_planner.py covering per-op
validation (ntile.n positive int + bool reject + integer-only;
time_shift periods required; lag default; unknown kwargs on rank-family
+ consecutive_periods; positional-arg rejection); transform aux slot
materialisation (single + multi partition keys, time_key
ColumnKey/TimeTruncKey); transform_layers population (per-slot, topo
order, no op-grouping); change → time_shift lowering identity
preservation (DEV-1446 nested-transform dedup).
3481 unit tests passing, 0 regressions, ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the dormant IsolatedCteCrossModelPlanner into plan_query. 1. New filter_referenced_slot_ids(bound_filter, registry) -> set[SlotId] in slayer/engine/planning.py. Walks _iter_slot_deps and looks each slottable key up in the registry. Codex HIGH #3/#4 fold-in: does not mutate BoundFilter.referenced_keys (those are pre-interning ValueKeys); does walk composite predicates so "rev >= 100 AND customers.revenue:sum < 500" resolves both leaves. Silently skips literals and other non-slottable refs. 2. stage_planner.plan_query now, after filters_by_phase is built, constructs HostFilterRouting records (with referenced_slot_ids sorted for deterministic plan snapshots — Codex LOW #3 fold-in) and iterates aggregate slots. For every AggregateKey slot whose source.path is non-empty, invokes cross_model_planner.plan() with the host_slots + host_filter_routings + slot.public_name + slot.hidden, and appends the resulting CrossModelAggregatePlan to PlannedQuery.cross_model_aggregate_plans. 3. cross_model_planner._aggregate_alias now uses slayer.core.refs.canonical_agg_name so parameterised aggregates (revenue:percentile(p=0.5) vs p=0.95) get distinct CTE column aliases (Codex HIGH #1 fold-in). Test exercises this with two percentile aggregates. Deferred to follow-ups: - Cross-model aggregates with column-valued kwargs (weighted_avg / corr crossing a join) — the host-local column ref can't be evaluated inside the customers CTE. Known limitation; documented for a follow-up issue. (Codex HIGH #2 in the impl review.) - FilterPhase.text / HostFilterRouting.text population for user filters — depends on the BoundExpr unification in stage 7b.6. 12 new tests in tests/test_cross_model_planner_wiring.py covering filter_referenced_slot_ids (simple column, composite predicate, no composite-only nodes in result, unknown slot silently skipped); plan_query wiring (local agg → no plan; cross-model agg → emits plan with target_model/datasource; aggregate_slot_id matches the slot; two distinct aggregates emit separate plans; parameterised aggs get distinct CTE aliases; local filter → DROP_HOST_LOCAL → no propagation; target model filters propagate); classify_host_filter exercise. 3493 unit tests passing, 0 regressions, ruff clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
slayer.engine.planned.BoundExpr was a separate Pydantic class with an optional sql_text cache; slayer.engine.binding.BoundExpr was the binder's typed output. Two classes meant ValueSlot.expression and FilterPhase.expression couldn't carry the binder's payload without type widening or a side adapter (Codex HIGH F2 in the earlier round). Unification: 1. slayer.engine.planned now re-exports slayer.engine.binding.BoundExpr as ``BoundExpr``. Single canonical class going forward. 2. The render-artifact ``sql_text`` field is dropped. Generator slices render from the typed value_key against the slot registry, not a cached string. The branch-new tests/test_planned.py:127 ``test_with_expression_payload`` is updated to assert against the value_key shape instead of sql_text. 3. ValueRegistry.intern now accepts an optional ``expression`` argument; ValueSlot.expression is populated for every materialised slot (public AND hidden) — auto-defaulted to BoundExpr(value_key=key) when no explicit binder output is passed. 4. stage_planner.plan_query now populates FilterPhase.expression for EVERY filter (user-supplied AND auto-generated date_range), so the SQL generator can render filters without re-parsing or consulting a side map. The previous auto_filter_ids tracking is removed (always-populate is simpler and matches the unification goal). 7 new tests in tests/test_boundexpr_unification.py covering: the type re-export identity; ValueSlot.expression populated for measures, dimensions, AND hidden filter-dep slots; FilterPhase.expression populated for user filters AND aggregate-phase (HAVING) filters; the expression's value_key matches the slot's / filter's key identity. 3500 unit tests passing, 0 regressions, ruff clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a typed resolution pipeline: typed keys/phases, Mode‑B parser and Mode‑A SQL canonicalizer, measure expansion, binder, projection/planning (ValueRegistry, ProjectionPlanner), cross‑model planning, source‑bundle builder, normalization rules, engine wiring, response metadata, sync derived‑column expander, structured errors/warnings, docs, and extensive tests. ChangesDEV-1450 Typed Pipeline and Planning
Sequence Diagram(s)sequenceDiagram
rect rgba(102, 153, 255, 0.5)
participant Client
participant Engine
end
rect rgba(153, 204, 255, 0.5)
participant Storage
participant SourceBundle
end
rect rgba(204, 255, 204, 0.5)
participant Parser
participant Binder
participant Planner
end
rect rgba(255, 204, 153, 0.5)
participant SQLGen
participant DB
end
Client->>Engine: execute(query)
Engine->>Storage: build_resolved_source_bundle
Storage-->>Engine: models
Engine->>Parser: normalize + parse
Parser-->>Engine: ParsedExpr
Engine->>Binder: bind_expr/bind_filter/bind_time_dimension
Binder-->>Engine: BoundExpr/BoundFilter
Engine->>Planner: plan_query/plan_stages
Planner-->>Engine: PlannedQuery
Engine->>SQLGen: generate_from_planned
SQLGen->>DB: run SQL
DB-->>Engine: rows
Engine-->>Client: SlayerResponse + warnings
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches🧪 Generate unit tests (beta)
|
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (8)
tests/test_time_trunc_key.py (1)
109-112: ⚡ Quick winSpecify the exact exception type for frozen instance mutation.
The test uses
pytest.raises(Exception)which is too broad. Specify the exact exception type that's raised when attempting to mutate a frozen Pydantic model or dataclass field (typicallyValidationErrorfor Pydantic v2 orFrozenInstanceErrorfor frozen dataclasses).♻️ Proposed fix to specify exception type
+from pydantic import ValidationError + class TestTimeTruncKeyImmutability: def test_is_frozen(self) -> None: k = TimeTruncKey(column=ColumnKey(leaf="ordered_at"), granularity="month") - with pytest.raises(Exception): + with pytest.raises(ValidationError): k.granularity = "day" # type: ignore[misc]Note: Use
ValidationErrorifTimeTruncKeyis a Pydantic model, ordataclasses.FrozenInstanceErrorif it's a frozen dataclass.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_time_trunc_key.py` around lines 109 - 112, The test test_is_frozen should assert the specific exception type instead of pytest.raises(Exception): replace the broad Exception with the concrete exception thrown when mutating a frozen TimeTruncKey (use pydantic.errors.ValidationError / pydantic.ValidationError if TimeTruncKey is a Pydantic model, or dataclasses.FrozenInstanceError if it is a frozen dataclass), update the test import(s) accordingly, and keep the mutating line k.granularity = "day" and references to TimeTruncKey and ColumnKey unchanged.slayer/engine/path_resolution.py (1)
75-75: ⚡ Quick winPrefer more specific type annotation.
The
nqvariable is assigned a dict value, sonq: dict[str, Any]would be clearer thannq: Any.♻️ Suggested improvement
- nq: Any = named_queries if named_queries is not None else {} + nq: dict[str, Any] = named_queries if named_queries is not None else {}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/engine/path_resolution.py` at line 75, The local variable `nq` is annotated too broadly as `Any`; change its annotation to a more specific mapping type (e.g., `dict[str, Any]`) so callers and linters know it holds a dict produced from `named_queries`. Update the assignment `nq: Any = named_queries if named_queries is not None else {}` to use `nq: dict[str, Any] = ...` (or use `cast(dict[str, Any], named_queries)` if necessary) and ensure `typing` imports include `Any`/`dict` typing forms compatible with the project's Python version.tests/test_path_resolution.py (1)
27-34: 💤 Low valueConsider using keyword arguments consistently in test calls.
The
_make_modelhelper has 2 parameters. Per coding guidelines, functions with more than 1 parameter should be called using keyword arguments. Several calls use positional arguments (e.g., lines 160, 175, 191).Example:
# Current return _make_model(model_name) # Preferred per guideline return _make_model(name=model_name)As per coding guidelines: "Use keyword arguments for functions with more than 1 parameter"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_path_resolution.py` around lines 27 - 34, The test helper _make_model accepts parameters (name, joins=None) but several test calls use positional args; update all usages to use keyword arguments per guideline: call _make_model(name=..., joins=...) instead of positional calls (e.g., replace _make_model(model_name) with _make_model(name=model_name) and any two-argument positional calls with explicit joins=...), locating usages by the function name _make_model in the tests and editing each call.tests/test_source_bundle.py (1)
20-29: 💤 Low valueConsider using keyword arguments consistently in test calls.
The
_modelhelper has 2 parameters. Per coding guidelines, functions with more than 1 parameter should be called using keyword arguments for clarity. Multiple calls throughout this file use positional arguments (e.g., lines 39, 49, 50, 56, 64, 74, 83, etc.).Example:
# Current m = _model("orders") m = _model("orders", ds="warehouse") # Preferred per guideline m = _model(name="orders") m = _model(name="orders", ds="warehouse")As per coding guidelines: "Use keyword arguments for functions with more than 1 parameter"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_source_bundle.py` around lines 20 - 29, The test helper _model(name: str, ds: str = "prod") is being invoked with positional arguments in multiple places; update all calls to use keyword arguments (e.g., _model(name="orders") or _model(name="orders", ds="warehouse")) so they follow the guideline for functions with more than one parameter; search for usages of _model in tests/test_source_bundle.py and replace positional invocations with keyword form, leaving the SlayerModel construction (and related Column/DataType references) unchanged.tests/test_boundexpr_unification.py (1)
161-161: ⚡ Quick winMove function-local imports to module scope.
Line 161 and Line 177 import inside test methods; this violates the repo rule to keep imports at the top of the file.
Suggested diff
from slayer.core.enums import DataType -from slayer.core.keys import AggregateKey, ColumnKey +from slayer.core.keys import AggregateKey, ArithmeticKey, ColumnKey from slayer.core.models import Column, SlayerModel from slayer.core.query import SlayerQuery from slayer.engine.binding import BoundExpr as BinderBoundExpr +from slayer.engine.binding import walk_value_keys from slayer.engine.planned import BoundExpr as PlannedBoundExpr from slayer.engine.source_bundle import ResolvedSourceBundle from slayer.engine.stage_planner import plan_query @@ - from slayer.core.keys import ArithmeticKey assert isinstance(fp.expression.value_key, ArithmeticKey) @@ - from slayer.engine.binding import walk_value_keys keys = list(walk_value_keys(fp.expression.value_key))As per coding guidelines: "
**/*.py: Place imports at the top of files".Also applies to: 177-177
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_boundexpr_unification.py` at line 161, Move the function-local imports to module scope: remove the in-test import of ArithmeticKey (from slayer.core.keys) and any other imports currently performed inside test functions and place them at the top of tests/test_boundexpr_unification.py; update the module-level import section to include "from slayer.core.keys import ArithmeticKey" (and any other symbols currently imported inside tests) so tests reference those symbols without local imports.tests/test_slack_normalization.py (2)
249-257: ⚡ Quick winAvoid
run_until_completein pytest async suites.The fixture manually drives the loop; this is fragile when a loop is already managed by pytest-asyncio. Prefer an async fixture and
awaitthe setup calls directly.As per coding guidelines "
**/tests/**/*.py: Tests must usepytest-asynciowithasyncio_mode = \"auto\"so test functions can beasync defandawaitdirectly."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_slack_normalization.py` around lines 249 - 257, The test uses asyncio.get_event_loop().run_until_complete to call storage.save_datasource and storage.save_model, which conflicts with pytest-asyncio-managed loops; change the fixture to be async (make the fixture function async def) and replace those run_until_complete calls with direct awaits: await storage.save_datasource(DatasourceConfig(...)) and await storage.save_model(_orders()), ensuring the fixture yields/returns as before so pytest-asyncio (asyncio_mode="auto") manages the event loop correctly.
225-229: ⚡ Quick winMove test imports to module scope.
These in-function imports make the test module inconsistent with repo style and harder to scan. Please hoist them to the top-level import block.
As per coding guidelines "
**/*.py: Place imports at the top of files."Also applies to: 249-257, 263-271, 287-295, 314-318, 343-351
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_slack_normalization.py` around lines 225 - 229, Several tests in test_slack_normalization.py contain in-function imports (e.g., DatasourceConfig, SlayerQueryEngine, YAMLStorage, sqlite3) — hoist these imports to the module-level import block at the top of the file, remove the now-redundant local imports inside the test functions, and consolidate/organize them with the existing top imports; repeat this change for the other in-function import sites flagged (around the commented ranges) to comply with the project import style and keep test-scoped symbols available to all functions.tests/test_dot_path_in_sql.py (1)
474-474: ⚡ Quick winHoist local imports to the module import section.
These test-local imports should be moved to top-level to match project conventions.
As per coding guidelines "
**/*.py: Place imports at the top of files."Also applies to: 489-490
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_dot_path_in_sql.py` at line 474, The test contains local imports (e.g., "from slayer.core.models import ModelMeasure" and the other imports around the later test block) that should be hoisted to the module-level import section; move these local imports to the top of the file with the other imports, remove the in-function/local import statements, and run the tests to ensure no circular import issues—if a circular dependency appears, refactor the import to a lazy import helper or import only the needed attribute at top-level.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@slayer/engine/binding.py`:
- Around line 155-156: The call to _bind uses a positional first argument;
change it to use keyword arguments for all parameters (e.g.,
_bind(parsed=parsed, scope=scope, bundle=bundle, in_filter=False)) and update
every other _bind invocation in this module (including the ones around the other
mentioned call sites) to the same keyword style so it conforms to the repo rule;
keep the surrounding return (BoundExpr(value_key=...)) unchanged.
In `@slayer/engine/syntax.py`:
- Around line 180-188: The current check uses _OVER_RE.search(text) and flags
any "OVER(" even when inside string literals (e.g. "status == 'OVER('"); change
the check to ignore matches inside quotes by scanning text
character-by-character tracking single-quote, double-quote, and escape state and
only applying _OVER_RE when not inside a quoted string, then raise
IllegalWindowInFilterError(filter_expr=text, source=..., suggestion=...) as
before; update the code path that currently calls _OVER_RE.search(text) (the
block raising IllegalWindowInFilterError) so it uses the new quote-aware
detection routine on the variable text.
- Around line 409-412: The code silently ignores keyword unpacking (**kwargs)
because the comprehension in _convert_call filters out ast.keyword entries with
arg is None; update _convert_call to detect any kw in node.keywords with kw.arg
is None and raise a clear error (e.g., ValueError or a parsing-specific
exception) rejecting **kwargs in Mode-B call parsing instead of dropping them,
then proceed to build kwargs by converting remaining keywords via _convert;
reference node.keywords, _convert_call, and _convert when implementing the check
and error raise.
In `@tests/test_agg_registry.py`:
- Around line 62-64: Update the test calls in tests/test_agg_registry.py to use
keyword arguments instead of positional arguments for helper functions with more
than one parameter: change calls to collect_reachable_agg_names(...),
resolve_aggregation(...), merge_agg_params(...), and any other multi-parameter
helper invocations (including the occurrences at lines noted in the comment) so
that each argument is passed as param_name=value (e.g., m=...,
resolve_join_target=..., named_queries=...) rather than by position; ensure you
update all instances mentioned (around the ranges 62-64, 79-81, 99-101, 117,
177, 183, 230, 238, 247, 257, 264) and run tests to confirm no signature
mismatches.
In `@tests/test_binding.py`:
- Around line 145-147: Update all calls to bind_expr and bind_filter (and
similar multi-parameter helpers) to pass the parsed expression/result as a
keyword instead of a positional first argument; e.g., replace
bind_expr(parse_expr("amount"), scope=_scope(), bundle=_bundle()) with a keyword
form like bind_expr(parsed=parse_expr("amount"), scope=_scope(),
bundle=_bundle()) (and do the same for bind_filter), applying this change at
every location listed in the comment so all multi-parameter calls use keyword
arguments for clarity and to follow the repository standard.
In `@tests/test_stage_planner.py`:
- Around line 203-220: The test test_stages_in_dependency_order currently
constructs stage1 and stage2 but passes them to plan_stages in dependency order,
so it doesn't verify reordering; update the test to pass stages in reverse order
(pass stage2 then stage1) to force the planner to reorder, and add an assertion
that the planned result is topologically correct (e.g., check planned[0].name ==
"stage1" and planned[1].name == "stage2" or equivalent checks on the planned
stages returned by plan_stages) so the test fails if plan_stages does not
reorder dependencies; references: test_stages_in_dependency_order, plan_stages,
SlayerQuery, planned.
---
Nitpick comments:
In `@slayer/engine/path_resolution.py`:
- Line 75: The local variable `nq` is annotated too broadly as `Any`; change its
annotation to a more specific mapping type (e.g., `dict[str, Any]`) so callers
and linters know it holds a dict produced from `named_queries`. Update the
assignment `nq: Any = named_queries if named_queries is not None else {}` to use
`nq: dict[str, Any] = ...` (or use `cast(dict[str, Any], named_queries)` if
necessary) and ensure `typing` imports include `Any`/`dict` typing forms
compatible with the project's Python version.
In `@tests/test_boundexpr_unification.py`:
- Line 161: Move the function-local imports to module scope: remove the in-test
import of ArithmeticKey (from slayer.core.keys) and any other imports currently
performed inside test functions and place them at the top of
tests/test_boundexpr_unification.py; update the module-level import section to
include "from slayer.core.keys import ArithmeticKey" (and any other symbols
currently imported inside tests) so tests reference those symbols without local
imports.
In `@tests/test_dot_path_in_sql.py`:
- Line 474: The test contains local imports (e.g., "from slayer.core.models
import ModelMeasure" and the other imports around the later test block) that
should be hoisted to the module-level import section; move these local imports
to the top of the file with the other imports, remove the in-function/local
import statements, and run the tests to ensure no circular import issues—if a
circular dependency appears, refactor the import to a lazy import helper or
import only the needed attribute at top-level.
In `@tests/test_path_resolution.py`:
- Around line 27-34: The test helper _make_model accepts parameters (name,
joins=None) but several test calls use positional args; update all usages to use
keyword arguments per guideline: call _make_model(name=..., joins=...) instead
of positional calls (e.g., replace _make_model(model_name) with
_make_model(name=model_name) and any two-argument positional calls with explicit
joins=...), locating usages by the function name _make_model in the tests and
editing each call.
In `@tests/test_slack_normalization.py`:
- Around line 249-257: The test uses asyncio.get_event_loop().run_until_complete
to call storage.save_datasource and storage.save_model, which conflicts with
pytest-asyncio-managed loops; change the fixture to be async (make the fixture
function async def) and replace those run_until_complete calls with direct
awaits: await storage.save_datasource(DatasourceConfig(...)) and await
storage.save_model(_orders()), ensuring the fixture yields/returns as before so
pytest-asyncio (asyncio_mode="auto") manages the event loop correctly.
- Around line 225-229: Several tests in test_slack_normalization.py contain
in-function imports (e.g., DatasourceConfig, SlayerQueryEngine, YAMLStorage,
sqlite3) — hoist these imports to the module-level import block at the top of
the file, remove the now-redundant local imports inside the test functions, and
consolidate/organize them with the existing top imports; repeat this change for
the other in-function import sites flagged (around the commented ranges) to
comply with the project import style and keep test-scoped symbols available to
all functions.
In `@tests/test_source_bundle.py`:
- Around line 20-29: The test helper _model(name: str, ds: str = "prod") is
being invoked with positional arguments in multiple places; update all calls to
use keyword arguments (e.g., _model(name="orders") or _model(name="orders",
ds="warehouse")) so they follow the guideline for functions with more than one
parameter; search for usages of _model in tests/test_source_bundle.py and
replace positional invocations with keyword form, leaving the SlayerModel
construction (and related Column/DataType references) unchanged.
In `@tests/test_time_trunc_key.py`:
- Around line 109-112: The test test_is_frozen should assert the specific
exception type instead of pytest.raises(Exception): replace the broad Exception
with the concrete exception thrown when mutating a frozen TimeTruncKey (use
pydantic.errors.ValidationError / pydantic.ValidationError if TimeTruncKey is a
Pydantic model, or dataclasses.FrozenInstanceError if it is a frozen dataclass),
update the test import(s) accordingly, and keep the mutating line k.granularity
= "day" and references to TimeTruncKey and ColumnKey unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 2ce23cea-7e91-49fa-a808-715b179c12a0
📒 Files selected for processing (43)
slayer/core/errors.pyslayer/core/keys.pyslayer/core/scope.pyslayer/core/warnings.pyslayer/engine/agg_registry.pyslayer/engine/binding.pyslayer/engine/cross_model_planner.pyslayer/engine/measure_expansion.pyslayer/engine/normalization.pyslayer/engine/path_resolution.pyslayer/engine/planned.pyslayer/engine/planning.pyslayer/engine/query_engine.pyslayer/engine/source_bundle.pyslayer/engine/stage_planner.pyslayer/engine/syntax.pyslayer/engine/variables.pyslayer/sql/sql_expr.pytests/test_agg_registry.pytests/test_binding.pytests/test_boundexpr_unification.pytests/test_cross_model_planner.pytests/test_cross_model_planner_wiring.pytests/test_dot_path_in_sql.pytests/test_error_messages.pytests/test_keys.pytests/test_model_measure_expansion.pytests/test_path_resolution.pytests/test_planned.pytests/test_projection_planner.pytests/test_scope_schema.pytests/test_slack_normalization.pytests/test_source_bundle.pytests/test_sql_expr.pytests/test_stage_planner.pytests/test_syntax.pytests/test_time_dimensions_filters.pytests/test_time_dimensions_planner.pytests/test_time_trunc_key.pytests/test_transform_lowerer.pytests/test_transforms_planner.pytests/test_value_registry.pytests/test_variables_planner.py
Original scope was a slayer/sql/parity_adapter.py mapping PlannedQuery
back to EnrichedQuery so the legacy SQLGenerator could render it as the
oracle for upcoming generator slices (7b.8-7b.13). A closer read of
slayer/engine/enrichment.py (2300 lines of resolution logic --
derived-column expansion, alias provenance, filter classification,
cross-model rerooting) made it clear that a faithful adapter would
duplicate the bulk of that file in throwaway test-only code, on top of
code already destined for deletion at end of 7b.15.
Pivot: drop the adapter entirely. Each upcoming slice writes parity
tests of the shape
legacy = await legacy_sql_for(engine, model, q)
new = generate_from_planned(plan_query(q, bundle), dialect=...)
assert_sql_equivalent(legacy, new)
This stage lands the shared helpers each slice imports:
* tests/parity_oracle.py
- legacy_sql_for(engine, model, query, named_queries=, dialect=)
routes through engine._enrich + SQLGenerator.generate, the
production legacy code path.
- assert_sql_equivalent(legacy, new) does whitespace-canonical
comparison with a token-level unified-diff on mismatch.
- norm_sql(sql) collapses runs of whitespace.
- build_storage_with_models(tmp_path, *models) seeds a YAMLStorage.
* tests/test_parity_oracle.py
- 9 smoke tests covering norm_sql, assert_sql_equivalent
(pass-on-whitespace-diff, raise-on-real-diff, pass-on-identity),
legacy_sql_for non-empty output, deterministic re-runs, and
joined-dim rendering through the helper.
Codex impl-review fold-ins:
* HIGH: legacy_sql_for now accepts named_queries= so multi-stage and
cross-model parity (where production passes a name -> SlayerQuery
map) works through the helper.
* MEDIUM: legacy_sql_for accepts an optional dialect= and threads it
through to both _enrich and SQLGenerator(dialect=...) so non-postgres
parity is on the table for later slices.
* MEDIUM: build_storage_with_models docstring rewritten to reflect
that save-time join-target validation is permissive, so order is a
convention not a requirement.
Plan amendments (in /home/james/.claude/plans/):
* read-the-linear-issue-mutable-crane.md - 7b.7 redefined as "parity
oracle test helpers"; 7b.8-7b.13 slice descriptions updated to
reference legacy_sql_for directly; 7b.15 safe-deletes list swaps
slayer/sql/parity_adapter.py for tests/parity_oracle.py +
tests/test_parity_oracle.py.
3509 unit tests passing, 0 regressions across the suite, ruff clean.
The legacy /enriched code path stays untouched; this commit only adds
test-side helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First SQL-generator slice on the typed pipeline. Adds `SQLGenerator.generate_from_planned(planned_query, *, bundle)` plus module-level `generate_from_planned(planned_query, *, bundle, dialect)` shim alongside the legacy `generate()` path. Mirrors the local-only branch of `_generate_base` but reads typed `PlannedQuery` fields (`row_slots` / `aggregate_slots` / `filters_by_phase` / `order`) instead of `EnrichedQuery`, and reuses legacy dialect helpers (`_resolve_sql` / `_build_agg` / `_wrap_cast_for_type` / `_parse_predicate` / `_apply_order_limit`) via a synthetic- `EnrichedMeasure` adapter so dialect parity holds with one emission codebase. Scope: single-model queries with row-phase dims, local aggregates, Mode-B row filters (`status == 'paid'`), ORDER BY a declared measure/dim alias, LIMIT/OFFSET, and dim-only deduplication. Cross-model, time dimensions, transforms, HAVING-phase filters, `column_filter_key`, hidden ORDER BY targets, and `*:<non-count>` all raise `NotImplementedError` with an explicit `DEV-1450 stage 7b.9+`/`7b.10+`/`7b.12` marker so silent SQL parity drift is impossible. Two planner-side gap fixes flagged in the 7b.7 checkpoint: * ORDER BY resolution against declared-measure aliases — new `(public_name, declared_name, canonical_alias) -> bound` map built from declared measures; order pass checks it before falling back to `bind_expr` (aggregate canonical aliases like `amount_sum` aren't columns, so the binder would have raised). * Pre-bind `ModelMeasure` expansion wired into `_declared_measures_from_query` via `expand_model_measures` (gated on `ModelScope` with a non-None `source_model` — downstream StageSchema stages don't expose saved measures). Tests (branch-new `tests/test_generator2_local.py`): * 13 parametrised parity fixtures + 5-dialect smoke (postgres, sqlite, duckdb, mysql, clickhouse) — each asserts `assert_sql_equivalent(legacy_sql_for(...), generate_from_planned(...))` via the 7b.7 parity oracle. * Regression tests for the four 7b.7-checkpoint planner gaps: ORDER BY canonical alias, `ModelMeasure` expansion, `column_filter_key` rejection (hand-built + xfail for the real planner-path gap), and multi-alias same-key (no-parity, direct shape assertion). * DEV-1443 collision invariants preserved. Codex review fold-ins (two rounds): * test review: ORDER BY without LIMIT case; `count_distinct` + GROUP BY case; column_filter_key xfail covering the real planner gap rather than only the hand-built guard. * impl review round 1: model.filters defer guard (HIGH); `cross_model_aggregate_plans` upfront guard (MEDIUM); unary minus handling in `_build_arithmetic_for_filter` (MEDIUM); custom/parameterised aggregations defer through new `_BUILTIN_BAREARG_AGGS_LOCAL_SLICE` constant (LOW). * impl review round 2: `*:<non-count>` rejection in synthetic measure adapter (MEDIUM); hidden ORDER BY slot defer (MEDIUM). 3533 unit tests pass (24 new + 1 xfailed for the deferred Column.filter planner gap), 0 regressions. Ruff clean. Integration tests not run locally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends generate_from_planned to render TimeTruncKey row-phase slots via _build_date_trunc (Postgres/DuckDB/MySQL/ClickHouse DATE_TRUNC, SQLite STRFTIME), and folds SlayerModel.filters (Mode-A SQL always-applied WHERE) into the same entry point with legacy WHERE ordering (date_range -> model.filters -> query.filters). Introduces BetweenKey, a typed value-key for col BETWEEN low AND high. The planner's _build_date_range_filter switches from ArithmeticKey(and, [GE, LE]) to BetweenKey, closing the syntactic parity gap with legacy generator.py:2533 (which emits BETWEEN). User- written `col >= a and col <= b` stays as ArithmeticKey -- the DSL parser never produces BetweenKey, so user-filter parity is preserved. stage_planner._validate_model_filter validates each entry via parse_sql_predicate (rejects DSL constructs, raw OVER), rejects refs to measures (enrichment.py:1147 parity), rejects refs to windowed columns (enrichment.py:1205 parity), and rejects refs to non-trivial derived Column.sql columns (deferred to follow-up; trivial-base columns per _is_trivial_base pass through). Filter is emitted as a text-only FilterPhase with text_columns extracted from ParsedFilter; the new _qualify_mode_a_sql_filter in the generator regex-prepends <source_relation>. to each bare-identifier ref, bit-identical to legacy _build_where_and_having:2566-2580. Test count: 3582 (+49 over 7b.8). 48 tests in tests/test_generator2_time_dims.py cover the granularity sweep (PG + SQLite, 8 each), TD + measures, date_range, multi-TD disambiguation, ORDER-BY-on-TD, model.filters with all rejection variants, whole_periods_only pre-snap integration, dialect cycle smoke, and round-2/round-3 codex regression cases. 1 trivial-base regression test pins _is_trivial_base parity. 7b.3c filter shape tests in tests/test_time_dimensions_filters.py updated to the new BetweenKey shape and user-filter-ordering invariant. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renders cumsum / lag / lead / rank / percent_rank / dense_rank / ntile / first / last through generate_from_planned via a Kahn-batched step CTE chain wrapped in an outer SELECT (legacy _generate_with_computed shape). Auto-partition by query dimensions only (not TDs); rank-family defaults to no PARTITION BY. POST-phase filters route through a _filtered wrapper between the chain and pagination. Planner side now attaches the active TD as TransformKey.time_key for every time-needing transform (cumsum / lag / lead / first / last / time_shift / consecutive_periods / change / change_pct), closing the 7b.4 carry-over gap. Lowering of change / change_pct runs after the patch so the desugared time_shift inherits the resolved time_key. Validation mirrors legacy enrichment.py:564 -- any unresolved time-needing transform raises "requires an unambiguous time dimension". PlannedQuery gains active_time_dimension_slot_id so the generator can look up the TD slot's alias without re-walking the model graph. time_shift / consecutive_periods / change(lowered) raise NotImplementedError with "7b.11" markers; composite transform inputs (e.g., cumsum(amount:sum / qty:sum)) raise with a follow-up marker. Codex impl-review folded in: list-per-slot alias tracking so duplicate public aliases (DEV-1450 C13) survive the CTE chain; strict lag / lead periods validation rejecting bool / non-integral; POST filter operator coverage (= / <> / unary - / n-ary and-or / typed scalar literals); order-only hidden refs materialised in the base CTE; BoundFilter.referenced_keys recomputed after time-key patching. 40 new tests in tests/test_generator2_window.py (38 pass + 1 typed-only skip for a pre-existing 7b.8 ModelMeasure.type gap + 1 covering nested cases). Full unit suite: 3621 passed, 3 skipped, 1 xfailed (+39 / +1 / 0 vs baseline); ruff clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…+ NOSONARs Three follow-ups from round-7 review: 1. Codex MAJOR: ``_render_aggregate_composite_expr``'s ``ScalarCallKey`` branch dropped ``composite_alias_by_key`` when recursing into operands (the ``ArithmeticKey`` branch forwarded it; only the ``ScalarCallKey`` recursion missed it). A filtered first/last operand inside ``coalesce(...)`` would then fall back to the placeholder ``__op__`` alias and miss the ``filtered_rn_map`` lookup. Forward ``composite_alias_by_key`` consistently in all three recursion branches. 2. Sonar quality-gate ERROR on ``new_duplicated_lines_density`` (4.0% > 3.0%): the round-6 cross-model AGG-phase filter materialization block was near-identical to the round-5 transform-deps block. Factor both into a single ``_add_local_aux_slots(include_order=, aggregates_only=)`` inner helper. Same control flow, one body. 3. Sonar S3776 NOSONAR markers on ``_render_aggregate_composite_expr`` (complexity 37 — sequential ValueKey dispatch with rn-state + composite-alias-by-key threading) and ``_render_with_cross_model_plans`` (complexity 80 — orchestration of host ``_base`` CTE + per-plan ``_cm_*`` CTEs + combined SELECT + transform chain + outer wrap). Also: reply on the CodeRabbit POST-drop re-flag thread (the third time this round) — the ``has_transforms`` guard from dc7a775 remains sufficient; POST in no-transform is planner-unreachable. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Codex review on PR #158 caught that the kwarg scanner classifies a call paren by callee name alone, missing the colon-aggregation context. The names `first` and `last` sit in both ALL_TRANSFORMS (FIRST_VALUE window) and the built-in aggregation set (`_AMBIGUOUS_AGG_TRANSFORMS` in `slayer/core/formula.py`), and after a `:` they are always aggregations — never transforms. So `revenue:first(order=created_at) > 0` was landing in the transform branch and the `order=` kwarg was being rewritten to `order==`. Detect colon-agg context in `_classify_paren` by checking whether `hist[-2]` is the `:` token. When so, drop the callee to `None` so `_is_kwarg_equals` takes the aggregation/unknown branch (kwargs after `(` or `,`). The `hist` cap of 2 retains exactly the prev-prev token needed; longer prefixes like `customers.revenue:first(...)` roll off without affecting the detection. Built-in `first`/`last` take their order column positionally today (`revenue:last(ordered_at)`), so the user-visible impact is on custom user-defined aggregations that override these names; the test pins the architectural rule with a concrete kwarg-shape input. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three follow-ups from round-8 review: 1. Codex MAJOR (slayer/sql/generator.py:6304+): a DERIVED first/last time arg (``ColumnSqlKey``) skipped join discovery. ``_resolve_explicit_time_col`` expands ``net_signed_at.sql = "customers.signed_up_at"`` so the ranked subquery's ORDER BY emits ``customers.signed_up_at`` — but ``_collect_joined_paths_for_base`` only walked ``ColumnKey`` args, so the ``customers`` join was not added to ``_base``. Extend the helper to accept ``source_model`` / ``source_relation`` / ``bundle`` and walk ``ColumnSqlKey`` args via the existing ``_expand_derived_column_sql`` + ``_joined_paths_in_sql`` pair (same machinery the ROW-derived TimeTruncKey path already uses). + regression test ``TestDev1501HiddenFirstLastRender.test_derived_time_arg_pulls_in_referenced_join``. 2. Sonar S3776 NOSONAR on ``_projection_rn_by_alias`` test helper (complexity 21 > 15) — sequential isinstance dispatch over outermost-SELECT projection layers (Alias / Cast / Max / Case / EQ / Column unwrap). Each layer is a structural-shape predicate whose failure short-circuits to the next projection; extracting per-layer helpers would scatter the predicate. 3. Sonar duplication gate (4.0% > 3.0% threshold, test_sql_generator.py 5.1% per-file). Extract ``_persist_and_engine(*models)`` async context-manager helper plus a ``_orders_with_paid_amount_model`` fixture; apply across the DEV-1501 test classes (TestDev1501HiddenFirstLastRender, TestDev1501BroadTriggerAndGuards, TestParameterizedAggCanonicalDistinct, TestTransformAmbiguousTimeDimension). One-line per call site (vs the prior 5-line ``tempfile + YAMLStorage + save_datasource + save_model + SlayerQueryEngine`` block) — drops ~140 net lines of new-line duplication. 3939 unit tests pass; ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-mangles-transform-kwargs-in-filters-ntilex DEV-1492: preserve transform kwargs in filter normalization
Codex MAJOR (slayer/sql/generator.py:6031+): the cross-model HAVING ``AggregateKey`` reconstruction in ``_render_filter_value_key_in_target_scope`` rerooted ``source`` to ``path=()`` but copied ``args`` / ``kwargs`` unchanged. The projection path (lines ~5685-5730) already handles all three symmetrically via ``_reroot_kwarg``. Mirror that logic here so a multi-hop routed filter (``customers.regions.amount:last(customers.regions.opened_at) > 0``) qualifies its time arg under the local CTE relation instead of the host-rooted ``__``-path alias that doesn't exist inside the target CTE. Inert for single-hop cases — both pre-fix and post-fix render ``customers.signed_up_at`` because ``_resolve_explicit_time_col``'s ``"__".join(path)`` happens to collapse to the same string when the target relation has the same single hop. Real fix lands for multi-hop shapes (``customers.regions.opened_at``) that the existing test suite doesn't yet exercise — kept the fix mechanical (same shape as the projection-path reroot) rather than introducing a multi-model test fixture that exercises an unrelated routing concern. 3939 unit tests pass; ruff clean. Sonar gate is fully green (duplication 0.9% < 3% threshold, 0 OPEN issues, 0 hotspots, no CI failures) after round 8's ``_persist_and_engine`` refactor. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…rst-with-different-explicit-time-columns DEV-1501: parametric first/last in ORDER BY / HAVING — materialise + outer trim
…apper A host aggregate whose Column.filter references a joined table (loss_payment_amt:sum where loss_payment_amt has filter="loss_payment.has_flag = 1") now hoists into its own _cm_ CTE host-rooted variant of the cross-model aggregate strategy. Without isolation, two such measures whose filter joins are different INNER joins on different tables would intersect in the host base to only the rows present in BOTH targets, silently corrupting both aggregates. Planner: trigger predicate extended to fire on column_filter_key.referenced_join_paths non-empty (not just source.path non-empty). _plan_filtered_local builds a host-rooted nested PlannedQuery and attaches via rerooted_plan / rerooted_grain_pairs / rerooted_agg_slot_id with cte_root_model = host.name. AGGREGATE-phase host filters referencing the isolated aggregate are skipped (not passed to the sub-plan) so the generator routes them to outer WHERE instead of HAVING-into-CTE (which would surface host rows as NULL via LEFT JOIN instead of dropping them). Generator: _render_with_cross_model_plans now identifies AGGREGATE filters walking an isolated AggregateKey, adds their ids to routed_ids so _base skips them, and renders each via _render_filter_for_outer_ wrapper as plain WHERE on the combined non-aggregating SELECT, substituting isolated AggregateKey refs with <_cm_>."<col>" and other slot refs with _base."<alias>". Hidden operand promotion for mixed filters (loss_payment_amt:sum > 1000 AND total_amount:sum > 10) flows through the existing _add_local_aux_slots(aggregates_only=True) pass. Identity: SqlExprKey gains referenced_join_paths so the planner's filtered-local trigger reads a typed field rather than re-parsing the column filter SQL. compute_column_filter_join_paths (new slayer/engine/column_filter_paths.py) is the planner-side discovery helper, swallowing internal sqlglot scope-analyser failures so the generator's parse-time security gate stays authoritative on malicious payloads. Tests: TestIsolatedFilteredMeasureCTEs xfails rewritten to assert _cm_/_base/outer-WHERE structure; new tests/test_filtered_local_ isolation.py covers SqlExprKey.referenced_join_paths, trigger dispatch (incl. cross-model-with-target-filter regression), recursion suppression, host model filter interactions, first/last sub-plan cleanliness. Docs: Strategy 3 section in docs/architecture/cross-model-aggregates.md; CLAUDE.md key-conventions bullet. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ardening
Correctness:
- Planner: fix date_range off-by-one in user-filter routing for the
filtered-local sub-plan. host_filter_routings is ordered
[date_range_routings..., user_filter_routings...]; slice the user
portion as host_filters[-len(host_query.filters):] instead of looking
up by f"f{i}" (off by n_date_range when a date_range time_dimension is
present; Codex review).
- Generator: outer-WHERE wrapper now maps EVERY cross-model aggregate
plan to its _cm_ CTE column (not just filtered-local with
cte_root_model set). A mixed predicate like loss_payment_amt:sum +
customers.revenue:sum > 100 routed to the wrapper via a filtered-local
operand can now resolve the forward cross-model operand at the outer
scope instead of raising NotImplementedError (CodeRabbit thread 2).
Test boundary assertions:
- POST-routing tests now assert layer boundaries: AGGREGATE predicate
inside the combined base CTE WHERE, POST predicate inside the
post-transform _filtered wrap, neither leaks into the other layer
(CodeRabbit thread 4).
- TestIsolatedFilteredMeasureCTEs + the customers-JOIN-inside-_cm_ test
switched from sql.index("\n)", start) manual slicing to the
balanced-paren _extract_cte_body helper so nested ranked subqueries
don't truncate the CTE body (CodeRabbit thread 3).
Hardening:
- SqlExprKey.referenced_join_paths gains a before-validator that
canonicalises to a sorted, de-duplicated tuple of tuples — identity is
now order-independent, callers can pass any iterable shape
(CodeRabbit nitpick).
- column_filter_paths: self-qualified anchor-local derived refs
(filter="orders.is_eu = 1" where orders is the anchor) now trigger
the derived-expansion gate identically to the bare form
(filter="is_eu = 1") so the cross-model path the expansion surfaces
is recorded on the SqlExprKey (CodeRabbit thread 1).
Refactors:
- Shared root-scope joined-path walker hoisted into column_expansion.py
as collect_root_scope_joined_paths; column_filter_paths and
generator both call it instead of duplicating the alias-walk-over-joins
loop (Sonar duplication clearance).
- compute_column_filter_join_paths splits its derived-expansion
preamble into _expand_filter_sql_if_anchor_derived (S3776 cognitive
complexity).
- IsolatedCteCrossModelPlanner.plan extracts the filtered-local trigger
precondition + dispatch into _dispatch_filtered_local (S3776).
- _plan_filtered_local extracts the host-query-filter classification
loop into the module-level _classify_subplan_filters helper (S3776).
- Test fixture: orders+customers+is_eu+eu_amount setup extracted to a
shared _orders_with_derived_eu_filter helper across three derived-ref
tests (Sonar duplication clearance).
- stage_planner.plan_query: NOSONAR(S3776) with reason — function's
pre-existing complexity is owned by multi-stage scope/bundle/projection/
filter-routing wiring tracked as a separate refactor.
Tests: full unit suite 4036 passed (was 4030). Ruff clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Correctness: - Composite projections over isolated filtered-local aggregates now render at the combined SELECT scope, not inline in _base. A measure like loss_payment_amt:sum + loss_reserve_amt:sum would otherwise pull BOTH filter-target INNER joins back into _base and intersect to rows in BOTH targets — silently corrupting both aggregates, the exact bug DEV-1503 set out to eliminate (Codex round 2 #1). Identification walks aggregate_slots + combined_expression_slots for ArithmeticKey / ScalarCallKey projections whose tree references any cross-model agg slot; rendering reuses _render_filter_for_outer_wrapper to substitute isolated AggregateKey → _cm_X."col" and other operands → _base."alias". Non-isolated agg operands are auto-promoted to hidden aux slots so _base materialises them. - No-dim aggregate-only isolated queries no longer emit SELECT 1 AS _placeholder FROM <host> (N rows) cross-joined with a scalar _cm_* CTE (1 row) — N × 1 duplicated the aggregate by host row count. The placeholder now drops the host FROM, yielding the expected single-row scalar (Codex round 2 #2). Tests: - test_formula_over_isolated_measures strengthened: asserts _base body contains NEITHER Loss_Payment NOR Loss_Reserve joins, and the composite alias appears in the outer combined SELECT referencing both _cm_* CTE columns. - test_base_not_empty_when_no_dims_all_measures_skipped strengthened: asserts the _base body does not reference the host table and carries the _placeholder literal. - test_cross_model_dimension_count_distinct_in_formula and test_rerooted_cross_model_in_formula promoted from xfail(strict=True) to PASS — the DEV-1499 cross-model-in-composite case is the same shape the DEV-1503 fix now handles. Sonar: - collect_root_scope_joined_paths (column_expansion.py) extracts the per-column alias-walk into _resolve_alias_to_join_segments (S3776 cognitive complexity). - _plan_filtered_local (cross_model_planner.py) extracts grain-pair matching, sub_agg slot finding, and cte_schema build into module-level helpers (_match_filtered_local_grain_pairs, _find_filtered_local_sub_agg_slot, _build_filtered_local_cte_schema) (S3776). - test_multi_hop_derived_filter_expands_inside_cm_cte converted from _re.search + manual sql.index("\n)") to _extract_cte_body — removes the remaining python:S5852 ReDoS hotspot. Tests: full unit suite 4038 passed (was 4036). Ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Generator (slayer/sql/generator.py:_render_with_cross_model_plans): - outer_composite_slot_ids scan now also walks slots referenced by planned_query.order, not just planned_query.projection. A hidden ORDER BY composite over isolated filtered-local aggregates was previously falling through to the order-only local path and rendering inline in _base, re-pulling filter-target joins into the host spine (Codex round 3 #1). - Outer-routed composite slots projected multiple times (C13 multi-alias same-key consolidation) cycle through cslot.public_aliases per emission and APPEND into combined_aliases_by_slot_id, rather than always emitting public_aliases[0] and overwriting the dict slot (CodeRabbit thread on alias cycling). - Order-only outer composites materialise as hidden combined-SELECT columns under the slot's declared_name so ORDER BY can resolve via the synthesised alias. _build_combined_order_by_sql: - New ``outer_composite_aliases`` kwarg maps each outer-routed composite slot id to the alias the combined SELECT projects it under. ORDER BY references those aliases bare (the projection belongs to the combined SELECT, not _base) — previously they would emit ``_base."<alias>"`` against a column that doesn't exist there (Codex round 3 #2). Tests: - test_order_by_projected_composite_over_isolated_resolves_at_combined pins the ORDER-BY-alias-at-combined-scope invariant: the composite must not leak into _base AND the ORDER BY must not reference _base. - test_outer_composite_with_multiple_user_aliases pins C13 alias cycling: the same composite formula declared under two user names surfaces both aliases in the outer SELECT. Tests: 4040 passed (was 4038). Ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Correctness (Codex round 4 / CodeRabbit): - empty_base placeholder now re-introduces the host FROM + WHERE + LIMIT 1 when ANY host-local ROW filter exists (filter id not in routed_ids and phase=ROW). The round-2 fix that dropped the host FROM to avoid N-row CROSS JOIN duplication also bypassed _build_where_having_from_planned, silently ignoring host-local ROW filters on no-dim queries with forward cross-model aggregates (e.g. customers.revenue:sum with filters=["status = 'active'"]). LIMIT 1 preserves the 1-row cardinality fix from round 2; WHERE applies the filter so the combined query returns 0 rows when no host row matches (correct semantics) and 1 row otherwise. When NO host-local filter exists, the placeholder stays bare (no FROM, no WHERE, no LIMIT) — matching round 2 for the unfiltered fast path. Sonar: - _build_combined_order_by_sql (generator.py:6528) extracts per-entry alias resolution into _resolve_combined_order_term — S3776 cognitive complexity 20 → 15. Tests: - test_no_dim_query_with_host_row_filter_applies_in_base pins the WHERE + LIMIT 1 invariant in the empty_base placeholder for a no-dim query with a host ROW filter. Tests: 4041 passed (was 4040). Ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Correctness (Codex round 5): - _build_first_last_base_select now accepts skip_filter_ids and forwards it to _build_where_having_from_planned. The cross-model path passes routed_ids to _build_base_select_for_planned, but the first/last branch was calling the inner helper without forwarding the set — a filter routed to a per-plan _cm_ CTE was double-applied: once inside the host ranked subquery's WHERE, once inside the _cm_ CTE. For a filter referencing a join path not in _base's FROM, the host-side rendering would emit invalid SQL. Test: - test_local_first_last_with_routed_cross_model_filter pins the routing: a query mixing total_amount:last (local) + loss_payment.has_flag:sum (forward cross-model) + a filter routed via PROPAGATE_HAVING asserts the filter literal appears only inside the _cm_ CTE body, never in the host _base ranked subquery. Tests: 4042 passed (was 4041). Ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Correctness (Codex round 6):
- The round-4 empty_base placeholder fix added FROM <host> + WHERE +
LIMIT 1 when a host-local ROW filter exists, but bypassed
_collect_filter_join_paths / _build_from_and_joins. A filter
referencing a joined alias (e.g. filters=["claim.claim_number =
'12345'"]) therefore emitted WHERE claim.claim_number = ... against a
_base that had no Claim join — invalid SQL ("missing FROM-clause
entry"). The empty_base branch now walks non-routed filters via
_collect_filter_join_paths, threads the resulting paths through
_build_from_and_joins, and attaches the LEFT JOINs to the placeholder
before the WHERE.
Test:
- test_no_dim_host_filter_referencing_joined_column_pulls_join pins
the join-discovery: a no-dim query with a host ROW filter on a
joined column asserts the Claim join appears in _base alongside the
filter literal AND LIMIT 1.
Tests: 4043 passed (was 4042). Ruff clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Correctness (Codex round 7):
- column_filter_paths.compute_column_filter_join_paths parsed every
Column.filter SQL with hard-coded dialect="postgres" and returned ()
on parse failure. For a non-Postgres datasource using dialect-specific
syntax (MySQL backticks, T-SQL square brackets, ClickHouse-specific
functions), the Postgres parse could fail silently — the planner then
saw no referenced join paths, the DEV-1503 trigger missed, and the
generator (dialect-aware) rendered the filter inline in _base, pulling
the cross-model join back into the host rowset and silently corrupting
the aggregate. The helper now walks a small dialect fallback chain
(Postgres → sqlglot default → MySQL → BigQuery → T-SQL) and only
returns () when every dialect rejects the input. The injection security
gate (UNION SELECT, etc.) is unchanged because malformed payloads still
fail every dialect.
Test:
- test_dialect_fallback_chain_recovers_mysql_backtick_filter pins the
recovery: a backtick-quoted joined alias ref ("loss_payment"."has_flag"
= 1) surfaces ("loss_payment",) in referenced_join_paths.
Tests: 4044 passed (was 4043). Ruff clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Correctness: - _render_with_cross_model_plans now honours where_consumed from the first/last base-rendering path: when the ranked subquery consumed WHERE, we no longer re-apply base_where on the outer _base SELECT. Double-application changed first/last semantics (the outer WHERE filtered AFTER ranking) and could dangle joined-column aliases on the outer scope (CodeRabbit thread). - Order-only outer composites (composite slots referenced by ORDER BY but absent from planned_query.projection) now render INLINE inside the combined ORDER BY rather than materialising as a hidden combined-SELECT column. The old shape leaked the synthetic alias as an extra user-visible result column on the no-transform path (Codex round 8) AND lost the slot in the cross-model transform chain's combined_aliases_by_slot_id carry-forward dict, breaking ORDER BY resolution on the chain output (CodeRabbit thread). A new outer_composite_order_expressions map carries the rendered SQL for each order-only outer composite; _build_combined_order_by_sql emits <expr> <direction> bare for those slots. Test: - test_first_last_with_host_filter_not_reapplied_outside_ranked_subquery pins the where_consumed gate: a query mixing total_amount:last (local first/last) + loss_payment.has_flag:sum (forward cross-model) + filters=["status = 'active'"] asserts the filter literal appears exactly once in _base — inside the ranked subquery. Tests: 4045 passed (was 4044). Ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Correctness (Codex round 9): - The round-7 dialect fallback chain covered the outer sqlglot.parse_one call and the re-parse after derived expansion, but expand_derived_refs_sync itself was still pinned to Postgres. A derived column whose Column.sql uses dialect-specific syntax (MySQL backticks, BigQuery struct literals, etc.) would fail the inner parse silently, dropping the join path the DEV-1503 trigger needs. A new _expand_derived_refs_any_dialect helper walks the same chain (excluding the None sentinel since expand_derived_refs_sync requires a dialect string) and returns the first non-None expansion. Test: - test_dialect_fallback_chain_recovers_derived_ref_with_backticks pins the recovery: a derived column whose sql uses ``CASE WHEN `customers`.`region` = 'EU'`` (MySQL backticks) referenced by a Column.filter still surfaces the (customers,) join path. Without the wrapper the helper falls back to Postgres only and returns (). Tests: 4046 passed (was 4045). Ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…cks-isolated-cte-handling-for-cross-model DEV-1503: filtered-local isolation on typed pipeline + outer-WHERE wrapper
In the typed pipeline, an AGGREGATE slot whose source is a derived (ColumnSqlKey) column whose Column.sql contains a __-delimited join-path alias (e.g. customers__regions.population) never had the implied LEFT JOINs added to the host base FROM — the aggregate body emitted SUM(customers__regions.population) referencing an undefined table alias. The dimension / time-dimension cases got this right under DEV-1484; the Column.filter case got it right under DEV-1494; the agg-source case was the symmetric remaining gap. Fix: add _collect_aggregate_source_join_paths next to _collect_column_filter_join_paths. Walks AGGREGATE-phase slots, recursing through ArithmeticKey / ScalarCallKey composite keys. For each AggregateKey with a ColumnSqlKey source and source.path == (), expand the column's sql via _expand_derived_column_sql and scan through _joined_paths_in_sql, appending discovered paths to needed_join_paths. Wire into _build_base_select_for_planned next to the existing filter-side collector — single wire-up covers regular AGGREGATE and local first/last (the ranked-subquery builder inherits from_clause / base_joins from the same call). Cross-model agg sources (source.path \!= ()) are skipped; their _cm_* CTE owns its own discovery (gap tracked as DEV-1526; xfail in place). The render-time agg-body expansion in _build_agg_render_spec_from_planned already produced correct SUM(<expanded>) SQL — only the join-discovery side was missing. Tests: un-xfail the existing tracking test + strengthen its body assertion; new TestMeasureSourceSqlJoinInference class covers single-hop / multi-hop, ArithmeticKey + ScalarCallKey composites, Mode-A function wrappers around the path alias, sibling derived chains, local last with derived source (ranked-subquery shape asserted), cumsum-wrapped path-aliased measure (POST-phase aux materialization), multiple agg sources sharing one join (dedupe), dim+measure sharing one join (dedupe), filter+source crossing different joins (DEV-1494/1502 coexistence), no-path-no-extra-join sanity, defensive cross-model skip with positive _cm_ assertion. Two strict xfails pin the follow-up gaps (DEV-1526 cross-model CTE source-sql gap, DEV-1527 parametric agg derived-kwarg gap). One SQLite integration test seeds orders → customers → regions and verifies region_pop:sum returns 350.0 end-to-end. Docs: paragraph in sql-generation.md listing the three symmetric host-base discovery sources (DEV-1484 dim, DEV-1494 filter, DEV-1502 agg-source). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SQLite returns REAL aggregates as floats; strict == 350.0 trips python:S1244 and is the right call to relax (the gate is currently red on new_reliability_rating=3 because of this one finding). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CodeRabbit caught that `"regions" not in _join_aliases(sql)` does an exact-match on the alias set — but the realistic leak shape from a misbehaving host-side collector walking through customers would be the path alias `customers__regions`, which the bare-name check silently misses. Assert both alias shapes so the defensive test actually pins the regression it was meant to guard. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The new host-base discovery section claimed cross-model aggregates own their own join discovery in the _cm_* CTE without qualification. True for the Column.filter side (DEV-1494 / DEV-1503), but the symmetric source-Column.sql case is the open DEV-1526 gap. Codex flagged the unqualified statement as misleading future readers. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Codex MAJOR caught that the test asserts dry-run SQL substrings but the emitted SQL fails at RUNTIME — the inner ranked subquery has the LEFT JOINs (DEV-1502 pulled them in), but the outer SELECT's MAX(CASE WHEN _last_rn = 1 THEN <expr> END) still references the cross-table- qualified expression directly, which is out of scope outside the subquery (only the source_relation alias is). Confirmed via SQLite execution: `no such column: customers__regions. payment_amount`. Pre-existing bug for ANY derived first/last source crossing a join (dotted form too, DEV-1410 territory); DEV-1502 just unmasked it for the __ alias case by getting the joins emitted at all. Pre-DEV-1502 this failed at "no JOINs"; post-DEV-1502 it fails at "outer scope missing". Convert the test to a strict xfail tracking DEV-1531; the assertion now pins the END STATE (the outer SELECT must not reference the cross-table ref directly) rather than just substring presence. Auto-promotes when DEV-1531's materialisation fix lands. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-path-alias-in-a-measures-columnsql-doesnt DEV-1502: pull joins crossed by an aggregate source's Column.sql
… into egor/dev-1484-dev-1452-stage-c-migrate-pre-existing-tests-off-legacy
…rivial cleanups + 2 NOSONAR Triaged Codex + CodeRabbit + Sonar against PR #153 and applied the valid findings. Stage C / dev-1450 catch-up merge ride-along. Group A — correctness bugs (5): - A1 (CR T4): `_reachable_aggs_for_save` now unions reachable custom-agg names across query-backed-model source_queries stages, and save_model also runs normalize_query on each stage so funcstyle aggs over joined- model custom aggs land in canonical form for query-backed models too. - A2 (CR T5): add `clickhouse` to `_PLANNER_PARSE_DIALECT_CHAIN` so ClickHouse-only Column.filter predicates parse and DEV-1503 join-path isolation kicks in on ClickHouse backends. - A3 (CR T6): stop slicing `host_query.filters` in `_classify_subplan_filters`; stage_planner now stamps the original user-filter text on each `HostFilterRouting`, cross_model_planner consumes `routing.text` directly. Eliminates the date_range + dedup off-by-one that mis-pairs phases against the deduped routings list. - A4 (Codex): pg_facade `$N` parameter substitution is now lexer-aware via a new `_iter_param_placeholders` walker. Skips `$N` inside string literals (incl. E'…' / standard '…'), double-quoted identifiers, `$tag$ … $tag$` dollar-quotes, and line/block comments — closes the regex-only path that would corrupt SQL like `WHERE note = '$1' AND status = $1`. - A5 (CR T7): DEV-1501 tie-break integration test row data flipped so the expected ORDER BY result contradicts natural alphabetical status ordering — a silently-dropped secondary key now fails the test. Group B — keyword-only helper signatures (CR style rule): - _expand_stage_as_model (tests/test_query_backed_models.py) - _engine_generate (tests/_engine_helpers.py + 20 call sites swept) - tests/test_aggregation_gating.py local _generate_sql delegated to the shared _engine_generate helper. Group C — trivial fixes: - _slot keyword arg in test_agg_render_spec.py - "legacy legacy" docstring typo in test_response_meta.py - Decimal(0) pin in test_transform_lowerer.py - module-top `Aggregation` import in test_validate_models.py - pytest.mark.parametrize keyword args in test_errors_hierarchy.py (x2) Group D — Sonar S3776 NOSONAR with rationale: - syntax.py:_convert (cognitive complexity 38) — flat dispatch over ast node kinds; splitting hides exhaustive coverage. - generator.py:_render_post_phase_filter_conditions (35) — one cohesive POST-phase filter walk; splitting hides shared registry/alias-map state that both wrap-CTE and outer-WHERE emission depend on. Stale outside-diff CR comments not actioned (already addressed by the landed fix PRs): agg-source ColumnSqlKey gap (closed by DEV-1502 PR #165's `_collect_aggregate_source_join_paths`); _cm_* CTE measure-source gap (tracked as DEV-1526 strict-xfail); is_root propagation (explicit `is_root=False` / `is_root=not key.path` already present at every joined-ColumnSqlKey expansion site). Also tightened tests/test_sql_generator.py:: test_date_trunc_casts_unknown_typed_time_dim — the broad `'CAST(' not in sql` assertion collided with the DEV-1361 `CAST(COUNT(*) AS INT)` count-cast carried by Stage C. Asserts `CAST(ORDERS.CREATED_AT...` / `CAST(CREATED_AT...` are absent, preserving the test's actual intent (the time-dim column itself isn't wrapped) without rejecting unrelated CAST emission. Suite: 4373 passed, 6 skipped, 21 xfailed, 0 failed. Ruff: clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ed PRs Round-2 review pass on PR #153. Codex finding (`_iter_param_placeholders` skips unquoted identifiers): the lexer-aware $N walker landed in the previous commit treated identifiers as a stream of normal characters, so `metric$1` would be read as `metric` + placeholder `$1`. Postgres unquoted-identifier syntax allows letter/_ followed by alnum/_/$, so the whole token must be one unit. Added an explicit identifier-skip branch; tightened the walker into a flat `c → _skip_*` dispatch over per-token-kind helpers (_skip_line_comment / _skip_block_comment / _skip_unquoted_identifier / _skip_e_string / _skip_single_quoted_string / _skip_double_quoted_identifier / _try_skip_dollar_quoted). Each helper takes/returns `i`; the main function reduces from cognitive complexity 101 to a 7-branch dispatch. Sonar S3776 (3 new issues on my last-commit code): - `save_model` (complexity 16): extracted module-level `_normalize_source_query_stages(model, *, custom_aggs)` — per-stage `normalize_query` loop for query-backed models. Brings save_model back into S3776's allowance and names the "why does save_model walk source_queries?" question explicitly. - `_reachable_aggs_for_save` (complexity 30): split into one main reading-as-a-recipe + three single-job helpers: `_make_join_target_resolver` (the swallow-all closure), `_walk_reachable_aggs` (one BFS over a model's joins, staticmethod), `_resolve_stage_source_model` (inline/string/skip dispatch). CI workflow: - `pull_request.branches: ['**']` so the lint + test job runs for every PR regardless of base branch. The stacked-PR style used here (PR #153 → dev-1450; future PRs → dev-1450 once #153 lands) previously fell out of the `[main]` filter entirely, so no automated test gate ran — only CodeRabbit + Sonar surfaced. Suite: 4373 passed, 6 skipped, 21 xfailed, 0 failed. Ruff: clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI failures (`lint-and-test (3.11)`): A1: `tests/integration/test_mcp_inspect.py::TestCollectDimProfile:: test_numeric_and_temporal_min_max` — failed because the profile query emitted `CAST(MIN(ordered_at) AS TIMESTAMP)` and SQLite has no TIMESTAMP type, so NUMERIC affinity coerced `'2025-01-15'` to int `2025`. Bug was not from this PR — came in via the local-dev-1450 catch-up merge, but the CI gap let it slip through. Fix: drop `type` from the profiling ext_columns. DEV-1361's CAST wrap is harmful for a min/max probe (we want the raw stored shape, not a re-cast). Backend-agnostic and lands clean on SQLite, Postgres, DuckDB, ClickHouse, MySQL. A2: `tests/integration/test_notebooks.py::test_notebook_runs_without_errors [09_lightning_talk/lightning_talk_nb.ipynb]` — `NotImplementedError: DEV-1450 stage 7b.12 cross-model partition`. Same DEV-1474 deferred path as the 04_time notebook (the hero query uses `change_pct` with `dimensions=['stores.name']`). Added the case to `_KNOWN_FAILING_NOTEBOOKS` so it xfails until DEV-1474 lands. Codex follow-up (B1): added `TestIterParamPlaceholders` with 15 focused tests covering every disambiguation branch — `metric$1` (identifier with `$`), `Etable$1` (identifier, not E-string), `E'$1'`, `'$1'`, `'a''b $1 c'` (doubled-quote escape), `"col $1"`, `$tag$ … $1 … $tag$`, `$$ … $1 … $$` (anonymous), `-- $1\n`, `/* $1 */`, nested block comments, `$1::int`, and the Codex repro `WHERE note = '$1' AND status = $1` (only one placeholder substituted). Plus an end-to-end `_substitute_params` test pinning the literal-$1-stays-literal contract. Sonar S3776 (B2): `_iter_param_placeholders` now under threshold — extracted `_handle_dollar(sql, i) -> (new_i, optional_placeholder)` for the dollar-quote-vs-placeholder branch so the main dispatch is a flat ``c → helper`` loop of 8 elifs, with the only generator-yield remaining at the top level. Suite: 4388 passed, 6 skipped, 21 xfailed, 0 failed (non-integration); 41 passed, 2 xfailed (integration smoke incl. the previously-failing two). Ruff: clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…stead of dismissing them Per the user's standing rule (feedback_fix_genuine_issues_in_scope — real, fixable issues get fixed regardless of "pre-existing" status), refactoring the 4 cognitive-complexity issues I'd previously left alone as "not from this PR": slayer/sql/generator.py:_render_value_key_against_aliases (35 → ~10): extracted module-level `_render_scalar_literal(v)` (None / bool / int / float / Decimal / str dispatch) — reused by both the LiteralKey branch and the ScalarCallKey arg-mixing branch where it previously duplicated the same 5-way isinstance ladder. Added a local `recurse(k)` closure so BetweenKey / InKey / ArithmeticKey / ScalarCallKey stop repeating the three-kwarg recursion. Behaviour unchanged; rendered SQL identical. tests/test_lightning_talk_notebook.py:_find_kwargs_for_call, test_teardown_cell_forgets_both_stable_ids, and _collect_forget_memory_call_args (24/42/17 → small): all three were AST-walking `forget_memory(...)` / arbitrary-callable calls with the same boilerplate (callee-name extraction, literal-or-unparse fallback, Call-node iteration). Extracted three module-level helpers: - `_call_name(node)` — flat dispatch over Attribute/Name/other - `_literal_or_unparse(value)` — `ast.literal_eval` with `ast.unparse` fallback - `_iter_calls_named(source, callee)` — generator yielding every matching ast.Call The three previously over-complex functions now read as the assertion they were always trying to express. Suite: 4388 passed, 6 skipped, 21 xfailed, 0 failed. Ruff: clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…migrate-pre-existing-tests-off-legacy DEV-1484 Stage C: migrate pre-existing tests off legacy enrichment
|



Summary
Work-in-progress on DEV-1450 — replacing today's tangled multi-path resolution with a typed pipeline of composable stages:
raw → slack normalize → parse_expr → bind_expr → ProjectionPlanner → plan_query → PlannedQuery.main, broken into ~25 numbered stages.See the DEV-1450 progress checkpoints for stage-by-stage detail.
Test plan
orders.revenue_sum,orders._count,orders.customers.regions.name, renamedorders.<user_name>).🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Tests