Skip to content

Commit 7bfba06

Browse files
committed
Add cross-dataset referential integrity checks
validate_rows is intra-row and single-table — its unique rule only dedupes within one batch. Add dbt-style generic checks: check_foreign_key (parent/child across two tables), check_unique_key (single/composite), check_accepted_values, and check_row_count, operating on rows already loaded by load_rows/query_sqlite. Wired through facade, executor (four AC_check_* commands), MCP, and the Script Builder with a headless test batch and EN/Zh docs.
1 parent 09e9cb8 commit 7bfba06

15 files changed

Lines changed: 421 additions & 1 deletion

File tree

README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-22) — Referential Integrity Checks](#whats-new-2026-06-22--referential-integrity-checks)
1617
- [What's new (2026-06-22) — URI-Scheme Value References](#whats-new-2026-06-22--uri-scheme-value-references)
1718
- [What's new (2026-06-21) — W3C Baggage Propagation](#whats-new-2026-06-21--w3c-baggage-propagation)
1819
- [What's new (2026-06-21) — Dataset Diff (Row-Set Change Report)](#whats-new-2026-06-21--dataset-diff-row-set-change-report)
@@ -138,6 +139,12 @@
138139

139140
---
140141

142+
## What's new (2026-06-22) — Referential Integrity Checks
143+
144+
Foreign-key, unique, accepted-values and row-count checks across tables. Full reference: [`docs/source/Eng/doc/new_features/v86_features_doc.rst`](docs/source/Eng/doc/new_features/v86_features_doc.rst).
145+
146+
- **`check_foreign_key` / `check_unique_key` / `check_accepted_values` / `check_row_count`** (`AC_check_foreign_key`, `AC_check_unique_key`, `AC_check_accepted_values`, `AC_check_row_count`): `validate_rows` is intra-row, single-table (its `unique` only dedupes within one batch). This adds dbt-style generic checks — parent/child foreign keys across two tables, single/composite key uniqueness, accepted-values, and row-count bounds — over rows from `load_rows`/`query_sqlite`. Pure-stdlib, deterministic.
147+
141148
## What's new (2026-06-22) — URI-Scheme Value References
142149

143150
Store pointers, not secrets, in config. Full reference: [`docs/source/Eng/doc/new_features/v85_features_doc.rst`](docs/source/Eng/doc/new_features/v85_features_doc.rst).

README/README_zh-CN.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-22) — 参照完整性检查](#本次更新-2026-06-22--参照完整性检查)
1516
- [本次更新 (2026-06-22) — URI-Scheme 值引用](#本次更新-2026-06-22--uri-scheme-值引用)
1617
- [本次更新 (2026-06-21) — W3C Baggage 传播](#本次更新-2026-06-21--w3c-baggage-传播)
1718
- [本次更新 (2026-06-21) — 数据集差异(数据行变更报告)](#本次更新-2026-06-21--数据集差异数据行变更报告)
@@ -137,6 +138,12 @@
137138

138139
---
139140

141+
## 本次更新 (2026-06-22) — 参照完整性检查
142+
143+
跨数据表的外键、唯一键、accepted-values 与行数检查。完整参考:[`docs/source/Zh/doc/new_features/v86_features_doc.rst`](../docs/source/Zh/doc/new_features/v86_features_doc.rst)
144+
145+
- **`check_foreign_key` / `check_unique_key` / `check_accepted_values` / `check_row_count`**(`AC_check_foreign_key``AC_check_unique_key``AC_check_accepted_values``AC_check_row_count`):`validate_rows` 是单行、单表(其 `unique` 只在单批次内去重)。本功能补上 dbt 风格通用检查 —— 跨两表的父子外键、单一/复合键唯一性、accepted-values、行数范围 —— 作用于 `load_rows`/`query_sqlite` 的数据行。纯标准库、确定。
146+
140147
## 本次更新 (2026-06-22) — URI-Scheme 值引用
141148

142149
在配置中存储指针而非机密。完整参考:[`docs/source/Zh/doc/new_features/v85_features_doc.rst`](../docs/source/Zh/doc/new_features/v85_features_doc.rst)

README/README_zh-TW.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-22) — 參照完整性檢查](#本次更新-2026-06-22--參照完整性檢查)
1516
- [本次更新 (2026-06-22) — URI-Scheme 值參照](#本次更新-2026-06-22--uri-scheme-值參照)
1617
- [本次更新 (2026-06-21) — W3C Baggage 傳播](#本次更新-2026-06-21--w3c-baggage-傳播)
1718
- [本次更新 (2026-06-21) — 資料集差異(資料列變更報告)](#本次更新-2026-06-21--資料集差異資料列變更報告)
@@ -137,6 +138,12 @@
137138

138139
---
139140

141+
## 本次更新 (2026-06-22) — 參照完整性檢查
142+
143+
跨資料表的外鍵、唯一鍵、accepted-values 與筆數檢查。完整參考:[`docs/source/Zh/doc/new_features/v86_features_doc.rst`](../docs/source/Zh/doc/new_features/v86_features_doc.rst)
144+
145+
- **`check_foreign_key` / `check_unique_key` / `check_accepted_values` / `check_row_count`**(`AC_check_foreign_key``AC_check_unique_key``AC_check_accepted_values``AC_check_row_count`):`validate_rows` 是單列、單表(其 `unique` 只在單批次內去重)。本功能補上 dbt 風格通用檢查 —— 跨兩表的父子外鍵、單一/複合鍵唯一性、accepted-values、筆數範圍 —— 作用於 `load_rows`/`query_sqlite` 的資料列。純標準函式庫、具決定性。
146+
140147
## 本次更新 (2026-06-22) — URI-Scheme 值參照
141148

142149
在設定中儲存指標而非機密。完整參考:[`docs/source/Zh/doc/new_features/v85_features_doc.rst`](../docs/source/Zh/doc/new_features/v85_features_doc.rst)
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
Referential Integrity Checks
2+
============================
3+
4+
``data_quality.validate_rows`` is strictly intra-row, single-table — its
5+
``unique`` rule only dedupes within one batch. There was no parent/child
6+
foreign-key check across two loaded tables, no composite-key uniqueness, and no
7+
standalone accepted-values / row-count assertion. This adds those dbt-style
8+
generic checks over rows already loaded by ``load_rows`` / ``query_sqlite``.
9+
10+
Pure standard library (``collections``); imports no ``PySide6``. Every function
11+
is pure (rows in, report out), so it is fully deterministic in CI.
12+
13+
Headless API
14+
------------
15+
16+
.. code-block:: python
17+
18+
from je_auto_control import (
19+
check_foreign_key, check_unique_key, check_accepted_values,
20+
check_row_count, load_rows,
21+
)
22+
23+
orders = load_rows("orders.csv")
24+
users = load_rows("users.csv")
25+
26+
fk = check_foreign_key(orders, "user_id", users, "id") # {ok, violations, missing}
27+
pk = check_unique_key(orders, ["region", "id"]) # {ok, duplicates}
28+
av = check_accepted_values(orders, "status", ["open", "closed"])
29+
rc = check_row_count(orders, minimum=1) # {ok, count}
30+
31+
``check_foreign_key`` flags non-null child values absent from the parent column
32+
(dbt ``relationships``). ``check_unique_key`` reports duplicate single or
33+
composite keys. ``check_accepted_values`` lists non-null values outside the
34+
allowed set. ``check_row_count`` verifies the count falls within optional
35+
``minimum`` / ``maximum`` bounds. Each returns an ``ok`` flag plus details.
36+
37+
Executor commands
38+
-----------------
39+
40+
``AC_check_foreign_key``, ``AC_check_unique_key``, ``AC_check_accepted_values``,
41+
and ``AC_check_row_count`` each take JSON ``rows`` (and a column / key / allowed
42+
list) and return the report. All are exposed as MCP tools
43+
(``ac_check_*``) and as Script Builder commands under **Data**.

docs/source/Eng/eng_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,7 @@ Comprehensive guides for all AutoControl features.
108108
doc/new_features/v83_features_doc
109109
doc/new_features/v84_features_doc
110110
doc/new_features/v85_features_doc
111+
doc/new_features/v86_features_doc
111112
doc/ocr_backends/ocr_backends_doc
112113
doc/observability/observability_doc
113114
doc/operations_layer/operations_layer_doc
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
參照完整性檢查
2+
============
3+
4+
``data_quality.validate_rows`` 嚴格限於單列、單表 —— 它的 ``unique`` 規則只在單一批次內去重。沒有跨兩個
5+
已載入資料表的父子外鍵檢查、沒有複合鍵唯一性,也沒有獨立的 accepted-values / row-count 斷言。本功能在
6+
已由 ``load_rows`` / ``query_sqlite`` 載入的資料列上,補上這些 dbt 風格的通用檢查。
7+
8+
純標準函式庫(``collections``);不匯入 ``PySide6``。每個函式皆為純函式(輸入列、輸出報告),因此在 CI
9+
中完全具決定性。
10+
11+
無頭 API
12+
--------
13+
14+
.. code-block:: python
15+
16+
from je_auto_control import (
17+
check_foreign_key, check_unique_key, check_accepted_values,
18+
check_row_count, load_rows,
19+
)
20+
21+
orders = load_rows("orders.csv")
22+
users = load_rows("users.csv")
23+
24+
fk = check_foreign_key(orders, "user_id", users, "id") # {ok, violations, missing}
25+
pk = check_unique_key(orders, ["region", "id"]) # {ok, duplicates}
26+
av = check_accepted_values(orders, "status", ["open", "closed"])
27+
rc = check_row_count(orders, minimum=1) # {ok, count}
28+
29+
``check_foreign_key`` 標記父欄位中不存在的非空子值(dbt ``relationships``)。``check_unique_key`` 回報
30+
重複的單一或複合鍵。``check_accepted_values`` 列出允許集合之外的非空值。``check_row_count`` 驗證筆數
31+
落在選用的 ``minimum`` / ``maximum`` 範圍內。每個皆回傳 ``ok`` 旗標加上細節。
32+
33+
執行器命令
34+
----------
35+
36+
``AC_check_foreign_key``、``AC_check_unique_key``、``AC_check_accepted_values`` 與 ``AC_check_row_count``
37+
各自接受 JSON ``rows``(以及欄位 / 鍵 / 允許清單)並回傳報告。全部皆以 MCP 工具(``ac_check_*``)以及
38+
Script Builder 中 **Data** 分類下的命令提供。

docs/source/Zh/zh_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,7 @@ AutoControl 所有功能的完整使用指南。
108108
doc/new_features/v83_features_doc
109109
doc/new_features/v84_features_doc
110110
doc/new_features/v85_features_doc
111+
doc/new_features/v86_features_doc
111112
doc/ocr_backends/ocr_backends_doc
112113
doc/observability/observability_doc
113114
doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,10 @@
167167
from je_auto_control.utils.dataset_diff import (
168168
cell_changes, diff_rows, summarize_diff,
169169
)
170+
# Cross-dataset referential integrity (foreign-key / unique / accepted values)
171+
from je_auto_control.utils.referential import (
172+
check_accepted_values, check_foreign_key, check_row_count, check_unique_key,
173+
)
170174
# i18n / l10n testing: pseudo-localize, overflow + catalog checks
171175
from je_auto_control.utils.i18n_test import (
172176
check_catalog, check_overflow, pseudo_localize, pseudo_localize_catalog,
@@ -841,6 +845,8 @@ def start_autocontrol_gui(*args, **kwargs):
841845
"infer_schema", "profile_rows",
842846
"categorical_drift", "detect_drift", "ks_two_sample", "psi",
843847
"cell_changes", "diff_rows", "summarize_diff",
848+
"check_accepted_values", "check_foreign_key", "check_row_count",
849+
"check_unique_key",
844850
"check_catalog", "check_overflow", "pseudo_localize",
845851
"pseudo_localize_catalog",
846852
"Checkpoint", "CheckpointStore", "run_resumable",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1775,6 +1775,48 @@ def _add_resilience_specs(specs: List[CommandSpec]) -> None:
17751775
),
17761776
description="Per-cell {key, column, old, new} changes between row-sets.",
17771777
))
1778+
specs.append(CommandSpec(
1779+
"AC_check_foreign_key", "Data", "Referential: Foreign Key",
1780+
fields=(
1781+
FieldSpec("child_rows", FieldType.STRING,
1782+
placeholder='[{"user_id": 1}]'),
1783+
FieldSpec("child_col", FieldType.STRING, placeholder="user_id"),
1784+
FieldSpec("parent_rows", FieldType.STRING,
1785+
placeholder='[{"id": 1}]'),
1786+
FieldSpec("parent_col", FieldType.STRING, placeholder="id"),
1787+
),
1788+
description="Every child value must exist in the parent column.",
1789+
))
1790+
specs.append(CommandSpec(
1791+
"AC_check_unique_key", "Data", "Referential: Unique Key",
1792+
fields=(
1793+
FieldSpec("rows", FieldType.STRING,
1794+
placeholder='[{"id": 1}, {"id": 1}]'),
1795+
FieldSpec("cols", FieldType.STRING,
1796+
placeholder='id (or ["region", "id"])'),
1797+
),
1798+
description="A single or composite key must be unique across rows.",
1799+
))
1800+
specs.append(CommandSpec(
1801+
"AC_check_accepted_values", "Data", "Referential: Accepted Values",
1802+
fields=(
1803+
FieldSpec("rows", FieldType.STRING,
1804+
placeholder='[{"status": "open"}]'),
1805+
FieldSpec("col", FieldType.STRING, placeholder="status"),
1806+
FieldSpec("allowed", FieldType.STRING,
1807+
placeholder='["open", "closed"]'),
1808+
),
1809+
description="Every non-null column value must be in the allowed set.",
1810+
))
1811+
specs.append(CommandSpec(
1812+
"AC_check_row_count", "Data", "Referential: Row Count",
1813+
fields=(
1814+
FieldSpec("rows", FieldType.STRING, placeholder='[{"id": 1}]'),
1815+
FieldSpec("minimum", FieldType.INT, optional=True),
1816+
FieldSpec("maximum", FieldType.INT, optional=True),
1817+
),
1818+
description="The row count must fall within the given bounds.",
1819+
))
17781820
specs.append(CommandSpec(
17791821
"AC_rate_limit", "Flow", "Rate Limit (Token Bucket)",
17801822
fields=(

je_auto_control/utils/executor/action_executor.py

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3129,6 +3129,43 @@ def _categorical_drift(reference: Any, current: Any) -> Dict[str, Any]:
31293129
return categorical_drift(reference, current)
31303130

31313131

3132+
def _json_rows(rows: Any) -> Any:
3133+
import json
3134+
return json.loads(rows) if isinstance(rows, str) else rows
3135+
3136+
3137+
def _check_foreign_key(child_rows: Any, child_col: str, parent_rows: Any,
3138+
parent_col: str) -> Dict[str, Any]:
3139+
"""Adapter: foreign-key referential check across two row-sets."""
3140+
from je_auto_control.utils.referential import check_foreign_key
3141+
return check_foreign_key(_json_rows(child_rows), child_col,
3142+
_json_rows(parent_rows), parent_col)
3143+
3144+
3145+
def _check_unique_key(rows: Any, cols: Any) -> Dict[str, Any]:
3146+
"""Adapter: single/composite key uniqueness check."""
3147+
import json
3148+
from je_auto_control.utils.referential import check_unique_key
3149+
if isinstance(cols, str) and cols.strip().startswith("["):
3150+
cols = json.loads(cols)
3151+
return check_unique_key(_json_rows(rows), cols)
3152+
3153+
3154+
def _check_accepted_values(rows: Any, col: str, allowed: Any) -> Dict[str, Any]:
3155+
"""Adapter: accepted-values check for a column."""
3156+
from je_auto_control.utils.referential import check_accepted_values
3157+
return check_accepted_values(_json_rows(rows), col, _json_rows(allowed))
3158+
3159+
3160+
def _check_row_count(rows: Any, minimum: Any = None,
3161+
maximum: Any = None) -> Dict[str, Any]:
3162+
"""Adapter: row-count bounds check."""
3163+
from je_auto_control.utils.referential import check_row_count
3164+
low = int(minimum) if minimum is not None else None
3165+
high = int(maximum) if maximum is not None else None
3166+
return check_row_count(_json_rows(rows), low, high)
3167+
3168+
31323169
def _coerce_diff_inputs(old_rows: Any, new_rows: Any, key: Any):
31333170
import json
31343171
if isinstance(old_rows, str):
@@ -4249,6 +4286,10 @@ def __init__(self):
42494286
"AC_categorical_drift": _categorical_drift,
42504287
"AC_diff_rows": _diff_rows,
42514288
"AC_cell_changes": _cell_changes,
4289+
"AC_check_foreign_key": _check_foreign_key,
4290+
"AC_check_unique_key": _check_unique_key,
4291+
"AC_check_accepted_values": _check_accepted_values,
4292+
"AC_check_row_count": _check_row_count,
42524293
"AC_unified_diff": _unified_diff,
42534294
"AC_apply_unified": _apply_unified,
42544295
"AC_three_way_merge": _three_way_merge,

0 commit comments

Comments
 (0)