diff --git a/README.md b/README.md index 2cecd8fd..1545d459 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ ## Table of Contents +- [What's new (2026-06-22) — RFC 8288 Link Header & Pagination](#whats-new-2026-06-22--rfc-8288-link-header--pagination) - [What's new (2026-06-22) — Referential Integrity Checks](#whats-new-2026-06-22--referential-integrity-checks) - [What's new (2026-06-22) — URI-Scheme Value References](#whats-new-2026-06-22--uri-scheme-value-references) - [What's new (2026-06-21) — W3C Baggage Propagation](#whats-new-2026-06-21--w3c-baggage-propagation) @@ -139,6 +140,12 @@ --- +## What's new (2026-06-22) — RFC 8288 Link Header & Pagination + +Parse `Link` headers and follow `rel="next"`. Full reference: [`docs/source/Eng/doc/new_features/v87_features_doc.rst`](docs/source/Eng/doc/new_features/v87_features_doc.rst). + +- **`parse_link_header` / `next_url` / `links_by_rel` / `paginate`** (`AC_parse_link_header`, `AC_next_url`): paginated REST APIs return `Link: <...>; rel="next"` but nothing parsed it. This parses the header (quoted values with commas, multiple links), indexes by relation, and `paginate` walks `rel="next"` over an injected `fetch` (transport/cassette) up to `max_pages`. Pure-stdlib, deterministic. + ## What's new (2026-06-22) — Referential Integrity Checks Foreign-key, unique, accepted-values and row-count checks across tables. Full reference: [`docs/source/Eng/doc/new_features/v86_features_doc.rst`](docs/source/Eng/doc/new_features/v86_features_doc.rst). diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index a68eaf0d..a124dba4 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -12,6 +12,7 @@ ## 目录 +- [本次更新 (2026-06-22) — RFC 8288 Link 标头与分页](#本次更新-2026-06-22--rfc-8288-link-标头与分页) - [本次更新 (2026-06-22) — 参照完整性检查](#本次更新-2026-06-22--参照完整性检查) - [本次更新 (2026-06-22) — URI-Scheme 值引用](#本次更新-2026-06-22--uri-scheme-值引用) - [本次更新 (2026-06-21) — W3C Baggage 传播](#本次更新-2026-06-21--w3c-baggage-传播) @@ -138,6 +139,12 @@ --- +## 本次更新 (2026-06-22) — RFC 8288 Link 标头与分页 + +解析 `Link` 标头并跟随 `rel="next"`。完整参考:[`docs/source/Zh/doc/new_features/v87_features_doc.rst`](../docs/source/Zh/doc/new_features/v87_features_doc.rst)。 + +- **`parse_link_header` / `next_url` / `links_by_rel` / `paginate`**(`AC_parse_link_header`、`AC_next_url`):分页的 REST API 返回 `Link: <...>; rel="next"`,但没有东西解析它。本功能解析该标头(含逗号的引号值、多个链接)、按关系索引,`paginate` 通过注入的 `fetch`(传输/卡带)跟随 `rel="next"`,上限为 `max_pages`。纯标准库、确定。 + ## 本次更新 (2026-06-22) — 参照完整性检查 跨数据表的外键、唯一键、accepted-values 与行数检查。完整参考:[`docs/source/Zh/doc/new_features/v86_features_doc.rst`](../docs/source/Zh/doc/new_features/v86_features_doc.rst)。 diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index 45e1d60f..619d7b3e 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -12,6 +12,7 @@ ## 目錄 +- [本次更新 (2026-06-22) — RFC 8288 Link 標頭與分頁](#本次更新-2026-06-22--rfc-8288-link-標頭與分頁) - [本次更新 (2026-06-22) — 參照完整性檢查](#本次更新-2026-06-22--參照完整性檢查) - [本次更新 (2026-06-22) — URI-Scheme 值參照](#本次更新-2026-06-22--uri-scheme-值參照) - [本次更新 (2026-06-21) — W3C Baggage 傳播](#本次更新-2026-06-21--w3c-baggage-傳播) @@ -138,6 +139,12 @@ --- +## 本次更新 (2026-06-22) — RFC 8288 Link 標頭與分頁 + +解析 `Link` 標頭並跟隨 `rel="next"`。完整參考:[`docs/source/Zh/doc/new_features/v87_features_doc.rst`](../docs/source/Zh/doc/new_features/v87_features_doc.rst)。 + +- **`parse_link_header` / `next_url` / `links_by_rel` / `paginate`**(`AC_parse_link_header`、`AC_next_url`):分頁的 REST API 回傳 `Link: <...>; rel="next"`,但沒有東西解析它。本功能解析該標頭(含逗號的引號值、多個連結)、依關係索引,`paginate` 透過注入的 `fetch`(傳輸/卡帶)跟隨 `rel="next"`,上限為 `max_pages`。純標準函式庫、具決定性。 + ## 本次更新 (2026-06-22) — 參照完整性檢查 跨資料表的外鍵、唯一鍵、accepted-values 與筆數檢查。完整參考:[`docs/source/Zh/doc/new_features/v86_features_doc.rst`](../docs/source/Zh/doc/new_features/v86_features_doc.rst)。 diff --git a/docs/source/Eng/doc/new_features/v87_features_doc.rst b/docs/source/Eng/doc/new_features/v87_features_doc.rst new file mode 100644 index 00000000..ad055730 --- /dev/null +++ b/docs/source/Eng/doc/new_features/v87_features_doc.rst @@ -0,0 +1,41 @@ +RFC 8288 Link Header & Pagination +================================= + +Paginated REST APIs return ``Link: <...>; rel="next"`` headers, but nothing +parsed them, so multi-page fetches needed manual glue. This parses the header +(handling quoted parameter values and multiple links), indexes links by their +relation, and walks ``rel="next"`` over an injected transport. + +Pure standard library (``re``); imports no ``PySide6``. The parser is pure and +``paginate`` takes an injected ``fetch`` callable, so pagination is CI-testable +without a live server. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import parse_link_header, next_url, links_by_rel, paginate + + header = '; rel="next", ; rel="last"' + links = parse_link_header(header) # [Link(uri=..., rel="next"), ...] + nxt = next_url(header) # "https://api/x?page=2" + last = links_by_rel(header)["last"].uri + + # Walk every page over an injected fetch (transport / cassette): + pages = paginate(start_url, fetch, max_pages=50) + +``parse_link_header`` returns a list of ``Link`` (``uri``, ``rel``, and all +``params``), tolerating quoted values that contain commas and multiple links in +one header. ``links_by_rel`` indexes by each (space-separated) relation, +``next_url`` is the ``rel="next"`` convenience, and ``paginate`` fetches a URL +and follows ``next`` links via the supplied ``fetch`` callable up to +``max_pages``. + +Executor commands +----------------- + +``AC_parse_link_header`` parses a header ``value`` into ``{links}``; +``AC_next_url`` returns ``{url}`` for ``rel="next"``. Both are exposed as MCP +tools (``ac_parse_link_header`` / ``ac_next_url``) and as Script Builder +commands under **Data**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index a9efc78d..f170e278 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -109,6 +109,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v84_features_doc doc/new_features/v85_features_doc doc/new_features/v86_features_doc + doc/new_features/v87_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v87_features_doc.rst b/docs/source/Zh/doc/new_features/v87_features_doc.rst new file mode 100644 index 00000000..cc92ff1f --- /dev/null +++ b/docs/source/Zh/doc/new_features/v87_features_doc.rst @@ -0,0 +1,34 @@ +RFC 8288 Link 標頭與分頁 +======================= + +分頁的 REST API 會回傳 ``Link: <...>; rel="next"`` 標頭,但沒有任何東西解析它,因此多頁抓取需要手動黏合。 +本功能解析該標頭(處理含引號的參數值與多個連結)、依關係索引連結,並透過注入的傳輸走訪 ``rel="next"``。 + +純標準函式庫(``re``);不匯入 ``PySide6``。解析器為純函式,``paginate`` 接受注入的 ``fetch`` callable, +因此分頁可在無線上伺服器的情況下於 CI 測試。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import parse_link_header, next_url, links_by_rel, paginate + + header = '; rel="next", ; rel="last"' + links = parse_link_header(header) # [Link(uri=..., rel="next"), ...] + nxt = next_url(header) # "https://api/x?page=2" + last = links_by_rel(header)["last"].uri + + # 透過注入的 fetch(傳輸 / 卡帶)走訪每一頁: + pages = paginate(start_url, fetch, max_pages=50) + +``parse_link_header`` 回傳 ``Link`` 清單(``uri``、``rel`` 與所有 ``params``),容許含逗號的引號值與單一 +標頭中的多個連結。``links_by_rel`` 依每個(以空白分隔的)關係索引,``next_url`` 是 ``rel="next"`` 的便利 +函式,``paginate`` 抓取一個 URL 並透過提供的 ``fetch`` callable 跟隨 ``next`` 連結,上限為 ``max_pages``。 + +執行器命令 +---------- + +``AC_parse_link_header`` 把標頭 ``value`` 解析成 ``{links}``;``AC_next_url`` 回傳 ``rel="next"`` 的 +``{url}``。兩者皆以 MCP 工具(``ac_parse_link_header`` / ``ac_next_url``)以及 Script Builder 中 **Data** +分類下的命令提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index 9e509380..fda74354 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -109,6 +109,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v84_features_doc doc/new_features/v85_features_doc doc/new_features/v86_features_doc + doc/new_features/v87_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 264fde4e..2fab3d57 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -412,6 +412,10 @@ from je_auto_control.utils.sse_client import ( SSEEvent, SSEParser, parse_event_stream, ) +# RFC 8288 Link header parsing + rel="next" pagination +from je_auto_control.utils.link_header import ( + Link, links_by_rel, next_url, paginate, parse_link_header, +) # W3C Trace Context propagation (traceparent / tracestate) from je_auto_control.utils.trace_context import ( SpanContext, TraceContextError, child_context, extract_context, @@ -933,6 +937,7 @@ def start_autocontrol_gui(*args, **kwargs): "HttpProblemError", "ProblemDetails", "is_problem", "parse_problem", "raise_for_problem", "SSEEvent", "SSEParser", "parse_event_stream", + "Link", "links_by_rel", "next_url", "paginate", "parse_link_header", "SpanContext", "TraceContextError", "child_context", "extract_context", "format_traceparent", "format_tracestate", "inject_context", "new_root_context", "new_span_id", "new_trace_id", "parse_traceparent", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 64ded93e..12a598bb 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -1710,6 +1710,22 @@ def _add_resilience_specs(specs: List[CommandSpec]) -> None: ), description="Parse a text/event-stream blob into events.", )) + specs.append(CommandSpec( + "AC_parse_link_header", "Data", "Link Header: Parse", + fields=( + FieldSpec("value", FieldType.STRING, + placeholder='; rel="next"'), + ), + description="Parse an RFC 8288 Link header into links.", + )) + specs.append(CommandSpec( + "AC_next_url", "Data", "Link Header: Next URL", + fields=( + FieldSpec("value", FieldType.STRING, + placeholder='; rel="next"'), + ), + description="Return the rel=next URL from a Link header.", + )) specs.append(CommandSpec( "AC_resolve_config", "Data", "Layered Config: Resolve", fields=( diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index 02d81517..7900fcf1 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3015,6 +3015,18 @@ def _resolve_refs(obj: Any) -> Dict[str, Any]: return {"resolved": resolve_refs_in(obj)} +def _parse_link_header(value: str) -> Dict[str, Any]: + """Adapter: parse an RFC 8288 Link header into {links}.""" + from je_auto_control.utils.link_header import parse_link_header + return {"links": [link.to_dict() for link in parse_link_header(value)]} + + +def _next_url(value: str) -> Dict[str, Any]: + """Adapter: return the rel=next URL from a Link header.""" + from je_auto_control.utils.link_header import next_url + return {"url": next_url(value)} + + def _baggage_parse(header: str) -> Dict[str, Any]: """Adapter: parse a W3C baggage header into {items}.""" from je_auto_control.utils.baggage import parse_baggage @@ -4274,6 +4286,8 @@ def __init__(self): "AC_baggage_format": _baggage_format, "AC_resolve_ref": _resolve_ref, "AC_resolve_refs": _resolve_refs, + "AC_parse_link_header": _parse_link_header, + "AC_next_url": _next_url, "AC_profile_rows": _profile_rows, "AC_infer_schema": _infer_schema, "AC_parse_problem": _parse_problem, diff --git a/je_auto_control/utils/link_header/__init__.py b/je_auto_control/utils/link_header/__init__.py new file mode 100644 index 00000000..76cd0c4e --- /dev/null +++ b/je_auto_control/utils/link_header/__init__.py @@ -0,0 +1,8 @@ +"""RFC 8288 Link header parsing and pagination for AutoControl.""" +from je_auto_control.utils.link_header.link_header import ( + Link, links_by_rel, next_url, paginate, parse_link_header, +) + +__all__ = [ + "Link", "links_by_rel", "next_url", "paginate", "parse_link_header", +] diff --git a/je_auto_control/utils/link_header/link_header.py b/je_auto_control/utils/link_header/link_header.py new file mode 100644 index 00000000..a4b78cb7 --- /dev/null +++ b/je_auto_control/utils/link_header/link_header.py @@ -0,0 +1,104 @@ +"""Parse RFC 8288 ``Link`` headers and follow ``rel="next"`` pagination. + +Paginated REST APIs return ``Link: <...>; rel="next"`` headers, but nothing +parsed them, so multi-page fetches needed manual glue. This parses the header +(handling quoted parameter values and multiple links), indexes links by their +relation, and walks ``rel="next"`` over an injected transport. + +Pure standard library (``re``); imports no ``PySide6``. The parser is pure and +``paginate`` takes an injected ``fetch`` callable, so pagination is CI-testable +without a live server. +""" +import re +from dataclasses import dataclass, field +from typing import Any, Callable, Dict, List, Mapping, Optional, Union + +_LINK_RE = re.compile(r"<([^>]*)>\s*(;[^<]*)?") + +Links = Union[str, List["Link"]] +Fetch = Callable[[str], Mapping[str, Any]] + + +@dataclass(frozen=True) +class Link: + """One RFC 8288 web link.""" + + uri: str + rel: Optional[str] = None + params: Dict[str, str] = field(default_factory=dict) + + def to_dict(self) -> Dict[str, Any]: + """Return a JSON-friendly view of the link.""" + return {"uri": self.uri, "rel": self.rel, "params": dict(self.params)} + + +def _strip_quotes(value: str) -> str: + if len(value) >= 2 and value[0] == '"' and value[-1] == '"': + return value[1:-1] + return value + + +def _parse_params(chunk: Optional[str]) -> Dict[str, str]: + params: Dict[str, str] = {} + for part in (chunk or "").split(";"): + cleaned = part.strip().rstrip(",").strip() + if not cleaned: + continue + key, sep, value = cleaned.partition("=") + key = key.strip().lower() + if key and sep: + params[key] = _strip_quotes(value.strip().rstrip(",").strip()) + return params + + +def parse_link_header(value: Optional[str]) -> List[Link]: + """Parse a ``Link`` header value into a list of :class:`Link`.""" + links: List[Link] = [] + for match in _LINK_RE.finditer(value or ""): + params = _parse_params(match.group(2)) + links.append(Link(uri=match.group(1).strip(), + rel=params.get("rel"), params=params)) + return links + + +def _as_links(links: Links) -> List[Link]: + return parse_link_header(links) if isinstance(links, str) else list(links) + + +def links_by_rel(links: Links) -> Dict[str, Link]: + """Index links by each (possibly space-separated) relation; last wins.""" + indexed: Dict[str, Link] = {} + for link in _as_links(links): + for token in (link.rel or "").split(): + indexed[token] = link + return indexed + + +def next_url(links: Links) -> Optional[str]: + """Return the ``rel="next"`` URI, if any.""" + link = links_by_rel(links).get("next") + return link.uri if link is not None else None + + +def _link_header_of(headers: Optional[Mapping[str, Any]]) -> Optional[str]: + for key, value in (headers or {}).items(): + if str(key).lower() == "link": + return str(value) + return None + + +def paginate(url: str, fetch: Fetch, *, + max_pages: int = 100) -> List[Mapping[str, Any]]: + """Fetch ``url`` then follow ``rel="next"`` Link headers via ``fetch``. + + ``fetch`` maps a URL to a response mapping with a ``headers`` dict. Stops at + ``max_pages`` or when no ``next`` link is present. Returns each response. + """ + responses: List[Mapping[str, Any]] = [] + current: Optional[str] = url + while current and len(responses) < max_pages: + response = fetch(current) + responses.append(response) + header = _link_header_of(response.get("headers")) + current = next_url(header) if header else None + return responses diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index 086ad577..ed2011d3 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -3343,6 +3343,28 @@ def rate_limit_tools() -> List[MCPTool]: ] +def link_header_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_parse_link_header", + description=("Parse an RFC 8288 Link header 'value' into {links} " + "(each {uri, rel, params}), handling quoted params and " + "multiple links."), + input_schema=schema({"value": {"type": "string"}}, ["value"]), + handler=h.parse_link_header, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_next_url", + description=("Return the rel=next URL from a Link header 'value' as " + "{url} (null when absent)."), + input_schema=schema({"value": {"type": "string"}}, ["value"]), + handler=h.next_url, + annotations=READ_ONLY, + ), + ] + + def referential_tools() -> List[MCPTool]: rows = {"type": "array", "items": {"type": "object"}} return [ @@ -5166,7 +5188,7 @@ def media_assert_tools() -> List[MCPTool]: trace_context_tools, baggage_tools, secret_ref_tools, data_profile_tools, http_problem_tools, dotenv_tools, sse_client_tools, layered_config_tools, data_drift_tools, - dataset_diff_tools, referential_tools, + dataset_diff_tools, referential_tools, link_header_tools, saga_tools, decision_table_tools, locator_repair_tools, pii_text_tools, sarif_tools, screen_record_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index b75a4bb5..72760dce 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -1746,6 +1746,16 @@ def resolve_refs(obj): return _resolve_refs(obj) +def parse_link_header(value): + from je_auto_control.utils.executor.action_executor import _parse_link_header + return _parse_link_header(value) + + +def next_url(value): + from je_auto_control.utils.executor.action_executor import _next_url + return _next_url(value) + + def profile_rows(rows, columns=None): from je_auto_control.utils.executor.action_executor import _profile_rows return _profile_rows(rows, columns) diff --git a/test/unit_test/headless/test_link_header_batch.py b/test/unit_test/headless/test_link_header_batch.py new file mode 100644 index 00000000..ef93cf00 --- /dev/null +++ b/test/unit_test/headless/test_link_header_batch.py @@ -0,0 +1,83 @@ +"""Headless tests for RFC 8288 Link header parsing. Pure stdlib, no Qt.""" +import je_auto_control as ac +from je_auto_control.utils.link_header import ( + links_by_rel, next_url, paginate, parse_link_header, +) + +_HEADER = ('; rel="next"; title="Page 2", ' + '; rel="last"') + + +def test_parse_multiple_links_and_params(): + links = parse_link_header(_HEADER) + assert len(links) == 2 + assert links[0].uri == "https://api.example.com/x?page=2" + assert links[0].rel == "next" and links[0].params["title"] == "Page 2" + assert links[1].rel == "last" + + +def test_links_by_rel_and_next_url(): + indexed = links_by_rel(_HEADER) + assert set(indexed) == {"next", "last"} + assert next_url(_HEADER) == "https://api.example.com/x?page=2" + assert next_url("; rel=\"prev\"") is None + + +def test_quoted_value_with_comma(): + header = '; rel="next"; title="a, b, c"' + [link] = parse_link_header(header) + assert link.params["title"] == "a, b, c" and link.rel == "next" + + +def test_space_separated_rel_indexes_each(): + indexed = links_by_rel('; rel="next prefetch"') + assert "next" in indexed and "prefetch" in indexed + + +def test_empty_header(): + assert parse_link_header("") == [] and next_url("") is None + + +def test_paginate_follows_next_over_injected_fetch(): + pages = { + "u1": {"headers": {"Link": '; rel="next"'}}, + "u2": {"headers": {"Link": '; rel="next"'}}, + "u3": {"headers": {}}, # no next → stop + } + responses = paginate("u1", lambda url: pages[url]) + assert len(responses) == 3 + assert responses is not None and responses[-1]["headers"] == {} + + +def test_paginate_respects_max_pages(): + response = {"headers": {"Link": '; rel="next"'}} + responses = paginate("self", lambda url: response, max_pages=4) + assert len(responses) == 4 + + +# --- wiring --------------------------------------------------------------- + +def test_executor_round_trip(): + rec = ac.execute_action([["AC_parse_link_header", {"value": _HEADER}]]) + links = next(v for v in rec.values() if isinstance(v, dict))["links"] + assert links[0]["rel"] == "next" + rec2 = ac.execute_action([["AC_next_url", {"value": _HEADER}]]) + assert next(v for v in rec2.values() if isinstance(v, dict))["url"] == \ + "https://api.example.com/x?page=2" + + +def test_wiring(): + known = ac.executor.known_commands() + assert {"AC_parse_link_header", "AC_next_url"} <= set(known) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_parse_link_header", "ac_next_url"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_parse_link_header", "AC_next_url"} <= specs + + +def test_facade_exports(): + for attr in ("Link", "parse_link_header", "links_by_rel", "next_url", + "paginate"): + assert hasattr(ac, attr) and attr in ac.__all__