Skip to content

feature - implement RFC 022 format functions (#39)#49

Open
dannymeijer wants to merge 8 commits into
mainfrom
feature/39-rfc022-format-functions
Open

feature - implement RFC 022 format functions (#39)#49
dannymeijer wants to merge 8 commits into
mainfrom
feature/39-rfc022-format-functions

Conversation

@dannymeijer
Copy link
Copy Markdown
Collaborator

@dannymeijer dannymeijer commented May 26, 2026

Summary

  • completes RFC 022's scalar format-function surface: deterministic hashes, URL helpers, JSON payload helpers, and CSV payload helpers
  • adds registry-backed helpers and Substrait extension mappings for concrete format helpers, with sha2(...) kept as a compatibility rewrite over concrete SHA-2 helpers
  • executes the full format helper set through the DataFusion adapter using native DataFusion functions where available and Incan-authored UDF callbacks for non-native helpers
  • removes the package-local Rust inql-datafusion-format adapter crate; the remaining adapter logic is Incan source using Rust bridge imports only where needed
  • records dynamic variant predicates as reserved/rejected until InQL has a variant value model, instead of exposing fake string predicates
  • marks RFC 022 Implemented and updates format reference docs, release notes, and the RFC index

Type of change

  • Bug fix
  • New feature
  • Refactor / maintenance
  • Documentation
  • RFC (adds/updates docs/rfcs/*)

Area(s)

  • Package & tests
  • Specification (RFCs)
  • Documentation
  • Automation & repo config
  • Other

Key details

  • User-facing behavior: Authors can use hash helpers (md5, sha1, concrete SHA-2 helpers, sha2, crc32, xxhash64), URL parse/encode/decode helpers, JSON validation/path/schema helpers, and CSV row/schema helpers in scalar projections.
  • Internals: Concrete helpers declare registry-owned Substrait extension mappings. DataFusion support is adapter-owned, but implemented in Incan source through src/session/datafusion_format_functions.incn rather than a package-local Rust crate.
  • Boundary: Dynamic variant predicates such as typeof, is_array, and is_object remain unavailable until a variant value model exists.
  • Compiler baseline: This branch expects Incan 0.3.0-rc23, which includes the Rust bridge callback/type-alias fixes needed by the Incan-authored DataFusion UDF bridge.

Testing / verification

Verified with Incan 0.3.0-rc23 candidate:

  • make fmt
  • git diff --check
  • incan test tests/test_hashing_functions.incn
  • incan test tests/test_format_functions.incn
  • incan test tests/test_function_registry.incn
  • incan test tests/test_substrait_plan.incn
  • incan test tests/test_session_projection.incn
  • make fmt-check
  • make test-style
  • make registry-metadata
  • make build
  • make build-locked
  • make test
  • make smoke-consumer
  • make pre-commit

Note: make registry-metadata still prints the known incan.lock is out of date warning for its package-root metadata command, but the command passes and make build-locked passes against the refreshed lockfile.

Docs impact

  • No docs changes needed
  • Docs updated

Links: docs/language/reference/functions/format.md, docs/language/reference/functions/index.md, docs/release_notes/v0_1.md, docs/rfcs/022_semi_structured_format_functions.md, docs/rfcs/README.md

Stack / Merge Order

Merge after Incan 0.3.0-rc23 is available on InQL CI's release/v0.3 checkout. This PR is full RFC 022 scope and closes #39.

Refs #39
Refs dannys-code-corner/incan#708
Refs dannys-code-corner/incan#709

@incan-triage-bot incan-triage-bot Bot added documentation Improvements or additions to documentation package Library source, tests, incan.toml specification docs/rfcs/ normative RFCs labels May 26, 2026
@dannymeijer dannymeijer force-pushed the feature/36-rfc019-window-functions branch from 083dfc2 to 298a26f Compare May 27, 2026 20:05
@dannymeijer dannymeijer force-pushed the feature/39-rfc022-format-functions branch from 4ebb788 to c786443 Compare May 28, 2026 07:10
@dannymeijer dannymeijer changed the title feature - implement RFC 022 hashing functions (#39) feature - implement RFC 022 format functions (#39) May 28, 2026
@dannymeijer dannymeijer changed the base branch from feature/36-rfc019-window-functions to main May 28, 2026 09:56
@dannymeijer dannymeijer marked this pull request as ready for review May 28, 2026 10:00
@incan-triage-bot incan-triage-bot Bot added the automation CI, Makefile, .github/, repo config label May 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automation CI, Makefile, .github/, repo config documentation Improvements or additions to documentation package Library source, tests, incan.toml specification docs/rfcs/ normative RFCs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC 022: Semi-structured and format functions

1 participant