Skip to content

Add ExecutionPlan try_to_proto / try_from_proto hooks + ProjectionExec reference#23495

Open
adriangb wants to merge 2 commits into
apache:mainfrom
pydantic:execution-plan-proto-base
Open

Add ExecutionPlan try_to_proto / try_from_proto hooks + ProjectionExec reference#23495
adriangb wants to merge 2 commits into
apache:mainfrom
pydantic:execution-plan-proto-base

Conversation

@adriangb

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

This is the foundation for porting ExecutionPlan serialization off the central downcast_ref chain in datafusion/proto/src/physical_plan/mod.rs, mirroring what #21929 did for PhysicalExpr (tracked in #22418). Everything else in #23494 depends on this PR; it can't be parallelized because it defines the ctx API and wiring that every per-plan follow-up uses.

What changes are included in this PR?

In datafusion-physical-plan (feature-gated #[cfg(feature = "proto")]):

  • ExecutionPlan::try_to_proto(&self, ctx: &ExecutionPlanEncodeCtx) -> Result<Option<PhysicalPlanNode>> — a new trait method defaulting to Ok(None) (so un-migrated plans are unchanged).
  • ExecutionPlanEncodeCtx / ExecutionPlanDecodeCtx plus the internal ExecutionPlanEncode / ExecutionPlanDecode dispatch traits (defined here, implemented in datafusion-proto) — the same dependency-inversion the PhysicalExpr ctx uses. The ctx exposes encode_child/decode_child (+ _children), encode_expr/decode_expr, and typed bytes-only encode_udaf/decode_udaf (+ udf/udwf) for function-carrying plans. No proto type is ever named by physical-plan beyond the pure prost types in datafusion-proto-models.
  • An expect_plan_variant! macro mirroring expect_expr_variant!.

In datafusion-proto:

  • ConverterPlanEncoder / ConverterPlanDecoder implement the dispatch traits over the existing PhysicalExtensionCodec + converter (the function serde reuses today's fun_definition byte semantics exactly). The encode dispatch calls the hook first (resolving downcast_delegate() beforehand so wrapper plans serialize as their delegate), then falls back to the existing downcast chain.

ProjectionExec is migrated as the reference implementation: its old encode arm is deleted and its decode arm reduced to ProjectionExec::try_from_proto, so a green roundtrip proves the hook is the only path.

Are these changes tested?

Yes — the existing proto_integration roundtrip suite (including TPC-H) passes unchanged (189 tests), which exercises the ProjectionExec migration end-to-end. cargo clippy --all-targets -- -D warnings and cargo fmt are clean.

Are there any user-facing changes?

Additive only in this PR (new trait method with a default, new pub ctx types). No wire-format change. The broader effort (#23494) will remove the pub-for-proto PhysicalPlanNodeExt scaffolding as plans migrate — that is a documented api change, but this base PR does not remove any public API.

adriangb and others added 2 commits July 12, 2026 12:50
…nExec reference (apache#22419)

General part: ExecutionPlanEncodeCtx/DecodeCtx + internal dispatch traits in
datafusion-physical-plan (mirrors the PhysicalExpr pattern), the try_to_proto
trait hook, expect_plan_variant! macro, and the ConverterPlanEncoder/Decoder
adapters wiring the hook into the central dispatch in datafusion-proto.

ProjectionExec migrated as the reference: encode downcast arm deleted, decode
arm reduced to a one-liner delegating to ProjectionExec::try_from_proto. All
189 proto_integration roundtrip tests pass (incl. TPC-H).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SfBPC2yYCwHUn4V8pQCVj3
…roto ctx (apache#22419)

Adds encode_udf/udaf/udwf + decode_udf/udaf/udwf to ExecutionPlanEncodeCtx/
DecodeCtx (and the internal dispatch traits). These take datafusion-expr types
+ Vec<u8> and never name a proto type; the datafusion-proto adapter backs them
over PhysicalExtensionCodec + the registry, owning the payload/registry
lookup-order policy in one place. Wire-identical to the existing fun_definition
semantics ((!buf.is_empty()).then_some(buf)).

This lets function-carrying plans (Aggregate, window, ScalarSubquery) ride the
polymorphic hook instead of staying as typed arms — possible because
physical-plan sits above datafusion-expr (unlike the expr-side ctx). apache#23421's
ScalarUdfCodec stays closed; ScalarFunctionExpr stays special-cased on the expr
side where the crate cycle forces it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SfBPC2yYCwHUn4V8pQCVj3
@github-actions github-actions Bot added proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 12, 2026
@adriangb adriangb added the api change Changes the API exposed to users of the crate label Jul 12, 2026
@github-actions

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  38.359s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.139s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  37.621s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.139s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.631s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  78.562s] datafusion-physical-plan
    Building datafusion-proto v54.0.0 (current)
       Built [  60.139s] (current)
     Parsing datafusion-proto v54.0.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-proto v54.0.0 (baseline)
       Built [  58.240s] (baseline)
     Parsing datafusion-proto v54.0.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-proto v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.238s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure trait_method_missing: pub trait method removed or renamed ---

Description:
A trait method is no longer callable, and may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-item-signature
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/trait_method_missing.ron

Failed in:
  method try_into_projection_physical_plan of trait PhysicalPlanNodeExt, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/37ab8be13065e117d97bf800e9e00efed983803e/datafusion/proto/src/physical_plan/mod.rs:734
  method try_from_projection_exec of trait PhysicalPlanNodeExt, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/37ab8be13065e117d97bf800e9e00efed983803e/datafusion/proto/src/physical_plan/mod.rs:2433

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 120.312s] datafusion-proto

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate auto detected api change Auto detected API change physical-plan Changes to the physical-plan crate proto Related to proto crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant