fix(backend): resolve use_secret_as_env pipeline params in ParallelFor

[nsingla]

Summary

Fix use_secret_as_env() (and related platform-config inputs) failing to resolve across sub-DAG boundaries (e.g., dsl.ParallelFor) by combining the broader fix from #13118 with API hardening from #13122.

What Changed

• Compiler/platform-config rewrite

  • Update sdk/python/kfp/compiler/pipeline_spec_builder.py to rewrite cross-boundary platform-config input references (including componentInputParameter and outer-DAG taskOutputParameter) to surfaced pipelinechannel--* inputs.
  • Apply rewrite logic in normal DAG compilation and exit-handler paths.

• Kubernetes config channel propagation

• Ensure platform-config-only channels are tracked so they propagate through sub-DAG boundaries.

• Update:

  • kubernetes_platform/python/kfp/kubernetes/common.py
  • kubernetes_platform/python/kfp/kubernetes/volume.py
  • Public SDK API hardening (from fix(backend): resolve use_secret_as_env pipeline params in ParallelFor #13122)

• Add PipelineTask.register_pipeline_channels(...) in sdk/python/kfp/dsl/pipeline_task.py.

  • Keep _register_pipeline_channels(...) as a compatibility wrapper.
  • Reuse the public helper where channel deduplication is needed (set_container_image path).
  • Add unit coverage in sdk/python/kfp/dsl/pipeline_task_test.py.

• Backend driver fix

  • Include DAG condition-file handling fix and tests:
    • backend/src/v2/cmd/driver/main.go
    • backend/src/v2/cmd/driver/main_test.go

• Regression tests and goldens

  • Add/extend tests for pipeline param + outer task output cases inside ParallelFor:
    • kubernetes_platform/python/test/unit/test_secret.py
    • sdk/python/kfp/compiler/pipeline_spec_builder_test.py
    • sdk/python/test/local_execution/local_execution_test.py
  • Add compiled workflow and SDK compiled pipeline goldens under test_data/compiled-workflows/ and test_data/sdk_compiled_pipelines/valid/....

Diff Size

19 files changed
+3161 / -19

Test Plan

• Added targeted unit/regression coverage for channel propagation and cross-DAG rewrite behavior.
• Updated compiled pipeline/workflow golden files for new cases.
• Run full CI suite to validate cross-language integration paths.

Checklist:

• You have signed off your commits
• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[Copilot]

Pull request overview

Fixes KFP v2 compilation/runtime gaps where Kubernetes platform-config inputs (e.g., use_secret_as_env(secret_name=<pipeline param>)) failed to resolve across sub-DAG boundaries such as dsl.ParallelFor, by (1) surfacing and propagating platform-config-only channels and (2) rewriting platform-config input references to the surfaced pipelinechannel--* inputs. Also hardens the SDK API for channel registration and adjusts driver behavior for DAG condition files.

Changes:

• Compiler: rewrite platform-config references (componentInputParameter, taskOutputParameter) to surfaced pipelinechannel--* names when compiling sub-DAGs (including exit-handler path).
• Kubernetes extension: ensure platform-config-only channels are registered so they propagate across sub-DAG boundaries; add a public SDK helper PipelineTask.register_pipeline_channels(...) and unit coverage.
• Backend driver: write "nil" condition for DAG driver type and add regression test; add/update goldens and regression pipelines.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test_data/sdk_compiled_pipelines/valid/pipeline_with_parallelfor_pipeline_param.yaml	New compiled-pipeline golden covering pipeline param + outer output inside ParallelFor.
test_data/sdk_compiled_pipelines/valid/pipeline_with_parallelfor_pipeline_param.py	New source pipeline to generate/validate the above golden.
test_data/sdk_compiled_pipelines/valid/critical/parallel_for_secret.yaml	New compiled-pipeline golden for the reported secret-in-ParallelFor regression.
test_data/sdk_compiled_pipelines/valid/critical/parallel_for_secret.py	New critical regression pipeline reproducing the secret-name pipeline param scenario.
test_data/sdk_compiled_pipelines/valid/critical/nested_parallel_for_secret.yaml	New golden for cross-DAG task output used as secret name in ParallelFor.
test_data/sdk_compiled_pipelines/valid/critical/nested_parallel_for_secret.py	New critical regression pipeline for cross-DAG taskOutputParameter rewriting.
test_data/compiled-workflows/pipeline_with_parallelfor_pipeline_param.yaml	New Argo workflow golden for parallel-for pipeline param propagation.
test_data/compiled-workflows/parallel_for_secret.yaml	New Argo workflow golden for secret-name pipeline param in ParallelFor.
test_data/compiled-workflows/nested_parallel_for_secret.yaml	New Argo workflow golden for cross-DAG output secret-name case.
sdk/python/test/local_execution/local_execution_test.py	Adds local execution coverage for the new ParallelFor pipeline-param scenario.
sdk/python/kfp/dsl/pipeline_task_test.py	Adds unit test for deduplicating channel registration via the new public helper.
sdk/python/kfp/dsl/pipeline_task.py	Introduces register_pipeline_channels(...) and reuses it to dedupe channels in existing paths.
sdk/python/kfp/compiler/pipeline_spec_builder_test.py	Adds end-to-end + unit tests for platform-config rewrite behavior across sub-DAG boundaries.
sdk/python/kfp/compiler/pipeline_spec_builder.py	Implements platform-config reference rewriting and wires it into DAG + exit-handler compilation.
kubernetes_platform/python/test/unit/test_secret.py	Adds unit tests asserting platform-config-only channels get registered for propagation.
kubernetes_platform/python/kfp/kubernetes/volume.py	Ensures PVC mount channel inputs propagate across sub-DAG boundaries.
kubernetes_platform/python/kfp/kubernetes/common.py	Registers pipeline channels referenced via Kubernetes platform config inputs.
backend/src/v2/cmd/driver/main_test.go	Adds regression test validating DAG driver writes a "nil" condition file.
backend/src/v2/cmd/driver/main.go	Writes "nil" condition for DAG driver type (in addition to ROOT_DAG/CONTAINER).

[hbelmiro]

/retest

[hbelmiro]

/retest

[mprahl]

@nsingla could you fix the CI failure?

[mprahl]

@nsingla could you fix the CI failure?

[jeffspahr]

test

[jeffspahr]

Two deterministic regressions remain after the parallel_and_nested/ fixture split.

I left the exact follow-up edits inline on the nearest changed hunks. GitHub does not expose the stale SDK golden paths or the stale Sanity fixture entry as diff lines in this PR, so those two spots cannot be attached as one-click suggestions even though the replacements are fully mechanical.

[jeffspahr]

/lgtm

[jeffspahr]

/retest

[mprahl]

/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mprahl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [mprahl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment