feat(spark): add Databricks OIDC connector authentication

[rohitrsh]

Stacked draft 3/4 — do not merge out of order.

This branch currently includes #3452 and #3453. After #3453 merges, this branch will be rebased onto master so the final review contains only connector workload OIDC federation.

Summary

• Adds explicit oidc_federation authentication for the connector workload identity.
• Reads a projected workload JWT and exchanges it for a short-lived Databricks token.
• Persists token-file, client-ID, and audience metadata so get/delete operations can refresh authentication.
• Adds task configuration, deployment documentation, token rotation tests, and connector-level coverage.
• Introduces no workflow-namespace ServiceAccount discovery or additional Kubernetes RBAC.

Stack order:

1. refactor(spark): isolate Databricks PAT authentication #3452 — PAT strategy refactor
2. feat(spark): add Databricks OAuth M2M authentication #3453 — OAuth M2M authentication
3. Connector workload OIDC federation (this PR)
4. Workflow-namespace ServiceAccount OIDC federation

Test plan

• Focused Databricks connector suite — 71 passed
• Full Spark plugin suite in Linux, Python 3.12, and OpenJDK 17 — 96 passed
• make lint
• Ruff lint and format hooks run twice on changed Python files
• git diff --check
• Spark plugin sdist/wheel build and import smoke test
• Changed-file, full-patch, commit-metadata, confidentiality, secret-pattern, and attribution audits

This PR remains draft until #3453 merges, the branch is rebased, all local gates are repeated, and hosted CI is green.