feat(spark): add Databricks OAuth M2M authentication

[rohitrsh]

Stacked draft 2/4 — do not merge out of order.

This branch currently includes the commit from #3452. After #3452 merges, this branch will be rebased onto master so the final review contains only the OAuth M2M change.

Summary

• Adds explicit oauth_m2m authentication while keeping PAT as the default.
• Resolves service-principal credentials from a workflow-namespace Secret or connector environment.
• Caches short-lived tokens and refreshes once after an HTTP 401 for get/delete operations.
• Adds task configuration, focused documentation, metadata persistence, and connector-level tests.

Stack order:

1. refactor(spark): isolate Databricks PAT authentication #3452 — PAT strategy refactor
2. OAuth M2M authentication (this PR)
3. Connector workload OIDC federation
4. Workflow-namespace ServiceAccount OIDC federation

Test plan

• Focused Databricks connector suite — 62 passed
• Full Spark plugin suite in Linux, Python 3.12, and OpenJDK 17 — 87 passed
• make lint
• Ruff lint and format hooks run twice on changed Python files
• git diff --check
• Spark plugin sdist/wheel build and import smoke test
• Changed-file, full-patch, commit-metadata, confidentiality, secret-pattern, and attribution audits

This PR remains draft until #3452 merges, the branch is rebased, all local gates are repeated, and hosted CI is green.