refactor(spark): isolate Databricks PAT authentication

[rohitrsh]

Summary

• Introduces a small DatabricksAuth strategy interface and a PATAuth implementation.
• Routes Databricks job creation through the strategy without changing namespace-secret lookup, connector environment fallback, persisted PAT metadata, or get/delete behavior.
• Adds focused tests for default and custom secret resolution and delegation to the existing token lookup.

This is the first of four smaller changes superseding #3429:

1. PAT strategy refactor (this PR)
2. OAuth M2M authentication
3. Connector workload OIDC federation
4. Workflow-namespace ServiceAccount OIDC federation

No new authentication mode, task configuration, or Kubernetes permission is introduced here. This also contributes to flyteorg/flyte#7319.

Test plan

• pytest tests/test_databricks_auth.py tests/test_databricks_token.py tests/test_connector.py -q — 49 passed
• Full Spark plugin coverage suite in Linux, Python 3.12, and OpenJDK 17 — 74 passed
• make lint
• make unit_test_codecov — 1,982 passed, 23 skipped
• Ruff lint and format hooks run twice on changed Python files
• git diff --check
• Spark plugin sdist/wheel build and import smoke test
• Changed files, full patch, commit metadata, internal-term denylist, and secret-pattern audit

The PR is draft until all hosted checks are green.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.79%. Comparing base (ca59398) to head (b787704).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3452   +/-   ##
=======================================
  Coverage   47.79%   47.79%           
=======================================
  Files         217      217           
  Lines       22943    22943           
  Branches     3036     3036           
=======================================
  Hits        10966    10966           
  Misses      11370    11370           
  Partials      607      607           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rohitrsh]

Hosted CI is complete. Every check relevant to this change, including build-plugins (3.12, flytekit-spark), passed.

The sole failure is build-plugins (3.12, flytekit-greatexpectations): its tests fetch https://raw.githubusercontent.com/superconductive/ge_tutorials/main/data/yellow_tripdata_sample_2019-01.csv, which now returns 404. The same check failed in the concurrent scheduled master run: https://github.com/flyteorg/flytekit/actions/runs/30547285133/job/90887953027.

I do not have permission to rerun that job. I am leaving this PR in draft as promised until a maintainer can rerun or confirm the unrelated baseline failure.