Add signed batch machine run context / ADR by krowvin · Pull Request #1772 · USACE/cwms-data-api

krowvin · 2026-06-09T04:41:36Z

Summary

This is the parent PR for the CWMS Batch Events M2M auth and dynamic runtime work.

CDA now supports the preferred production shape when Keycloak can mint the batch machine context directly into the normal access token. CDA consumes validated OIDC claims such as machine_auth and run_as_office, while still rejecting unregistered machine principals instead of auto-creating them.

The signed X-CWMS-Job-Context path remains in the design as a fallback for cases where Keycloak cannot provide the needed machine-run context without a custom extension. The current rollout avoids a Keycloak SPI by using office-scoped scheduler and runner service accounts.

Related Draft PRs

Area	PR	Notes
Batch Events registry/API/UI/dispatcher	USACE-WaterManagement/cwms-batch-events#184	Script registry, command/file source, resource profiles, schedule timezones, local timeout enforcement, repo browser, runtime broker.
Shared runner image	USACE/cwbi-wm-images#51	Runner fetches brokered env, mints CDA bearer token with office runner client, supports Python/Node/Java/shell.
cwms-python client	HydrologicEngineeringCenter/cwms-python#295	Supports `CDA_BEARER_TOKEN` and `BATCH_JOB_CONTEXT_TOKEN`.
cwms-cli client	HydrologicEngineeringCenter/cwms-cli#226	Routes CLI CDA sessions through batch-aware auth helper.
Airflow DAG config	USACE/airflow-config#429	Registry-driven scheduled jobs, per-office scheduler clients, schedule timezone handling.
SWT proof jobs	USACE-WaterManagement/swt-wm-cwbi-jobs#23	Office proof branch for bearer-token batch runtime.

Blocked by repository permissions / fork policy:

Area	Local branch	Status
AWS Batch CDK infra	`cwms-batch` `batch-dynamic-runtime/cdk-runtime-jobdefs`	Local branch is ready, but upstream rejected push and GitHub policy blocks forking `cwbi-dev-infrastructure/cwms-batch`.
Airflow infra secrets	`airflow` `batch-dynamic-runtime/airflow-batch-events-auth`	Local branch is ready, but upstream rejected push and GitHub policy blocks forking `cwbi-dev-infrastructure/airflow`.

Diagrams

Validation

CDA:

JAVA_HOME=C:\Program Files\Java\jdk-21 ./gradlew.bat :cwms-data-api:test --tests cwms.cda.security.BatchJobContextTest --no-daemon --stacktrace
JAVA_HOME=C:\Program Files\Java\jdk-21 ./gradlew.bat :cwms-data-api:integrationTests --tests cwms.cda.api.auth.OpenIdConnectTestIT --no-daemon --stacktrace
The integration test proves Keycloak service-account client_credentials tokens can carry machine_auth/run_as_office, CDA rejects unregistered machine principals, and CDA accepts registered office-scoped machine principals.

Cross-repo/local E2E evidence:

Batch Events local UI created and ran jobs using the new registry model.
A command-mode timeout job with timeoutMinutes=1 and a 90-second sleep failed locally with Local executor timeout after 60 seconds.
Airflow helper tests verify registry timezone evaluation, DST gap skip, and repeated local occurrence run-once behavior.
Batch Events UI build passed after Scripts Manager source/command/path browser/timezone changes.

Checklist

AI tools used

MikeNeilson

I'm being a bit pushy here. AND I may in fact be wrong about what we can accomplish with Keycloak and thus may end up having to go this route (there's nothing fundmentally wrong with the design... other that too much reliance on properties, though that's an initial design problem not a new problem) but I think if we can get Keycloak to handle the load the transition is a lot simpler.

MikeNeilson · 2026-06-09T13:48:18Z

                final String email = claims.get(EMAIL_CLAIM, String.class);
-                return dao.createUser(preferredUserName, oidcPrincipal, givenName, email);
+                DataApiPrincipal dataApiPrincipal = dao.createUser(preferredUserName, oidcPrincipal, givenName, email);
+                BatchJobContext.prepareContext(ctx, dataApiPrincipal);


If the "batch" user is getting created randomly here, we have an issue, In the context of a batch process this should be a failure.

Corrected in bc40f8b

MikeNeilson · 2026-06-09T13:53:20Z

+                throw new CwmsAuthException("Batch job context missing run_as_office",
+                    HttpServletResponse.SC_UNAUTHORIZED);
+            }
+            ctx.attribute(RUN_AS_OFFICE_ATTR, office.toUpperCase(Locale.ROOT));


While appropriate to put in to a future logging context... these shouldn't need to be part of the Request/Response attributes. Downstream components needing to know that is a definite code smell.

CDA now consumes this from the validated Keycloak/OIDC access token via claims like machine_auth=true and run_as_office=<office>, rather than relying on Batch to issue a separate signed context.

The remaining CDA-side knowledge is just the claim contract and normal user/principal lookup. It still rejects unregistered machine principals instead of auto-creating them. Locally I verified this with per-office Keycloak service accounts that mint the claims directly into the access token.

MikeNeilson · 2026-06-09T13:58:12Z

+
+Provide CWMS Data API with a trusted batch run context for jobs that execute through a shared machine identity.
+
+Batch runtimes will authenticate to CDA with a service account (via Keycloak). Each job will also provide a signed context token that identifies the authorized job launch context, including the office for which the scheduler or API approved the run.


Setting up additional signing is going to be difficult to get right, and will involve yet-more CDK changes. Granted they won't be difficult.

I would like us to first determine if we can, in some way, setup keycloak to be able to receive the required information (such as the office, and specific job identification) and return it back in the already signed JWT access token.

Some of the other information can, and should, still be provided but it doesn't really required singing it's just informational. The office would be used to set the session context (or by the future authorization system) to appropriately limit operations.

e.g. office + job identification can be readily tied to a policy to appropriately limit operations..

Addressed this some here

#1772 (comment)

…ime/cda-job-context # Conflicts: # docs/source/decisions/index.rst

krowvin · 2026-06-26T01:20:56Z

CDA now takes a validated OIDC claims machine_auth and run_as_office

Stoped auto creating machine principle

After checking the Keycloak paths, I do not think we can get dynamic per-job values like run_as_office and job_id into the access token using normal realm/client configuration alone. Built in protocol mappers work for static claims, service account/user attributes, roles, audiences, etc but they do not give us a clean supported way for Batch to send trusted per job context during client_credentials and have Keycloak validate and mint it into the access token.

Keycloak has a way to do this via an SPI/provider, such as a custom protocol mapper or script mapper deployed into Keycloak. Keycloak SPI docs https://www.keycloak.org/docs/latest/server_development/index.html#_providers

I do not think CWBI is likely to accept that operationally. If they did, we would need an owner and process for packaging, deploying, versioning, and maintaining that custom provider inside their Keycloak infrastructure.

Using a signed X-CWMS-Job-Context looks to be the way to go when Keycloak cannot create dynamic job context for us. CDA validates the normal Keycloak token for the machine principal, then separately validates the dispatcher signed job context for the run office.

I'm working right now to verify all this works completely on localhost and make sure it works as expected

MikeNeilson

Based on recent decussions, Needs to be updated.

Will leave it to you if you want to leave stale information place as reference, but ADR should reflect the current state of understanding.

MikeNeilson · 2026-07-07T21:47:31Z

+
+| Claim           | Description                                  |
+| --------------- | -------------------------------------------- |
+| `machine_auth`  | Marks the access token as a batch machine run |


Based on conversations today it was determined this was not pratical.

The machine_auth claim is fixed, so that one shouldn't be difficult to have provided.

MikeNeilson · 2026-07-07T21:53:52Z

+
+- **Pros**: Office context is represented directly by the service account.
+- **Cons**: Recreates service-account and secret sprawl as offices and runtimes grow.
+- **Selected for current rollout**: This avoids Keycloak custom extensions while still allowing AWS Batch job definitions and runner images to be shared across offices.


Statement needs updating.

MikeNeilson · 2026-07-07T21:56:43Z

+
+| Setting                                  | Description                                                    |
+| ---------------------------------------- | -------------------------------------------------------------- |
+| `cwms.dataapi.batch.jobContext.secret`   | Signing secret for validating HS256 job context tokens         |


would be better to do async and use a public key here, with the secret key over in cwms-batch secrets.... or even better some sort of AWS provided thing (might cost more though.)

krowvin added 4 commits June 8, 2026 19:28

Add signed batch run context

efbd26f

Report expired batch job context tokens

c7833ea

Document signed batch machine run context

aec1c95

Test signed batch job context

db72d62

krowvin changed the title ~~Add signed batch machine run context~~ Add signed batch machine run context / ADR Jun 9, 2026

krowvin marked this pull request as ready for review June 9, 2026 12:03

krowvin requested a review from MikeNeilson June 9, 2026 12:03

Merge branch 'develop' into batch-dynamic-runtime/cda-job-context

6b821f8

MikeNeilson reviewed Jun 9, 2026

View reviewed changes

krowvin added 4 commits June 10, 2026 07:55

Harden batch machine context handling

bc40f8b

Merge branch 'develop' into batch-dynamic-runtime/cda-job-context

2b732ff

Support Keycloak batch run context claims

819a2a1

Merge remote-tracking branch 'origin/develop' into batch-dynamic-runt…

ee7b8b7

…ime/cda-job-context # Conflicts: # docs/source/decisions/index.rst

krowvin added 2 commits June 25, 2026 22:29

Add local Keycloak batch service accounts

6a75856

Add local cwms audience mapper

f8a69f8

krowvin requested a review from MikeNeilson June 26, 2026 04:28

krowvin added 6 commits June 26, 2026 06:50

Add local SWT Airflow service account

8a1fbf3

Add scheduled M2M flow diagram to ADR

7b2ad68

Add local Batch Events admin user

5eb849e

Align local Batch Events admin principal

1fb7f67

Align batch M2M ADR with office service accounts

362e8bf

Update batch M2M ADR diagrams

a88f35e

krowvin self-assigned this Jun 27, 2026

Document batch Keycloak service accounts

dca48b1

MikeNeilson requested a review from msweier July 7, 2026 21:45

MikeNeilson requested a review from DanielTOsborne July 7, 2026 21:45

MikeNeilson requested changes Jul 7, 2026

View reviewed changes


		Provide CWMS Data API with a trusted batch run context for jobs that execute through a shared machine identity.

		Batch runtimes will authenticate to CDA with a service account (via Keycloak). Each job will also provide a signed context token that identifies the authorized job launch context, including the office for which the scheduler or API approved the run.

Uh oh!

Conversation

krowvin commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Draft PRs

Diagrams

System Overview

End User UI Flow

Airflow Scheduled Flow

Validation

Checklist

Uh oh!

MikeNeilson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krowvin commented Jun 26, 2026

Uh oh!

MikeNeilson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

krowvin commented Jun 9, 2026 •

edited

Loading