Preserve Feature Flagging configuration source semantics

[leoromanovsky]

Motivation

Until now, Java customers could activate the Datadog OpenFeature provider only by setting DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true. That experimental input did more than enable a provider: the implementation it unlocked subscribed to the Agent's FFE_FLAGS Remote Configuration product. Customers who set it therefore adopted Feature Flags with Remote Configuration's Agent connectivity, network path, and delivery assumptions.

The new configuration model makes CDN-backed agentless delivery the default for new customers so they no longer need an experimental opt-in. We cannot reinterpret the existing legacy opt-in as agentless, however, because that would silently move an established customer to a different delivery system. During the migration window, explicit legacy true must remain on Remote Configuration, explicit legacy false must remain disabled, and only an absent legacy value may receive the new agentless default.

This also has a billing boundary. Requests to Remote Configuration and the CDN are billing surfaces, so "agentless is the default" cannot mean that installing or initializing the Java agent starts polling. The migration separates the concerns previously collapsed into the experimental switch: DD_FEATURE_FLAGS_ENABLED is the global kill switch, DD_FEATURE_FLAGS_CONFIGURATION_SOURCE chooses a delivery system when explicit, and application initialization or access to the Datadog OpenFeature provider activates default agentless polling. This PR implements that precedence on top of sealed #11892 without creating phantom traffic or breaking the existing RC cohort.

flowchart TD
  Start["Agent starts"] --> Enabled{"DD_FEATURE_FLAGS_ENABLED"}
  Enabled -->|"false"| Disabled["Provider disabled<br/>No RC subscription<br/>No CDN polling"]
  Enabled -->|"true or unset"| Explicit{"Explicit configuration source?"}
  Explicit -->|"remote_config"| RC["Eager FFE_FLAGS subscription"]
  Explicit -->|"agentless"| Wait["Wait for application provider access"]
  Explicit -->|"unsupported offline or invalid"| Closed["Fail closed<br/>No network delivery"]
  Explicit -->|"absent"| Legacy{"Legacy experimental variable?"}
  Legacy -->|"true"| RC
  Legacy -->|"false"| Disabled
  Legacy -->|"absent"| Wait
  Wait --> Access["Application initializes or accesses provider"]
  Access --> CDN["Start CDN-backed polling"]

Loading

Changes

Separate enablement from source selection. FeatureFlaggingConfig.Resolution now carries an enabled state independently from the selected source. A disabled provider has a null source; enabled configurations select only agentless or remote_config.

Apply the compatibility precedence once. Global false wins first, an explicit supported source wins next, and the legacy variable is consulted only when the new source is absent. The result is fixed during agent bootstrap.

Activate only the selected delivery path. Explicit and grandfathered Remote Configuration remain eager. Agentless remains lazy until application initialization or provider access. Invalid input, including reserved offline, fails closed without RC or CDN traffic.

Preserve migration observability. Legacy variable usage continues through the existing deprecation path so adoption can be measured during the migration window.

Decisions

The kill switch is absolute. DD_FEATURE_FLAGS_ENABLED=false disables the provider and prevents both CDN polling and the FFE_FLAGS subscription. true merely permits source resolution.

Existing customers stay on Remote Configuration. With no explicit new source, legacy true preserves RC and legacy false preserves disabled behavior. Only customers with no legacy value receive the new default.

Agentless is the default, not startup traffic. New customers default to agentless, but polling begins only after application initialization or provider access. An explicit agentless or remote_config selection still overrides the legacy value.

Only two sources are supported today. agentless and remote_config are valid. offline is reserved for future startup-provided UFC bytes and currently fails closed; it is not the disable switch.

Validation

Cross-repository system-tests proof

The exact Java head 7647879607f741c5961a450932c6a323689580a8 was validated against merged system-tests main at 54649784b2d2807522ebeaabae49ffe70e921014, which contains DataDog/system-tests#7332.

The parametric image was supplied all three locally built artifacts, including dd-openfeature rather than the Maven-released provider:

Artifact	SHA-256
dd-java-agent-1.65.0-SNAPSHOT.jar	6f5a773fb29e740766869b4a9393bc9ab7bce333a352734f1b04975205f581c7
dd-openfeature-1.65.0-SNAPSHOT.jar	ea7fd8f937be7b197119c4923f5d634e60c13fde318abb981c479c797992b3e2
dd-trace-api-1.65.0-SNAPSHOT.jar	ee0fc95dcfdc98530d3ce83ebab699d0fbc87b8dcfc24bf8d72997b443218fcd

At runtime the image reported java@1.65.0-SNAPSHOT+7647879607, tying the result back to this PR head.

PYTEST_XDIST_AUTO_NUM_WORKERS=12 TEST_LIBRARY=java ./run.sh PARAMETRIC --skip-parametric-build \
  -F 'tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Selection' \
  -F 'tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Poller_Concurrency::test_delayed_no_overlap' \
  'tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Selection' \
  'tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Poller_Concurrency::test_delayed_no_overlap'

Result: 20 passed in 33.04s.

Contract proven	Evidence in the passing selection
No phantom billing for new customers	Default agentless sends zero CDN requests and advertises no RC product at startup; provider access starts CDN delivery and produces the expected evaluation.
Explicit source precedence	Explicit agentless beats legacy true; explicit remote_config beats legacy false. Only the selected delivery path becomes active.
Existing-customer compatibility	Legacy true selects RC; legacy false disables delivery. The same behavior is proven when the legacy value is present but empty or whitespace.
Stable enablement semantics	Stable true permits lazy default agentless behavior. Stable false suppresses default agentless, explicit RC, and grandfathered legacy RC.
Missing-value parsing	Absent, empty, and whitespace-only source inputs all resolve as default agentless rather than as an invalid explicit source.
RC isolation	RC with no payload never falls back to the CDN.
Unsupported inputs	Invalid values and reserved offline both fail closed with RC and CDN silent.
Poller concurrency	A response slower than the poll interval still maintains max_in_flight == 1; polling requests do not overlap.

For completeness, an exploratory run of the entire 29-case file produced 28 passed, 1 failed. The remaining case is server-error-to-valid cold recovery: the current Java parametric application hard-codes a 100 ms provider initialization timeout while that scenario intentionally returns 500 first and configures the retry at 1 second. The provider consequently reports the expected initialization timeout, then transitions to PROVIDER_READY after the retry; the test cannot also require the original synchronous /ffe/start call to succeed within 100 ms. The source-selection and billing contract above is fully green; the remaining result is a system-test harness timing mismatch rather than a failure to recover.

No-Agent dogfooding proof

The same Java artifacts were staged into ffe-dogfooding at c2ca5be1ae59d7ef083ef731957881d23f95ff00. The run used the CDN JSON:API response shape (data.type=universal-flag-configuration, UFC under data.attributes) and started only mock-intake and app-java for the activation boundary. No datadog-agent service or container was present. The evaluator was added only after that boundary had passed.

Check	Observed result
Runtime artifacts	App log: Using local Java agent: .../dd-java-agent-1.65.0-SNAPSHOT.jar and App built with local dd-openfeature JAR.
Tracer startup	App log: Feature Flagging system awaiting application provider activation. After three seconds, mock configuration_requests_total remained 0.
Application activation	POST /activate-provider changed providerActivated to true; mock request count became 1 at /api/v2/feature-flagging/config/rules-based/server.
Provider lifecycle	/health reported providerReady=true, providerState=PROVIDER_READY, and a PROVIDER_READY event.
Direct application evaluation	POST /evaluate returned flag ffe-dogfooding-string-flag, variant dogfood-treatment, allocation dogfood-allocation, and reason STATIC.
Dogfood evaluator	/evaluate recorded Java HTTP 200/success with the same flag, treatment, allocation, and STATIC reason.
Dogfood dashboard	/api/latest-results reported Java PROVIDER_READY; the Java result was success with variant dogfood-treatment and reason STATIC.

The mock's JSON:API envelope adaptation and isolated Compose file were validation-only. The dogfood checkout was restored to a clean state after the run, and its mock tests still pass.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.04 s	13.90 s	[+0.4%; +1.6%] (maybe worse)
startup:insecure-bank:tracing:Agent	12.90 s	12.98 s	[-1.1%; -0.0%] (maybe better)
startup:petclinic:appsec:Agent	16.83 s	16.88 s	[-1.1%; +0.6%] (no difference)
startup:petclinic:iast:Agent	16.37 s	16.96 s	[-7.8%; +0.8%] (no difference)
startup:petclinic:profiling:Agent	16.64 s	16.81 s	[-2.2%; +0.1%] (no difference)
startup:petclinic:sca:Agent	16.89 s	16.93 s	[-1.2%; +0.7%] (no difference)
startup:petclinic:tracing:Agent	16.13 s	15.72 s	[-1.7%; +6.9%] (no difference)

Commit: b32988ca · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.