[Fleet] Include APM enrichment by default when switching data type to traces

[MichelLosier]

Summary

Resolves: https://github.com/elastic/ingest-dev/issues/7126

• When an otel input package that is not dynamic_signal_types has its data type switched in the integration policy UI to traces, this makes sure:
  • use_apm var is added for apm enrichment
  • The APM enrichment toggle is shown defaulted as on
• The effective data type from the compiled policy is used in the otel pipeline id
  • Example: if in input.yml.hbs defines service.pipelines[logs]:.., and data type is switched to traces this pipeline id should read as traces/otelcol-filelogreceiver... and not logs/otelcol-filelogreceiver...
• Fix traces span transform to not set datastream.dataset so its routed to the traces-generic.otel-<namespace> stream, as to spec: https://github.com/elastic/opentelemetry-dev/blob/main/docs/ingest/mapping/traces-mapping.md#span-events

Example config from agent policy

"service": {
      "pipelines": {
        "traces/otelcol-filelogreceiver-4c47088c-07be-4f5b-9892-4a65b393d36b-otelcol-filelog_otel-filelogreceiver-4c47088c-07be-4f5b-9892-4a65b393d36b": {
          "receivers": [
            "filelog/otelcol-filelogreceiver-4c47088c-07be-4f5b-9892-4a65b393d36b-otelcol-filelog_otel-filelogreceiver-4c47088c-07be-4f5b-9892-4a65b393d36b"
          ],
          "exporters": [
            "elasticapm/default",
            "forward"
          ],
          "processors": [
            "elasticapm/default",
            "transform/otelcol-filelogreceiver-4c47088c-07be-4f5b-9892-4a65b393d36b-otelcol-filelog_otel-filelogreceiver-4c47088c-07be-4f5b-9892-4a65b393d36b-routing"
          ]
        },
        "metrics/default-aggregated-apm-metrics": {
          "receivers": [
            "elasticapm/default"
          ],
          "processors": [
            "transform/default-apm-namespace-routing"
          ],
          "exporters": [
            "forward"
          ]
        },
        "traces": {
          "receivers": [
            "forward"
          ],
          "exporters": [
            "elasticsearch/default"
          ]
        },
        "metrics": {
          "receivers": [
            "forward"
          ],
          "exporters": [
            "elasticsearch/default"
          ]
        }
      }
    },
    "processors": {
      "transform/otelcol-filelogreceiver-4c47088c-07be-4f5b-9892-4a65b393d36b-otelcol-filelog_otel-filelogreceiver-4c47088c-07be-4f5b-9892-4a65b393d36b-routing": {
        "trace_statements": [
          {
            "context": "span",
            "statements": [
              """set(attributes["data_stream.type"], "traces")""",
              """set(attributes["data_stream.dataset"], "filelogreceiver")""",
              """set(attributes["data_stream.namespace"], "default")"""
            ]
          },
          {
            "context": "spanevent",
            "statements": [
              """set(attributes["data_stream.type"], "logs")""",
              """set(attributes["data_stream.namespace"], "default")"""
            ]
          }
        ]
      },
      "elasticapm/default": {},
      "transform/default-apm-namespace-routing": {
        "metric_statements": [
          {
            "context": "datapoint",
            "statements": [
              """set(attributes["data_stream.namespace"], "default")"""
            ]
          }
        ]
      }
    },
    "connectors": {
      "elasticapm/default": {},
      "forward": {}
    },
    "exporters": {
      "elasticsearch/default": {
        "endpoints": [
          "http://192.168.50.40:9200"
        ]
      }
    },

To test:

• Install the filelog_otel integration package
• Create a integration policy for it and adjust the data type to traces.
• Observe the APM Enrichment toggle appearing in enabled state
• Play with toggling enrichment toggle and data type radio
• When created, view the full agent policy to review the otel configuration GET kbn:/api/fleet/agent_policies/{id}/full

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
• Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• See some risk examples
• ...

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces automatic APM enrichment behavior when switching data stream type to traces for OTel inputs, affecting both frontend UI logic and server-side config generation. The author does not own any of the 13 modified files, all of which belong to @elastic/fleet, warranting review by the designated code owners.

^{You can customize Macroscope's approvability policy. Learn more.}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 420fe30f-a432-49ac-bdc7-6c5feabc4282

📥 Commits

Reviewing files that changed from the base of the PR and between 3bc6d3e and 1ef470b.

📒 Files selected for processing (11)

• x-pack/platform/plugins/shared/fleet/common/services/index.ts
• x-pack/platform/plugins/shared/fleet/common/services/policy_template.test.ts
• x-pack/platform/plugins/shared/fleet/common/services/policy_template.ts
• x-pack/platform/plugins/shared/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/components/steps/components/package_policy_input_panel.test.tsx
• x-pack/platform/plugins/shared/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/components/steps/components/package_policy_input_panel.tsx
• x-pack/platform/plugins/shared/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/components/steps/components/package_policy_input_stream.test.tsx
• x-pack/platform/plugins/shared/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/components/steps/components/package_policy_input_stream.tsx
• x-pack/platform/plugins/shared/fleet/server/services/agent_policies/otel_collector.test.ts
• x-pack/platform/plugins/shared/fleet/server/services/agent_policies/otel_collector.ts
• x-pack/platform/plugins/shared/fleet/server/services/epm/packages/input_type_packages.test.ts
• x-pack/platform/plugins/shared/fleet/server/services/epm/packages/input_type_packages.ts

💤 Files with no reviewable changes (1)

• x-pack/platform/plugins/shared/fleet/server/services/epm/packages/input_type_packages.test.ts

---

📝 Walkthrough

Walkthrough

This pull request centralizes APM variable and dynamic signal type detection logic by exporting utility functions from the fleet common services module. New helpers hasDynamicSignalTypes, shouldIncludeUseAPMVar, and addUseAPMVarIfNotPresent are introduced and re-exported, enabling consistent handling of use_apm stream variables across frontend and backend components. Existing duplicated implementations are consolidated to use the centralized helpers, and pipeline-key alignment logic is added for non-dynamic OTel packages. Comprehensive test coverage is added for the new functionality.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

• 🛠️ Update Documentation: Commit on current branch
• 🛠️ Update Documentation: Create PR

Warning

Tools execution failed with the following error:

Failed to run tools: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 1ef470b

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	2.3MB	2.3MB	+340.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	200.8KB	201.3KB	+459.0B

History

• 💔 Build #420749 failed feb10c4
• 💛 Build #419292 was flaky ece892a
• 💔 Build #419272 failed b7627ab
• 💔 Build #418253 failed a5d58d2

cc @MichelLosier

[juliaElastic]

LGTM