[BUG] datadogAgentInternal enablement doesn't support duplicate datadogAgent custom resources.

[jess-belliveau]

Pre-submission Checklist

• I have searched existing issues and this is not a duplicate
• This is a Datadog Operator issue (CRDs, reconciliation, etc.), not a Datadog Agent or Datadog service problem (dashboards, monitors, etc.)

Operator version

1.23.0

Operator Helm chart version

2.18.0

Bug Report

What happened:
Upgrade of the helm chart set;

datadogAgentInternal:
  # datadogAgentInternal.enabled -- Enables the Datadog Agent Internal controller
  enabled: true

This caused the operator to fail to reconcile my existing datadogAgent custom resources causing all agents to be deleted/created in a loop

What I expected:
No change to existing datadogAgent deployments.

Steps to Reproduce

1. Install operator with helm chart 2.13.1 defaults
2. Deploy two datadogAgent custom resources
3. Upgrade operator via helm chart 2.18.0 using defaults
4. Operator will fail to reconcile existing datadogAgent deployments.

Environment

Kubernetes version: v1.32.11-eks-ac2d5a0
Helm version: v3.17.3
Cloud provider: AWS

Additional Context

Setting the following helm values for the operator chart disabled DDAI allows for deployment of datadogAgent CRs as expected;

        datadogAgentInternal:
          enabled: false
        datadogCRDs:
          crds:
            datadogAgentInternals: false

Screenshot shows successful deployment of two datadogAgent custom resources with the new datadogAgentInternal controller disabled:

[levan-m]

Hi @jess-belliveau,
Thanks for submitting issue and sorry the change affected your setup. Unfortunately we don't support multiple DatadogAgent resources in a given cluster. Our recommended way for managing multiple Agent daemonsets is via DatadogAgentProfile/DAP CRD which is now out of Preview.
Curious if you tried DAP and if anything is preventing you from using it instead of multiple DatadogAgents?

[jess-belliveau]

thanks @levan-m - i was worried that was going to be the answer.

I found the DAP page a little confusing, are these the only supported changes we can make per-agent "profile": https://github.com/DataDog/datadog-operator/blob/main/docs/datadog_agent_profiles.md#supported-settings

Or am I misunderstanding and you can pass a different features list for each profile? We run a limited set of APM and USM enabled nodes for one product team, hence our split agent requirement.

From memory, this old issue might explain my reluctance to trying DAP: #1529

Feature	apm agent	default agent
apm.enabled	true	false
logCollection.enabled	true	true
logCollection.containerCollectAll	true	true
liveProcessCollection.enabled	true	—
liveContainerCollection.enabled	true	—
processDiscovery.enabled	true	—
oomKill.enabled	true	—
tcpQueueLength.enabled	true	—
ebpfCheck.enabled	true	—
npm.enabled	true	—
usm.enabled	true	false
dogstatsd.unixDomainSocketConfig.enabled	true	—
otlp.receiver.protocols.grpc.enabled	true	true
remoteConfiguration.enabled	true	false
sbom.enabled	true	—
eventCollection.collectKubernetesEvents	true	—
orchestratorExplorer.enabled	true	true
kubeStateMetricsCore.enabled	true	—
admissionController.enabled	false	true
admissionController.agentCommunicationMode	—	socket
clusterChecks.enabled	false	—
prometheusScrape.enabled	true	true
externalMetricsServer.enabled	—	true
externalMetricsServer.useDatadogMetrics	—	true
externalMetricsServer.registerAPIService	—	false