ArgoCD Entity data submission

argocd/README.md

@@ -165,6 +165,45 @@ See the [Autodiscovery Integration Templates][3] for guidance on applying the pa
 
 [Run the Agent's status subcommand][6] and look for `argocd` under the Checks section.
 
+### Generic resources (Beta)
+
+The Argo CD check can ship `Application`, `Cluster`, and `Repository` objects to Datadog as generic resources. The feature is disabled by default and is opt-in per instance.
+
+To enable it, set the following in your `argocd.d/conf.yaml`:
+
+```yaml
+instances:
+  - app_controller_endpoint: http://argocd-metrics:8082/metrics
+    collect_genresources: true
+    generic_resources_endpoint: https://<ARGOCD_HOST>
+    generic_resources_auth_token: <BEARER_TOKEN>
+```
+
+The token used by the collector needs `get` and `list` permissions on `applications`, `clusters`, and `repositories` in Argo CD. For example, with the built-in RBAC system:
+
+```
+p, role:datadog-genresources, applications, get, */*, allow
+p, role:datadog-genresources, applications, list, */*, allow
+p, role:datadog-genresources, clusters, get, *, allow
+p, role:datadog-genresources, clusters, list, *, allow
+p, role:datadog-genresources, repositories, get, *, allow
+p, role:datadog-genresources, repositories, list, *, allow
+```
+
+`generic_resources_auth_token` is an optional raw bearer token. When set, the collector adds an `Authorization: Bearer <token>` header to each REST request. Leave it unset to inherit whatever request authentication is already configured on the instance.
+
+Operator-tunable options:
+
+- `genresources_ttl_seconds` (default `21600`): time-to-live applied to each emitted resource. Resources expire `ttl_seconds` after the last observation.
+- `max_resources_per_cycle` (default `10000`): per-cycle cap, applied independently to each resource type. When an Argo CD API endpoint returns more, the excess is dropped and a warning is logged.
+- `extra_redaction_paths` (default `[]`): additional JSON paths appended to the built-in redaction deny-list. The list is additive; it cannot remove paths from the baseline.
+
+Behavioral notes:
+
+- On every Agent restart with `collect_genresources` enabled, the collector re-emits every `Application`, `Cluster`, and `Repository` on its first cycle. The burst is bounded by `max_resources_per_cycle` (applied per type) and self-corrects on subsequent cycles.
+- Disabling `collect_genresources` does not immediately delete previously-emitted resources. Resources expire on their own via `expire_at` (default 6 hours after the last observation).
+- Argo CD REST API reachability is reported as the `argocd.genresources.api.up` gauge tagged with `resource_type:argocd_application`, `resource_type:argocd_cluster`, or `resource_type:argocd_repository`. The gauge is `1` when the endpoint returns a successful response and `0` when it errors, so failures in one endpoint do not mask the health of the others.
+
 ## Data Collected
 
 ### Metrics

argocd/assets/configuration/spec.yaml

@@ -55,6 +55,65 @@ files:
         display_default: null
         example: http://argocd-commit-server:8087/metrics
         type: string
+    - name: collect_genresources
+      description: |
+        Enable the generic resources pilot collector that ships ArgoCD
+        Applications, Clusters, and Repositories to Datadog as generic
+        resources. Disabled by default.
+      value:
+        type: boolean
+        example: false
+    - name: generic_resources_endpoint
+      description: |
+        Base URL of the ArgoCD REST API (for example, ``https://argocd.example.com``).
+        Required when ``collect_genresources`` is set to ``true``.
+      value:
+        display_default: null
+        example: https://<ARGOCD_HOST>
+        type: string
+    - name: generic_resources_auth_token
+      description: |
+        Raw bearer token used to authenticate against the ArgoCD REST API.
+        When set, the collector adds ``Authorization: Bearer <token>`` to
+        each REST request. Leave unset to inherit the request authentication
+        configured on the instance (for example, the structured ``auth_token``
+        config object handled by the HTTP wrapper).
+      secret: true
+      value:
+        display_default: null
+        example: <BEARER_TOKEN>
+        type: string
+    - name: genresources_ttl_seconds
+      description: |
+        Time-to-live in seconds applied to every emitted resource.
+        Resources expire ``ttl_seconds`` after the last observation.
+        Minimum of 1.
+      value:
+        type: integer
+        example: 21600
+        minimum: 1
+    - name: max_resources_per_cycle
+      description: |
+        Maximum number of items emitted per resource type per check cycle.
+        When an ArgoCD API endpoint returns more than this, the excess is
+        dropped and a warning is logged. Applied independently to
+        Applications, Clusters, and Repositories.
+      value:
+        type: integer
+        example: 10000
+        minimum: 1
+    - name: extra_redaction_paths
+      description: |
+        Additional dotted JSON paths appended to the baseline redaction
+        deny-list of every collected resource type. ``[*]`` denotes array
+        iteration. A path that does not match any field for a given type
+        is silently skipped. This list is additive; it cannot remove paths
+        from the baseline.
+      value:
+        type: array
+        items:
+          type: string
+        example: []
     - template: instances/openmetrics
       overrides:
         openmetrics_endpoint.required: false

argocd/datadog_checks/argocd/check.py

@@ -15,6 +15,7 @@
     NOTIFICATIONS_CONTROLLER_METRICS,
     REPO_SERVER_METRICS,
 )
+from .resources import ArgocdResourceCollector
 
 (
     API_SERVER_NAMESPACE,
@@ -40,6 +41,12 @@ def __init__(self, name, init_config, instances):
         super(ArgocdCheck, self).__init__(name, init_config, instances)
         self.check_initializations.appendleft(self.parse_config)
         self.check_initializations.append(self.configure_additional_transformers)
+        self._resource_collector = ArgocdResourceCollector(self)
+
+    def check(self, instance):
+        if self.instance.get("collect_genresources"):
+            self._resource_collector.collect()
+        super().check(instance)
 
     def parse_config(self):
         endpoint_configs = [
@@ -97,7 +104,7 @@ def argocd_cluster_connection_status_transformer(_metric, sample_data, _runtime_
         return argocd_cluster_connection_status_transformer
 
     def configure_additional_transformers(self):
-        endpoints = [key for key in self.instance.keys() if "_endpoint" in key]
+        endpoints = [key for key in self.instance.keys() if "_endpoint" in key and self.instance[key] in self.scrapers]
         for endpoint in endpoints:
             if endpoint == "app_controller_endpoint":
                 self.scrapers[self.instance[endpoint]].metric_transformer.add_custom_transformer(

argocd/datadog_checks/argocd/config_models/__init__.py

@@ -1,7 +1,3 @@
-# (C) Datadog, Inc. 2022-present
-# All rights reserved
-# Licensed under a 3-clause BSD style license (see LICENSE)
-
 # This file is autogenerated.
 # To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
 #     ddev -x validate config -s <INTEGRATION_NAME>

argocd/datadog_checks/argocd/config_models/defaults.py

@@ -1,7 +1,3 @@
-# (C) Datadog, Inc. 2022-present
-# All rights reserved
-# Licensed under a 3-clause BSD style license (see LICENSE)
-
 # This file is autogenerated.
 # To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
 #     ddev -x validate config -s <INTEGRATION_NAME>
@@ -36,6 +32,10 @@ def instance_collect_counters_with_distributions():
     return False
 
 
+def instance_collect_genresources():
+    return False
+
+
 def instance_collect_histogram_buckets():
     return True
 
@@ -56,6 +56,10 @@ def instance_enable_legacy_tags_normalization():
     return True
 
 
+def instance_genresources_ttl_seconds():
+    return 21600
+
+
 def instance_histogram_buckets_as_distributions():
     return False
 
@@ -80,6 +84,10 @@ def instance_log_requests():
     return False
 
 
+def instance_max_resources_per_cycle():
+    return 10000
+
+
 def instance_min_collection_interval():
     return 15
 

argocd/datadog_checks/argocd/config_models/instance.py

@@ -1,7 +1,3 @@
-# (C) Datadog, Inc. 2022-present
-# All rights reserved
-# Licensed under a 3-clause BSD style license (see LICENSE)
-
 # This file is autogenerated.
 # To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
 #     ddev -x validate config -s <INTEGRATION_NAME>
@@ -101,6 +97,7 @@ class InstanceConfig(BaseModel):
     cache_metric_wildcards: Optional[bool] = None
     cache_shared_labels: Optional[bool] = None
     collect_counters_with_distributions: Optional[bool] = None
+    collect_genresources: Optional[bool] = None
     collect_histogram_buckets: Optional[bool] = None
     commit_server_endpoint: Optional[str] = None
     connect_timeout: Optional[float] = None
@@ -113,6 +110,10 @@ class InstanceConfig(BaseModel):
     exclude_metrics_by_labels: Optional[MappingProxyType[str, Union[bool, tuple[str, ...]]]] = None
     extra_headers: Optional[MappingProxyType[str, Any]] = None
     extra_metrics: Optional[tuple[Union[str, MappingProxyType[str, Union[str, ExtraMetrics]]], ...]] = None
+    extra_redaction_paths: Optional[tuple[str, ...]] = None
+    generic_resources_auth_token: Optional[str] = None
+    generic_resources_endpoint: Optional[str] = None
+    genresources_ttl_seconds: Optional[int] = Field(None, ge=1)
     headers: Optional[MappingProxyType[str, Any]] = None
     histogram_buckets_as_distributions: Optional[bool] = None
     hostname_format: Optional[str] = None
@@ -128,6 +129,7 @@ class InstanceConfig(BaseModel):
     kerberos_keytab: Optional[str] = None
     kerberos_principal: Optional[str] = None
     log_requests: Optional[bool] = None
+    max_resources_per_cycle: Optional[int] = Field(None, ge=1)
     metric_patterns: Optional[MetricPatterns] = None
     metrics: Optional[tuple[Union[str, MappingProxyType[str, Union[str, Metrics]]], ...]] = None
     min_collection_interval: Optional[float] = None

argocd/datadog_checks/argocd/config_models/shared.py

@@ -1,7 +1,3 @@
-# (C) Datadog, Inc. 2022-present
-# All rights reserved
-# Licensed under a 3-clause BSD style license (see LICENSE)
-
 # This file is autogenerated.
 # To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
 #     ddev -x validate config -s <INTEGRATION_NAME>

argocd/datadog_checks/argocd/data/conf.yaml.example

@@ -76,6 +76,52 @@ instances:
     #
     # commit_server_endpoint: http://argocd-commit-server:8087/metrics
 
+    ## @param collect_genresources - boolean - optional - default: false
+    ## Enable the generic resources pilot collector that ships ArgoCD
+    ## Applications, Clusters, and Repositories to Datadog as generic
+    ## resources. Disabled by default.
+    #
+    # collect_genresources: false
+
+    ## @param generic_resources_endpoint - string - optional
+    ## Base URL of the ArgoCD REST API (for example, ``https://argocd.example.com``).
+    ## Required when ``collect_genresources`` is set to ``true``.
+    #
+    # generic_resources_endpoint: https://<ARGOCD_HOST>
+
+    ## @param generic_resources_auth_token - string - optional
+    ## Raw bearer token used to authenticate against the ArgoCD REST API.
+    ## When set, the collector adds ``Authorization: Bearer <token>`` to
+    ## each REST request. Leave unset to inherit the request authentication
+    ## configured on the instance (for example, the structured ``auth_token``
+    ## config object handled by the HTTP wrapper).
+    #
+    # generic_resources_auth_token: <BEARER_TOKEN>
+
+    ## @param genresources_ttl_seconds - integer - optional - default: 21600
+    ## Time-to-live in seconds applied to every emitted resource.
+    ## Resources expire ``ttl_seconds`` after the last observation.
+    ## Minimum of 1.
+    #
+    # genresources_ttl_seconds: 21600
+
+    ## @param max_resources_per_cycle - integer - optional - default: 10000
+    ## Maximum number of items emitted per resource type per check cycle.
+    ## When an ArgoCD API endpoint returns more than this, the excess is
+    ## dropped and a warning is logged. Applied independently to
+    ## Applications, Clusters, and Repositories.
+    #
+    # max_resources_per_cycle: 10000
+
+    ## @param extra_redaction_paths - list of strings - optional
+    ## Additional dotted JSON paths appended to the baseline redaction
+    ## deny-list of every collected resource type. ``[*]`` denotes array
+    ## iteration. A path that does not match any field for a given type
+    ## is silently skipped. This list is additive; it cannot remove paths
+    ## from the baseline.
+    #
+    # extra_redaction_paths: []
+
     ## @param raw_metric_prefix - string - optional
     ## A prefix that is removed from all exposed metric names, if present.
     ## All configuration options will use the prefix-less name.