diff --git a/configuring/cluster-logging-collector.adoc b/configuring/cluster-logging-collector.adoc index c7bdfda2f7af..6564d50fd0ae 100644 --- a/configuring/cluster-logging-collector.adoc +++ b/configuring/cluster-logging-collector.adoc @@ -63,4 +63,15 @@ include::modules/configuring-network-policy-rule-set-for-logfilemetricexporter.a include::modules/creating-an-adminnetworkpolicy-rule-for-collector-network-policy.adoc[leveloffset=+2] +[id="collector-metrics-and-monitoring-impact_{context}"] +== Collector metrics and monitoring impact + +The Vector log collector exposes metrics that can affect the {product-title} monitoring stack in environments with complex log forwarding configurations. + +include::modules/collector-metrics-cardinality-impact.adoc[leveloffset=+2] + +include::modules/best-practices-multitenant-logging.adoc[leveloffset=+2] + +include::modules/troubleshooting-collector-metrics-cardinality.adoc[leveloffset=+2] + //include::modules/cluster-logging-collector-tuning.adoc[leveloffset=+1] diff --git a/modules/best-practices-multitenant-logging.adoc b/modules/best-practices-multitenant-logging.adoc new file mode 100644 index 000000000000..b9f5b6ececd6 --- /dev/null +++ b/modules/best-practices-multitenant-logging.adoc @@ -0,0 +1,296 @@ +// Module included in the following assemblies: +// +// * observability/logging/log_collection_forwarding/cluster-logging-collector.adoc + +:_mod-docs-content-type: REFERENCE +[id="best-practices-multitenant-logging_{context}"] += Best practices for multitenant logging configurations + +[role="_abstract"] +In multitenant {product-title} clusters, you can configure logging to isolate logs between tenants while minimizing the impact on the monitoring stack. The key is to balance tenant isolation requirements with the metrics cardinality that your configuration creates. + +Instead of creating separate inputs for each tenant or namespace, use label selectors to route logs from multiple sources through a single input. + +The following configuration example shows an anti-pattern with separate inputs per namespace: +[source,yaml] +---- +apiVersion: observability.openshift.io/v1 +kind: ClusterLogForwarder +metadata: + name: tenant-logs + namespace: openshift-logging +spec: + inputs: + - name: tenant-a-logs + type: application + application: + namespaces: + - tenant-a + - name: tenant-b-logs + type: application + application: + namespaces: + - tenant-b + # ... 98 more tenant inputs + outputs: + - name: tenant-a-splunk + type: splunk + # ... + - name: tenant-b-splunk + type: splunk + # ... + pipelines: + - name: tenant-a-pipeline + inputRefs: + - tenant-a-logs + outputRefs: + - tenant-a-splunk + - name: tenant-b-pipeline + inputRefs: + - tenant-b-logs + outputRefs: + - tenant-b-splunk +---- + +Each tenant input creates multiple Vector components, increasing cardinality. This configuration with 100 tenants creates approximately 400-500 unique `component_id` values. + +The following configuration example shows the recommended approach with a single input and filtering: +[source,yaml] +---- +apiVersion: observability.openshift.io/v1 +kind: ClusterLogForwarder +metadata: + name: tenant-logs + namespace: openshift-logging +spec: + inputs: + - name: all-tenants + type: application + application: + selector: + matchLabels: + tenant-logging: "enabled" + filters: + - name: tenant-a-filter + type: drop + drop: + - test: + - field: .kubernetes.namespace_name + notMatches: "^tenant-a$" + - name: tenant-b-filter + type: drop + drop: + - test: + - field: .kubernetes.namespace_name + notMatches: "^tenant-b$" + outputs: + - name: tenant-a-splunk + type: splunk + # ... + - name: tenant-b-splunk + type: splunk + # ... + pipelines: + - name: tenant-a-pipeline + inputRefs: + - all-tenants + filterRefs: + - tenant-a-filter + outputRefs: + - tenant-a-splunk + - name: tenant-b-pipeline + inputRefs: + - all-tenants + filterRefs: + - tenant-b-filter + outputRefs: + - tenant-b-splunk +---- + +A single input for all tenant logs reduces component count. Use pod labels to control which pods are included in log collection. This configuration creates far fewer Vector components because there is only one input source. + +If multiple tenants send logs to the same destination system (for example, the same Splunk instance or `LokiStack`), use a single output rather than creating separate outputs per tenant. + +The following configuration example shows an anti-pattern with separate outputs per tenant to the same destination: +[source,yaml] +---- +spec: + outputs: + - name: tenant-a-splunk + type: splunk + splunk: + url: https://splunk.example.com:8088 + token: + secretName: tenant-a-splunk-token + - name: tenant-b-splunk + type: splunk + splunk: + url: https://splunk.example.com:8088 + token: + secretName: tenant-b-splunk-token +---- + +Both outputs point to the same Splunk instance, creating duplicate components. + +The following configuration example shows the recommended approach with a single output and tenant identification: +[source,yaml] +---- +spec: + outputs: + - name: shared-splunk + type: splunk + splunk: + url: https://splunk.example.com:8088 + token: + secretName: splunk-token + pipelines: + - name: tenant-a-pipeline + inputRefs: + - all-tenants + filterRefs: + - tenant-a-filter + outputRefs: + - shared-splunk + - name: tenant-b-pipeline + inputRefs: + - all-tenants + filterRefs: + - tenant-b-filter + outputRefs: + - shared-splunk +---- + +A single output reduces component count. Multiple pipelines can share the same output. Tenant isolation is maintained by the namespace information in the log records. Use Splunk, Loki, or other destination capabilities to filter and route logs by tenant. + +Each pipeline creates additional components for routing and processing. Where possible, combine pipelines that share inputs and outputs. + +Separate pipelines are necessary when: +* Logs require different transformations before reaching different outputs +* Security or compliance requires strict separation of processing paths +* Different tenants require different delivery guarantees (for example, `AtLeastOnce` versus `AtMostOnce`) + +Pipelines can be combined when: +* Logs go through the same filters to reach the same output +* Only difference is the source namespace or labels +* Tenant isolation is handled at the destination + +Creating multiple `ClusterLogForwarder` custom resources increases the overall component count because each `ClusterLogForwarder` deploys a separate collector pod with its own set of components. + +Use multiple `ClusterLogForwarder` instances when: +* Different service accounts are required for different log collection purposes +* Different security or network policies apply +* Logs from different sources require completely different processing pipelines + +A single `ClusterLogForwarder` is sufficient when: +* All logs can use the same service account +* Tenant isolation is achieved through filtering and routing +* Network policies allow a single collector to reach all destinations + +The following example shows a multitenant logging configuration that balances tenant isolation with low metrics cardinality: + +[source,yaml] +---- +apiVersion: observability.openshift.io/v1 +kind: ClusterLogForwarder +metadata: + name: multitenant-logging + namespace: openshift-logging +spec: + serviceAccount: + name: logcollector + inputs: + - name: application-logs + type: application + application: + selector: + matchExpressions: + - key: logging-enabled + operator: In + values: + - "true" + - name: infrastructure-logs + type: infrastructure + filters: + - name: add-tenant-label + type: detectMultilineException + outputs: + - name: tenant-lokistack + type: lokiStack + lokiStack: + target: + name: logging-loki + namespace: openshift-logging + - name: compliance-s3 + type: s3 + s3: + bucketName: audit-logs + # ... + pipelines: + - name: application-to-loki + inputRefs: + - application-logs + filterRefs: + - add-tenant-label + outputRefs: + - tenant-lokistack + - name: infrastructure-to-loki + inputRefs: + - infrastructure-logs + outputRefs: + - tenant-lokistack + - name: audit-to-s3 + inputRefs: + - audit + outputRefs: + - compliance-s3 +---- ++ +This configuration uses: ++ +-- +* A single input for all application logs, controlled by pod label +* A single filter that applies to all tenants +* A single LokiStack output for all tenant application and infrastructure logs +* A separate output for compliance because it goes to a different destination type +* Three pipelines for different log types, sharing inputs and outputs where possible +-- ++ +This configuration creates approximately 40-60 `component_id` values, compared to 400-500 values for a per-tenant input/output design. + +Tenant isolation is achieved in `LokiStack` through the namespace labels in the logs. Use `LogQL` queries to filter logs by tenant: + +[source,logql] +---- +{kubernetes_namespace_name="tenant-a"} +---- + +Use the following rough estimates to predict the cardinality impact of your `ClusterLogForwarder` configuration: + +* Each input creates approximately 5-10 components (source + metadata + routing) +* Each output creates approximately 3-5 components (routing + sink + buffer management) +* Each filter creates approximately 1-2 components +* Each pipeline adds approximately 2-4 components for routing + +For example, a `ClusterLogForwarder` with: + +* 5 inputs → ~40 components +* 3 outputs → ~12 components +* 2 filters → ~4 components +* 5 pipelines → ~15 components + +Estimated total: ~70 component_id values + +Each histogram metric creates 10 time series per component_id: + +* `vector_component_received_events_count_bucket`: 70 × 10 = 700 series +* `vector_buffer_send_duration_seconds_bucket`: 70 × 10 = 700 series + +These estimates are approximate. Use the diagnostic procedures in "Troubleshooting high collector metrics cardinality" to measure actual cardinality. + +[role="_additional-resources"] +.Additional resources + +* xref:../configuring/cluster-logging-collector.adoc#collector-metrics-cardinality-impact_cluster-logging-collector[Understanding collector metrics cardinality and monitoring impact] +* xref:../configuring/cluster-logging-collector.adoc#troubleshooting-collector-metrics-cardinality_cluster-logging-collector[Troubleshooting high collector metrics cardinality] +* xref:../configuring/configuring-log-forwarding.adoc#configuring-inputs_configuring-log-forwarding[Configuring inputs] +* xref:../configuring/configuring-log-forwarding.adoc#configuring-filters_configuring-log-forwarding[Configuring filters] diff --git a/modules/collector-metrics-cardinality-impact.adoc b/modules/collector-metrics-cardinality-impact.adoc new file mode 100644 index 000000000000..c908a9283c99 --- /dev/null +++ b/modules/collector-metrics-cardinality-impact.adoc @@ -0,0 +1,109 @@ +// Module included in the following assemblies: +// +// * observability/logging/log_collection_forwarding/cluster-logging-collector.adoc + +:_mod-docs-content-type: CONCEPT +[id="collector-metrics-cardinality-impact_{context}"] += Understanding collector metrics cardinality and monitoring impact + +[role="_abstract"] +The Vector log collector exposes Prometheus metrics that track the performance and health of log collection. In deployments with many inputs, outputs, and pipelines, these metrics can exhibit high cardinality, which increases resource consumption in the {product-title} monitoring stack. + +Understanding how `ClusterLogForwarder` configuration affects metrics cardinality helps you balance tenant isolation requirements with monitoring stack stability. + +Metrics cardinality refers to the number of unique time series created by a metric. Each unique combination of metric name and label values creates a separate time series that Prometheus must store and query. For example, if a metric has labels `component_id` and `namespace`, and there are 100 unique `component_id` values and 50 unique `namespace` values, the potential cardinality is 100 × 50 = 5,000 time series for that metric. + +High cardinality metrics require more memory in Prometheus pods to store time series, more CPU to process queries across many series, more storage to persist historical data, and longer query response times. + +The Vector log collector creates internal components for each stage of log processing. Each component is identified by a `component_id` label in Vector metrics. The number of components grows with the complexity of your `ClusterLogForwarder` configuration. + +A `ClusterLogForwarder` with the following configuration: + +* 3 inputs (application, infrastructure, audit) +* 2 outputs (LokiStack, Splunk) +* 2 pipelines (one per output) + +Creates these types of components: + +* Input source components (3) +* Metadata enrichment transforms per input (3) +* Filter transforms if configured (varies) +* Routing and transformation components (varies) +* Output sink components (2) + +Each input typically requires multiple transformation stages before reaching an output, creating 5-10 components per input-output path. In this example, you might have 30-50 unique `component_id` values. + +In multitenant environments with many separate inputs, outputs, or pipelines for tenant isolation, the number of components can grow to hundreds: + +* 100 tenant-specific inputs +* 50 tenant-specific outputs +* 100 tenant-specific pipelines + +This configuration can create 400-500 unique `component_id` values or more. + +The following Vector metrics use the `component_id` label and are most affected by high cardinality: + +`vector_component_received_events_count_bucket`:: Histogram tracking the number of events received by each component. This is typically the highest cardinality metric. + +`vector_buffer_send_duration_seconds_bucket`:: Histogram tracking how long it takes to send events from buffers. High cardinality because it tracks every component that sends data. + +`vector_source_lag_time_seconds_bucket`:: Histogram tracking lag time for source components. + +`vector_adaptive_concurrency_*_bucket`:: Multiple histogram metrics tracking adaptive concurrency behavior for outputs that support it. Each output creates multiple series. + +Histogram metrics are particularly problematic because they create multiple time series per unique label combination (one per bucket plus sum and count). + +A single `component_id` value in a histogram metric with 8 buckets creates: + +* 8 bucket series (`le="0.001"`, `le="0.01"`, ... `le="+Inf"`) +* 1 sum series (`_sum`) +* 1 count series (`_count`) + +Total: 10 time series per `component_id`. + +With 450 unique `component_id` values: + +* `vector_component_received_events_count_bucket`: 450 × 10 = 4,500 time series +* `vector_buffer_send_duration_seconds_bucket`: 450 × 10 = 4,500 time series +* Additional histogram metrics contribute similarly + +High cardinality from Vector metrics affects the {product-title} monitoring stack in several ways: + +Memory consumption:: Prometheus stores all active time series in memory. High cardinality can cause Prometheus pods to consume significantly more memory, potentially triggering out-of-memory (OOM) conditions. + +CPU usage:: Query processing time increases with the number of time series. High cardinality metrics slow down Prometheus queries and dashboard rendering. + +Storage growth:: Prometheus persistently stores time series data. High cardinality increases the rate of storage consumption, potentially exhausting available storage faster than expected. + +Query performance:: Queries that aggregate across all `component_id` values become slower as cardinality increases. This affects monitoring dashboards and alerting evaluation. + +Cluster stability:: The monitoring stack is critical for cluster operations including: ++ +-- +* Cluster upgrades +* Horizontal pod autoscaling (HPA) +* Vertical pod autoscaling (VPA) +* Node management +* Alerting +-- ++ +Monitoring stack instability or resource exhaustion can impact these operations. + +Consider reviewing your `ClusterLogForwarder` configuration if: + +* You have more than 50 inputs, outputs, or pipelines across all `ClusterLogForwarder` instances +* You have separate `ClusterLogForwarder` instances per tenant or namespace +* Prometheus pods are consuming more memory than expected +* Prometheus storage is growing faster than anticipated +* Prometheus queries are slow or timing out +* You observe high cardinality warnings in Prometheus logs + +You can check the current cardinality of Vector metrics by using the Prometheus API. See "Troubleshooting high collector metrics cardinality" for diagnostic procedures. + +[role="_additional-resources"] +.Additional resources + +* xref:../configuring/cluster-logging-collector.adoc#best-practices-multitenant-logging_cluster-logging-collector[Best practices for multitenant logging configurations] +* xref:../configuring/cluster-logging-collector.adoc#troubleshooting-collector-metrics-cardinality_cluster-logging-collector[Troubleshooting high collector metrics cardinality] +* link:https://prometheus.io/docs/practices/naming/#labels[Prometheus metric and label naming best practices] +* link:https://www.robustperception.io/cardinality-is-key[Understanding cardinality in Prometheus] diff --git a/modules/troubleshooting-collector-metrics-cardinality.adoc b/modules/troubleshooting-collector-metrics-cardinality.adoc new file mode 100644 index 000000000000..60aad02a3862 --- /dev/null +++ b/modules/troubleshooting-collector-metrics-cardinality.adoc @@ -0,0 +1,236 @@ +// Module included in the following assemblies: +// +// * observability/logging/log_collection_forwarding/cluster-logging-collector.adoc + +:_mod-docs-content-type: PROCEDURE +[id="troubleshooting-collector-metrics-cardinality_{context}"] += Troubleshooting high collector metrics cardinality + +[role="_abstract"] +If you suspect that Vector collector metrics are causing high cardinality in Prometheus, you can diagnose the issue by analyzing the Prometheus time series database and identifying which `ClusterLogForwarder` configurations contribute most to cardinality. + +.Prerequisites + +* You have access to the cluster as a user with the `cluster-admin` cluster role. +* You have installed the {oc-first}. + +.Procedure + +. Analyze the Prometheus time series database to identify metrics with the highest cardinality: ++ +[source,terminal] +---- +$ oc exec -n openshift-monitoring prometheus-k8s-0 -c prometheus -- \ + /bin/sh -c "promtool tsdb analyze /prometheus" +---- ++ +.Example output (truncated) +[source,terminal] +---- +Block ID: 01ARZ3NDEKTSV4RRFFQ69G5FAV +Duration: 2h0m0s +Series: 142853 +Label names: 156 +Label pairs: 3642847 + +Most common label pairs: +3642847 endpoint=metrics +3166018 namespace=openshift-logging +3116791 container=collector +3105098 app_kubernetes_io_component=collector + +Highest cardinality metric names: +727740 vector_component_received_events_count_bucket +719960 vector_buffer_send_duration_seconds_bucket +166536 etcd_request_duration_seconds_bucket +150240 vector_source_lag_time_seconds_bucket +137720 vector_adaptive_concurrency_limit_bucket +---- ++ +Vector metrics have significantly higher cardinality than other cluster metrics. + +. Count the number of unique `component_id` values for Vector metrics: ++ +[source,terminal] +---- +$ oc exec -n openshift-monitoring prometheus-k8s-0 -- \ + promtool query instant http://localhost:9090 \ + 'count(sum by (component_id)(rate(vector_component_received_events_count_bucket[1m])))' +---- ++ +.Example output +[source,terminal] +---- +484 +---- ++ +This output shows there are 484 unique `component_id` values contributing to cardinality. + +. Count how many of these components are inputs: ++ +[source,terminal] +---- +$ oc exec -n openshift-monitoring prometheus-k8s-0 -- \ + promtool query instant http://localhost:9090 \ + 'count(sum by (component_id)(rate(vector_component_received_events_count_bucket{component_id=~"input_.*"}[1m])))' +---- ++ +.Example output +[source,terminal] +---- +188 +---- ++ +This output shows that 188 of the 484 components are input sources. + +. Identify which `ClusterLogForwarder` instances contribute most to cardinality: ++ +[source,terminal] +---- +$ oc exec -n openshift-monitoring prometheus-k8s-0 -- \ + promtool query instant http://localhost:9090 \ + 'count(sum by (app_kubernetes_io_instance, component_id)(rate(vector_component_received_events_count_bucket[1m]))) by (app_kubernetes_io_instance)' +---- ++ +This query returns the number of components per `ClusterLogForwarder` instance. ++ +.Example output +[source,terminal] +---- +{app_kubernetes_io_instance="instance"} 31 +{app_kubernetes_io_instance="app-logforward-customers"} 453 +---- ++ +This output shows that the `app-logforward-customers` ClusterLogForwarder creates 453 unique components. + +. Retrieve the configuration of the `ClusterLogForwarder` with high component count: ++ +[source,terminal] +---- +$ oc get clusterlogforwarder app-logforward-customers -n openshift-logging -o yaml +---- + +. Count the number of inputs, outputs, and pipelines in the configuration: ++ +[source,terminal] +---- +$ oc get clusterlogforwarder app-logforward-customers -n openshift-logging -o yaml | \ + grep -E "^ inputs:|^ outputs:|^ pipelines:" | wc -l +---- + +. Analyze the configuration to identify opportunities for consolidation: ++ +-- +* Count the number of inputs: ++ +[source,terminal] +---- +$ oc get clusterlogforwarder app-logforward-customers -n openshift-logging -o yaml | \ + yq '.spec.inputs | length' +---- + +* Count the number of outputs: ++ +[source,terminal] +---- +$ oc get clusterlogforwarder app-logforward-customers -n openshift-logging -o yaml | \ + yq '.spec.outputs | length' +---- + +* Count the number of pipelines: ++ +[source,terminal] +---- +$ oc get clusterlogforwarder app-logforward-customers -n openshift-logging -o yaml | \ + yq '.spec.pipelines | length' +---- +-- + +.Troubleshooting + +Based on your diagnosis, apply the appropriate remediation: + +Consolidate inputs by using label selectors:: If you have many inputs that differ only by namespace or pod labels, replace them with a single input that uses label selectors to include all required pods. ++ +For example, instead of separate inputs for `tenant-a`, `tenant-b`, and so on, create one input with: ++ +[source,yaml] +---- +inputs: + - name: all-tenants + type: application + application: + selector: + matchLabels: + tenant-logging: "enabled" +---- ++ +See "Best practices for multitenant logging configurations" for detailed examples. + +Consolidate outputs to the same destination:: If multiple outputs point to the same destination system, combine them into a single output and use multiple pipelines to route different log streams to it. ++ +For example, instead of `tenant-a-splunk` and `tenant-b-splunk` both pointing to `https://splunk.example.com`, create one `shared-splunk` output. + +Reduce the number of pipelines:: Each pipeline adds components for routing. If pipelines differ only in their source namespace or labels, consider: ++ +-- +* Using filters within a single pipeline to route logs +* Relying on destination system capabilities to separate tenant logs +* Combining pipelines that share the same input and output +-- + +Use fewer ClusterLogForwarder instances:: If you have multiple `ClusterLogForwarder` custom resources that could be combined, consider consolidating them into a single instance. Each `ClusterLogForwarder` deploys separate collector pods with their own component sets. ++ +Only use multiple `ClusterLogForwarder` instances when you need: ++ +-- +* Different service accounts for different collection purposes +* Different security or network policies +* Completely different processing pipelines +-- + +Review filter usage:: While filters are necessary for log processing, excessive filtering can add components. Ensure that filters serve a clear purpose and consider whether filtering can be done at the destination instead. + +.Verification + +After making configuration changes to reduce cardinality: + +. Wait approximately 5-10 minutes for the collector to restart and metrics to stabilize. + +. Rerun the diagnostic commands to verify the component count decreased: ++ +[source,terminal] +---- +$ oc exec -n openshift-monitoring prometheus-k8s-0 -- \ + promtool query instant http://localhost:9090 \ + 'count(sum by (component_id)(rate(vector_component_received_events_count_bucket[1m])))' +---- + +. Monitor Prometheus pod resource usage over the next few hours: ++ +[source,terminal] +---- +$ oc adm top pod -n openshift-monitoring -l app.kubernetes.io/name=prometheus +---- ++ +You should see memory usage stabilize or decrease as old time series expire. + +. Check Prometheus storage growth rate using the web console: ++ +-- +.. Navigate to *Observe* -> *Metrics*. +.. Run the following query to see storage size: ++ +[source,promql] +---- +prometheus_tsdb_storage_blocks_bytes +---- +-- + +[role="_additional-resources"] +.Additional resources + +* xref:../configuring/cluster-logging-collector.adoc#collector-metrics-cardinality-impact_cluster-logging-collector[Understanding collector metrics cardinality and monitoring impact] +* xref:../configuring/cluster-logging-collector.adoc#best-practices-multitenant-logging_cluster-logging-collector[Best practices for multitenant logging configurations] +* link:https://prometheus.io/docs/prometheus/latest/storage/[Prometheus storage documentation] +* link:https://access.redhat.com/solutions/7137995[Red Hat Knowledgebase: Prometheus storage size impacted by high cardinality of Vector metrics]