From 05602cb6d56ba3c72a49ed74c783556b371dba3a Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 17 Jul 2026 12:21:00 -0400 Subject: [PATCH 01/17] new labeling strategy LJ --- .github/CODEOWNERS | 1 + .../assets/images/label-multiplication.svg | 90 +++++++++++++++++++ .../assets/images/log-stream-splitting.svg | 60 +++++++++++++ .../assets/images/series-anatomy.svg | 56 ++++++++++++ .../assets/images/unbounded-label-growth.svg | 42 +++++++++ .../business-value/content.json | 22 +++++ .../business-value/manifest.json | 17 ++++ .../business-value/website.yaml | 12 +++ labeling-strategy-lj/content.json | 26 ++++++ .../correlate-dont-inflate/content.json | 30 +++++++ .../correlate-dont-inflate/manifest.json | 17 ++++ .../correlate-dont-inflate/website.yaml | 21 +++++ labeling-strategy-lj/end-journey/content.json | 22 +++++ .../end-journey/manifest.json | 17 ++++ labeling-strategy-lj/end-journey/website.yaml | 21 +++++ .../explore-your-cardinality/content.json | 86 ++++++++++++++++++ .../explore-your-cardinality/manifest.json | 17 ++++ .../explore-your-cardinality/website.yaml | 20 +++++ .../fix-metrics-cardinality/content.json | 81 +++++++++++++++++ .../fix-metrics-cardinality/manifest.json | 17 ++++ .../fix-metrics-cardinality/website.yaml | 20 +++++ .../logs-labeling-strategy/content.json | 39 ++++++++ .../logs-labeling-strategy/manifest.json | 17 ++++ .../logs-labeling-strategy/website.yaml | 20 +++++ labeling-strategy-lj/manifest.json | 44 +++++++++ .../metrics-labeling-strategy/content.json | 39 ++++++++ .../metrics-labeling-strategy/manifest.json | 17 ++++ .../metrics-labeling-strategy/website.yaml | 20 +++++ .../understand-cardinality/content.json | 40 +++++++++ .../understand-cardinality/manifest.json | 17 ++++ .../understand-cardinality/website.yaml | 20 +++++ labeling-strategy-lj/website.yaml | 37 ++++++++ 32 files changed, 1005 insertions(+) create mode 100644 labeling-strategy-lj/assets/images/label-multiplication.svg create mode 100644 labeling-strategy-lj/assets/images/log-stream-splitting.svg create mode 100644 labeling-strategy-lj/assets/images/series-anatomy.svg create mode 100644 labeling-strategy-lj/assets/images/unbounded-label-growth.svg create mode 100644 labeling-strategy-lj/business-value/content.json create mode 100644 labeling-strategy-lj/business-value/manifest.json create mode 100644 labeling-strategy-lj/business-value/website.yaml create mode 100644 labeling-strategy-lj/content.json create mode 100644 labeling-strategy-lj/correlate-dont-inflate/content.json create mode 100644 labeling-strategy-lj/correlate-dont-inflate/manifest.json create mode 100644 labeling-strategy-lj/correlate-dont-inflate/website.yaml create mode 100644 labeling-strategy-lj/end-journey/content.json create mode 100644 labeling-strategy-lj/end-journey/manifest.json create mode 100644 labeling-strategy-lj/end-journey/website.yaml create mode 100644 labeling-strategy-lj/explore-your-cardinality/content.json create mode 100644 labeling-strategy-lj/explore-your-cardinality/manifest.json create mode 100644 labeling-strategy-lj/explore-your-cardinality/website.yaml create mode 100644 labeling-strategy-lj/fix-metrics-cardinality/content.json create mode 100644 labeling-strategy-lj/fix-metrics-cardinality/manifest.json create mode 100644 labeling-strategy-lj/fix-metrics-cardinality/website.yaml create mode 100644 labeling-strategy-lj/logs-labeling-strategy/content.json create mode 100644 labeling-strategy-lj/logs-labeling-strategy/manifest.json create mode 100644 labeling-strategy-lj/logs-labeling-strategy/website.yaml create mode 100644 labeling-strategy-lj/manifest.json create mode 100644 labeling-strategy-lj/metrics-labeling-strategy/content.json create mode 100644 labeling-strategy-lj/metrics-labeling-strategy/manifest.json create mode 100644 labeling-strategy-lj/metrics-labeling-strategy/website.yaml create mode 100644 labeling-strategy-lj/understand-cardinality/content.json create mode 100644 labeling-strategy-lj/understand-cardinality/manifest.json create mode 100644 labeling-strategy-lj/understand-cardinality/website.yaml create mode 100644 labeling-strategy-lj/website.yaml diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 44e0d036..dbc0baf5 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -54,6 +54,7 @@ /k8s-cpu/ @BeverlyJaneJ /k8s-mem/ @BeverlyJaneJ /kafka-monitoring-lj/ @tacole02 +/labeling-strategy-lj/ @Eve832 /linux-server-integration-lj/ @chri2547 /logql-101/ @nicolevanderhoeven /macos-integration-lj/ @chri2547 diff --git a/labeling-strategy-lj/assets/images/label-multiplication.svg b/labeling-strategy-lj/assets/images/label-multiplication.svg new file mode 100644 index 00000000..6fe4d386 --- /dev/null +++ b/labeling-strategy-lj/assets/images/label-multiplication.svg @@ -0,0 +1,90 @@ + + + + + + + + + + + + + Label values multiply into series + + + + status + 200, 404, 500 + 3 values + + × + + + action + GET, POST, PUT, + PATCH, DELETE + 5 values + + = + + + 15 + unique series + + + + Add one more label + + + + status + 3 values + + × + + + action + 5 values + + × + + + endpoint + 3 values + + = + + + 45 + unique series + + One added label tripled the series count. You pay for every one. + diff --git a/labeling-strategy-lj/assets/images/log-stream-splitting.svg b/labeling-strategy-lj/assets/images/log-stream-splitting.svg new file mode 100644 index 00000000..5dd12b1b --- /dev/null +++ b/labeling-strategy-lj/assets/images/log-stream-splitting.svg @@ -0,0 +1,60 @@ + + + + + + + + + + + + + Every label combination splits your logs into another stream + + + + One application + app="api" + + + + + + + + + + {app="api", env="prod", level="info"} + Stream 1 + + + {app="api", env="prod", level="error"} + Stream 2 + + + {app="api", env="dev", level="info"} + Stream 3 + + + {app="api", env="dev", level="error"} + Stream 4 + + Two labels with two values each: four streams, four sets of chunks to store and query. + Each stream is indexed, batched, and stored separately. + diff --git a/labeling-strategy-lj/assets/images/series-anatomy.svg b/labeling-strategy-lj/assets/images/series-anatomy.svg new file mode 100644 index 00000000..372647ca --- /dev/null +++ b/labeling-strategy-lj/assets/images/series-anatomy.svg @@ -0,0 +1,56 @@ + + + + + + + + + + One series or stream per unique label combination + + + + Time series (metrics) + + + http_requests_total{method="GET", status="200", env="prod"} + + + + + + Metric name + Labels: key-value pairs + + Each unique combination of label values is one active series + + + + Log stream (logs) + + + 2026-07-17T10:01:02Z {app="nginx", env="prod"} GET /about + + + + + + + Timestamp + Labels: indexed + Log line: not indexed + + Each unique combination of label values is one log stream + diff --git a/labeling-strategy-lj/assets/images/unbounded-label-growth.svg b/labeling-strategy-lj/assets/images/unbounded-label-growth.svg new file mode 100644 index 00000000..6cd27c28 --- /dev/null +++ b/labeling-strategy-lj/assets/images/unbounded-label-growth.svg @@ -0,0 +1,42 @@ + + + + + + + + + + Bounded and unbounded label values over time + + + + + + + + + Time + Active series + + + + env="prod": bounded + + + + user_id="...": unbounded + + A bounded label stays flat. An unbounded label creates a new series for every new value. + Every unique user, request, or IP address becomes another series you pay for. + diff --git a/labeling-strategy-lj/business-value/content.json b/labeling-strategy-lj/business-value/content.json new file mode 100644 index 00000000..a588e0f1 --- /dev/null +++ b/labeling-strategy-lj/business-value/content.json @@ -0,0 +1,22 @@ +{ + "id": "labeling-strategy-business-value", + "title": "Understand the business value", + "blocks": [ + { + "type": "markdown", + "content": "Labeling decisions are made at instrumentation time but billed at ingestion time, and nothing warns you in between. A team that enables a monitoring integration with all its defaults can multiply its telemetry volume overnight, because every combination of label values creates another unique series or stream to store and pay for." + }, + { + "type": "markdown", + "content": "This pattern is one of the most common causes of unexpected billing spikes: a team with a fixed monthly observability budget turns everything on, sends thousands of metrics it never queries, and exceeds its contract in a fraction of the expected time. The root cause is rarely one bad decision. It's the compounding effect of labels nobody chose deliberately." + }, + { + "type": "markdown", + "content": "A deliberate labeling strategy protects three things:\n\n- **Cost**: fewer unique label combinations means fewer active series and streams to pay for.\n- **Performance**: smaller indexes and fewer streams mean faster queries.\n- **Clarity**: labels chosen on purpose make your telemetry easier to navigate and attribute to teams." + }, + { + "type": "markdown", + "content": "This path gives you the mental model first, then the hands-on tools: you'll see the cardinality math, inspect your own stack, and learn the decision frameworks for metrics and logs." + } + ] +} diff --git a/labeling-strategy-lj/business-value/manifest.json b/labeling-strategy-lj/business-value/manifest.json new file mode 100644 index 00000000..11849259 --- /dev/null +++ b/labeling-strategy-lj/business-value/manifest.json @@ -0,0 +1,17 @@ +{ + "id": "labeling-strategy-business-value", + "type": "guide", + "description": "Learn why a deliberate labeling strategy protects your Grafana Cloud costs, query performance, and telemetry clarity.", + "category": "take-action", + "author": { + "name": "Eve832", + "team": "Grafana Documentation" + }, + "testEnvironment": { + "tier": "cloud" + }, + "depends": [], + "recommends": ["labeling-strategy-understand-cardinality"], + "suggests": [], + "provides": [] +} diff --git a/labeling-strategy-lj/business-value/website.yaml b/labeling-strategy-lj/business-value/website.yaml new file mode 100644 index 00000000..57182987 --- /dev/null +++ b/labeling-strategy-lj/business-value/website.yaml @@ -0,0 +1,12 @@ +menuTitle: Business value +weight: 100 +step: 2 +layout: single-journey +cta: + type: continue +keywords: +- labels +- cardinality +- cost-optimization +- business-value +description: Learn why a deliberate labeling strategy protects your Grafana Cloud costs, query performance, and telemetry clarity. diff --git a/labeling-strategy-lj/content.json b/labeling-strategy-lj/content.json new file mode 100644 index 00000000..3da75b37 --- /dev/null +++ b/labeling-strategy-lj/content.json @@ -0,0 +1,26 @@ +{ + "id": "labeling-strategy-lj", + "title": "Design a labeling strategy to control cardinality", + "blocks": [ + { + "type": "markdown", + "content": "Welcome to the Grafana Cloud learning path that teaches you how to design a labeling strategy that keeps cardinality, cost, and query performance under control.\n\nEvery label you attach to a metric or log multiplies the number of unique series or streams you store and pay for. Most billing surprises in Grafana Cloud trace back to labeling decisions made at instrumentation time, often by accepting defaults without realizing their cost impact. This path teaches you the math behind cardinality, how to inspect your own cardinality, and how to choose labels deliberately for metrics and logs." + }, + { + "type": "markdown", + "content": "## Here's what to expect\n\nWhen you complete this path, you'll be able to:\n\n- Explain how label combinations multiply into unique active series and log streams\n- Inspect your own cardinality with the Cardinality tab in the Cost Management and Billing app\n- Choose metric labels deliberately and avoid unbounded values\n- Reduce existing metrics cardinality with relabeling and Adaptive Metrics\n- Decide when log data belongs in a label, in structured metadata, or in the log line\n- Move high-cardinality context to the right signal instead of deleting it" + }, + { + "type": "markdown", + "content": "## Before you begin\n\nBefore you design your labeling strategy, ensure that you have:\n\n- A Grafana Cloud account. To create an account, refer to [Grafana Cloud](https://grafana.com/signup/cloud/connect-account). A stack that's already receiving metrics and logs shows richer data in the hands-on milestones.\n- The permissions granted by the **Billing and usage reader** role, including cardinality read access. Refer to [Manage access to the Cost Management and Billing app](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/set-up/cost-management-billing-rbac/).\n- Basic familiarity with metrics and logs concepts." + }, + { + "type": "markdown", + "content": "## Troubleshooting\n\nIf you get stuck, we've got your back! Where appropriate, troubleshooting information is just a click away." + }, + { + "type": "markdown", + "content": "## More to explore\n\nWe understand you might want to explore other capabilities not strictly on this path. We'll provide you opportunities where it makes sense." + } + ] +} diff --git a/labeling-strategy-lj/correlate-dont-inflate/content.json b/labeling-strategy-lj/correlate-dont-inflate/content.json new file mode 100644 index 00000000..e61d78d0 --- /dev/null +++ b/labeling-strategy-lj/correlate-dont-inflate/content.json @@ -0,0 +1,30 @@ +{ + "id": "labeling-strategy-correlate-dont-inflate", + "title": "Correlate signals instead of inflating labels", + "blocks": [ + { + "type": "markdown", + "content": "Sometimes you genuinely need a high-cardinality value, like a request ID, when you're debugging. The answer isn't to delete it, and it isn't to put it on a metric label. It's to put the value on the signal designed to hold it, and correlate across signals when you investigate." + }, + { + "type": "markdown", + "content": "## The pattern\n\nWhen a high-cardinality value is attached to a metric label, every unique value multiplies your active series. The same value attached to a log costs a fraction of that, because Loki doesn't index log content:\n\n- **Keep the metric low-cardinality**: the metric answers \"how many\" and \"how fast\" per bounded dimension, such as `service` and `status`.\n- **Put the high-cardinality detail on the correlated log**: attach it as structured metadata, or leave it in the log line. The log answers \"which exact request\".\n- **Correlate at query time**: shared bounded labels, such as `service` and `env`, connect the metric that alerted you to the logs that explain it." + }, + { + "type": "markdown", + "content": "## A worked example\n\nSuppose you track request latency and want to find slow requests for specific users.\n\n- **Instead of** `http_request_duration_seconds{user_id=\"u-48221\", ...}`, which creates a series per user per label combination,\n- **keep the metric as** `http_request_duration_seconds{service=\"checkout\", env=\"prod\"}`,\n- **and log each request** with `user_id` and `request_id` attached as structured metadata.\n\nWhen latency spikes, the metric tells you where and when. Then you query the logs for that service and window, filtering by structured metadata: `{app=\"checkout\"} | user_id=\"u-48221\"`. You keep full debugging detail without paying for it on every series." + }, + { + "type": "markdown", + "content": "## Why this works\n\nEach signal is priced and indexed for a different job. Metrics are cheap per series but multiply by label combinations. Logs carry arbitrary detail, and structured metadata makes high-cardinality fields queryable without indexing them. Putting each value where it belongs is what a labeling strategy actually is: not fewer data points, just the right home for each one." + }, + { + "type": "markdown", + "content": "You now have the complete strategy across both signals. The last milestone recaps the frameworks and points you to what's next." + }, + { + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [What is structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/)\n- [Use telemetry signals together](https://grafana.com/docs/grafana-cloud/telemetry-signals/use-signals-together/key-concepts/)" + } + ] +} diff --git a/labeling-strategy-lj/correlate-dont-inflate/manifest.json b/labeling-strategy-lj/correlate-dont-inflate/manifest.json new file mode 100644 index 00000000..c9023ba1 --- /dev/null +++ b/labeling-strategy-lj/correlate-dont-inflate/manifest.json @@ -0,0 +1,17 @@ +{ + "id": "labeling-strategy-correlate-dont-inflate", + "type": "guide", + "description": "Learn how to keep metrics low-cardinality by moving high-cardinality detail to correlated logs and querying it with structured metadata.", + "category": "take-action", + "author": { + "name": "Eve832", + "team": "Grafana Documentation" + }, + "testEnvironment": { + "tier": "cloud" + }, + "depends": ["labeling-strategy-logs-labeling-strategy"], + "recommends": ["labeling-strategy-end-journey"], + "suggests": [], + "provides": [] +} diff --git a/labeling-strategy-lj/correlate-dont-inflate/website.yaml b/labeling-strategy-lj/correlate-dont-inflate/website.yaml new file mode 100644 index 00000000..b4934c47 --- /dev/null +++ b/labeling-strategy-lj/correlate-dont-inflate/website.yaml @@ -0,0 +1,21 @@ +menuTitle: Correlate, don't inflate +weight: 700 +step: 8 +layout: single-journey +cta: + type: continue +keywords: +- correlation +- metrics +- logs +- structured metadata +- cardinality +side_journeys: + title: More to explore (optional) + heading: 'At this point in your journey, you can explore the following paths:' + items: + - title: What is structured metadata + link: /docs/loki/latest/get-started/labels/structured-metadata/ + - title: Use telemetry signals together + link: /docs/grafana-cloud/telemetry-signals/use-signals-together/key-concepts/ +description: Learn how to keep metrics low-cardinality by moving high-cardinality detail to correlated logs and querying it with structured metadata. diff --git a/labeling-strategy-lj/end-journey/content.json b/labeling-strategy-lj/end-journey/content.json new file mode 100644 index 00000000..256bb2af --- /dev/null +++ b/labeling-strategy-lj/end-journey/content.json @@ -0,0 +1,22 @@ +{ + "id": "labeling-strategy-end-journey", + "title": "Destination reached!", + "blocks": [ + { + "type": "markdown", + "content": "Congratulations on completing this journey! Job well done!" + }, + { + "type": "markdown", + "content": "Throughout this journey, you have:\n\n- Learned how label combinations multiply into unique active series and log streams, and that this is what you're billed for\n- Explored your own stack's cardinality with the Cardinality tab in the Cost Management and Billing app\n- Learned the rules for metric labels: bounded values only, added only when queried, and audited at ingestion setup\n- Reviewed Adaptive Metrics recommendations to reduce cardinality you already have\n- Applied the logs decision framework: labels for origin, structured metadata for high-cardinality fields, filter expressions for the rest\n- Learned to correlate signals so high-cardinality detail lives on logs, not metric labels" + }, + { + "type": "markdown", + "content": "Keep the two questions from this path in your team's review habits: *Is this label value bounded?* and *Will anyone query by it?* Most cardinality problems never happen when someone asks those questions at instrumentation time." + }, + { + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nContinue your cost management journey by exploring:\n\n- [Understand billing and usage](https://grafana.com/docs/learning-paths/billing-usage/)\n- [Optimize metrics with Adaptive Metrics](https://grafana.com/docs/learning-paths/adaptive-metrics/)\n- [Optimize logs with Adaptive Logs](https://grafana.com/docs/learning-paths/adaptive-logs/)" + } + ] +} diff --git a/labeling-strategy-lj/end-journey/manifest.json b/labeling-strategy-lj/end-journey/manifest.json new file mode 100644 index 00000000..c1420904 --- /dev/null +++ b/labeling-strategy-lj/end-journey/manifest.json @@ -0,0 +1,17 @@ +{ + "id": "labeling-strategy-end-journey", + "type": "guide", + "description": "Congratulations on completing the labeling strategy learning path! Discover what you've accomplished and explore related paths.", + "category": "take-action", + "author": { + "name": "Eve832", + "team": "Grafana Documentation" + }, + "testEnvironment": { + "tier": "cloud" + }, + "depends": ["labeling-strategy-correlate-dont-inflate"], + "recommends": [], + "suggests": ["billing-usage-lj", "adaptive-metrics-lj", "adaptive-logs-lj"], + "provides": [] +} diff --git a/labeling-strategy-lj/end-journey/website.yaml b/labeling-strategy-lj/end-journey/website.yaml new file mode 100644 index 00000000..6e7d60a3 --- /dev/null +++ b/labeling-strategy-lj/end-journey/website.yaml @@ -0,0 +1,21 @@ +menuTitle: Destination reached! +weight: 800 +step: 9 +layout: single-journey +cta: + type: conclusion +keywords: +- labels +- cardinality +- journey-completion +side_journeys: + title: More to explore (optional) + heading: 'Continue your cost management journey by exploring:' + items: + - title: Understand billing and usage in Grafana Cloud + link: /docs/learning-paths/billing-usage/ + - title: Optimize metrics with Adaptive Metrics + link: /docs/learning-paths/adaptive-metrics/ + - title: Optimize logs with Adaptive Logs + link: /docs/learning-paths/adaptive-logs/ +description: Congratulations on completing the labeling strategy learning path. Review the decision frameworks you learned and find resources to keep optimizing. diff --git a/labeling-strategy-lj/explore-your-cardinality/content.json b/labeling-strategy-lj/explore-your-cardinality/content.json new file mode 100644 index 00000000..f3cd7b45 --- /dev/null +++ b/labeling-strategy-lj/explore-your-cardinality/content.json @@ -0,0 +1,86 @@ +{ + "id": "labeling-strategy-explore-your-cardinality", + "title": "Explore your own cardinality", + "blocks": [ + { + "type": "markdown", + "content": "The Cardinality tab in the Cost Management and Billing app shows up to 30 days of historical cardinality for your stack, highlights metrics that are new or unused, and links to remediation actions. In this milestone, you'll open the tab and learn to read its tables." + }, + { + "type": "markdown", + "content": "Access to the Cardinality tab requires the `grafana-cmab-app.cardinality:read` permission. Refer to [Manage access to the Cost Management and Billing app](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/set-up/cost-management-billing-rbac/). If your stack is new or isn't receiving metrics yet, the tables will be sparse; the reading skills still apply once data arrives." + }, + { + "type": "markdown", + "content": "In this section, you'll open the Cardinality tab and identify your highest-cardinality metrics and labels." + }, + { + "type": "section", + "id": "read-cardinality-tab", + "title": "Read the Cardinality tab", + "blocks": [ + { + "type": "interactive", + "action": "navigate", + "reftarget": "/a/grafana-cmab-app/usage", + "content": "Open the **Usage** tab of the Cost Management and Billing app.", + "verify": "on-page:/a/grafana-cmab-app/usage" + }, + { + "type": "interactive", + "action": "highlight", + "reftarget": "button[data-testid='data-testid Tab Cardinality']", + "content": "Select the **Cardinality** tab.", + "requirements": ["exists-reftarget", "on-page:/a/grafana-cmab-app/usage"] + }, + { + "type": "interactive", + "action": "noop", + "content": "Review the summary stats at the top: **Unique metrics**, **Total active series**, **Unique labels**, and **Total label value pairs**. **Total active series** is the number your metrics bill is based on." + }, + { + "type": "interactive", + "action": "noop", + "content": "Scroll to the **Highest cardinality metrics** table. The **Series** column shows how many active series each metric produces, and **% Total** shows its share of your stack's series count. Start your investigations here: these metrics contribute the most to cost." + }, + { + "type": "interactive", + "action": "noop", + "content": "Review the **Highest cardinality labels** table. Labels with many distinct values, such as IDs or addresses, are the multiplication factors you learned about in the previous milestone." + }, + { + "type": "interactive", + "action": "noop", + "content": "Check the **Status** column for **New** and **Unused** badges. A **New** badge marks a metric or label first ingested in the last 30 days, which helps you spot recently introduced cardinality. An **Unused** badge marks a metric that isn't used in queries, dashboards, recording rules, or alerting rules, so it may be costing you without providing value.", + "skippable": true + }, + { + "type": "interactive", + "action": "noop", + "content": "Click a metric row to open the details drawer. It shows the metric's cardinality history, a label breakdown, its query count, and the dashboards and rules that reference it. The **Actions** menu links to deeper analysis and remediation, including Adaptive Metrics.", + "skippable": true + }, + { + "type": "markdown", + "content": "You opened the Cardinality tab and identified which metrics and labels drive your active series count." + } + ] + }, + { + "type": "markdown", + "content": "When you read the tables, ask three questions:\n\n- **Which metrics produce the most series?** They're your biggest cost levers.\n- **Which labels have the most values?** They're the multiplication factors to challenge.\n- **Is anything both high-cardinality and unused?** That's cost with no value, and the easiest win." + }, + { + "type": "markdown", + "content": "For a complete tour of the Cost Management and Billing app, including usage breakdowns and billing alerts, take the [Understand billing and usage path](https://grafana.com/docs/learning-paths/billing-usage/)." + }, + { + "type": "markdown", + "content": "In the next milestone, you'll learn how to choose metric labels so this table never becomes a problem." + }, + { + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Analyze historical metrics cardinality](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/metrics-costs/prometheus-metrics-costs/historical-cardinality/)\n- [Live cardinality management dashboards](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/metrics-costs/prometheus-metrics-costs/cardinality-management/)" + } + ] +} diff --git a/labeling-strategy-lj/explore-your-cardinality/manifest.json b/labeling-strategy-lj/explore-your-cardinality/manifest.json new file mode 100644 index 00000000..d2c6259d --- /dev/null +++ b/labeling-strategy-lj/explore-your-cardinality/manifest.json @@ -0,0 +1,17 @@ +{ + "id": "labeling-strategy-explore-your-cardinality", + "type": "guide", + "description": "Learn how to read the Cardinality tab in the Cost Management and Billing app to find the metrics and labels that drive your active series count.", + "category": "take-action", + "author": { + "name": "Eve832", + "team": "Grafana Documentation" + }, + "testEnvironment": { + "tier": "cloud" + }, + "depends": ["labeling-strategy-understand-cardinality"], + "recommends": ["labeling-strategy-metrics-labeling-strategy"], + "suggests": [], + "provides": [] +} diff --git a/labeling-strategy-lj/explore-your-cardinality/website.yaml b/labeling-strategy-lj/explore-your-cardinality/website.yaml new file mode 100644 index 00000000..edc907ec --- /dev/null +++ b/labeling-strategy-lj/explore-your-cardinality/website.yaml @@ -0,0 +1,20 @@ +menuTitle: Explore your cardinality +weight: 300 +step: 4 +layout: single-journey +cta: + type: continue +keywords: +- cardinality +- cost-management +- metrics +- active series +side_journeys: + title: More to explore (optional) + heading: 'At this point in your journey, you can explore the following paths:' + items: + - title: Analyze historical metrics cardinality + link: /docs/grafana-cloud/cost-management-and-billing/analyze-costs/metrics-costs/prometheus-metrics-costs/historical-cardinality/ + - title: Live cardinality management dashboards + link: /docs/grafana-cloud/cost-management-and-billing/analyze-costs/metrics-costs/prometheus-metrics-costs/cardinality-management/ +description: Learn how to read the Cardinality tab in the Cost Management and Billing app to find the metrics and labels that drive your active series count. diff --git a/labeling-strategy-lj/fix-metrics-cardinality/content.json b/labeling-strategy-lj/fix-metrics-cardinality/content.json new file mode 100644 index 00000000..380912b4 --- /dev/null +++ b/labeling-strategy-lj/fix-metrics-cardinality/content.json @@ -0,0 +1,81 @@ +{ + "id": "labeling-strategy-fix-metrics-cardinality", + "title": "Reduce metrics cardinality you already have", + "blocks": [ + { + "type": "markdown", + "content": "If the Cardinality tab already shows high-cardinality or unused metrics, you don't have to re-instrument anything to fix it. Adaptive Metrics reduces cardinality from inside Grafana Cloud by aggregating away the labels nobody uses, while keeping the metric itself." + }, + { + "type": "markdown", + "content": "## How Adaptive Metrics helps\n\nAdaptive Metrics analyzes how your organization uses its metrics across dashboards, alerts, recording rules, and queries. It then recommends aggregating underutilized metrics into lower-cardinality versions. Applying a recommendation doesn't delete the metric: it aggregates away unused label dimensions, and your existing dashboards, rules, and queries continue to work without modification. If usage changes later, you can remove the aggregation and the metric is stored at full cardinality again.\n\nThink of it as enforcement at the door: unnecessary series are stopped before they're persisted, without touching your Alloy or collector configuration." + }, + { + "type": "markdown", + "content": "In this section, you'll open Adaptive Metrics and review its recommendations for your stack." + }, + { + "type": "section", + "id": "review-adaptive-metrics", + "title": "Review your recommendations", + "blocks": [ + { + "type": "interactive", + "action": "highlight", + "reftarget": "[aria-label=\"Expand section: Adaptive Telemetry\"]", + "content": "In the navigation menu, expand **Adaptive Telemetry**. If the section is already expanded, skip this step.", + "requirements": ["navmenu-open", "exists-reftarget"], + "skippable": true + }, + { + "type": "multistep", + "content": "Navigate to **Adaptive Metrics** > **Metrics**.", + "requirements": ["navmenu-open"], + "steps": [ + { + "action": "highlight", + "reftarget": "a[href=\"/a/grafana-adaptive-metrics-app/overview\"]", + "tooltip": "Click Adaptive Metrics" + }, + { + "action": "highlight", + "reftarget": "a[href=\"/a/grafana-adaptive-metrics-app/rule-management\"]", + "tooltip": "Click Metrics" + } + ] + }, + { + "type": "interactive", + "action": "noop", + "content": "Review the recommendations list. Each recommendation shows which metric it affects and which labels it proposes to aggregate away, based on how the metric is actually used." + }, + { + "type": "interactive", + "action": "noop", + "content": "Expand a recommendation to review its details before applying it, including the recommended rule information and the metric's usage stats.", + "skippable": true + }, + { + "type": "markdown", + "content": "You opened Adaptive Metrics and reviewed the aggregation recommendations for your stack." + } + ] + }, + { + "type": "markdown", + "content": "A good remediation order:\n\n1. **Unused metrics first**: metrics with the **Unused** badge in the Cardinality tab are cost with no value.\n2. **High-cardinality labels on used metrics next**: aggregate away the label dimensions nobody queries, and keep the ones they do.\n3. **Protect what matters**: define exemptions for metrics that must never be aggregated." + }, + { + "type": "markdown", + "content": "For the complete Adaptive Metrics workflow, including rules, exemptions, segmentation, and auto-apply, take the [Optimize metrics with Adaptive Metrics path](https://grafana.com/docs/learning-paths/adaptive-metrics/)." + }, + { + "type": "markdown", + "content": "In the next milestone, you'll switch signals and apply labeling strategy to logs." + }, + { + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Introduction to Adaptive Metrics](https://grafana.com/docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/introduction/)\n- [Manage recommendations](https://grafana.com/docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/manage-recommendations/)" + } + ] +} diff --git a/labeling-strategy-lj/fix-metrics-cardinality/manifest.json b/labeling-strategy-lj/fix-metrics-cardinality/manifest.json new file mode 100644 index 00000000..c3c476ef --- /dev/null +++ b/labeling-strategy-lj/fix-metrics-cardinality/manifest.json @@ -0,0 +1,17 @@ +{ + "id": "labeling-strategy-fix-metrics-cardinality", + "type": "guide", + "description": "Learn how to reduce existing metrics cardinality with Adaptive Metrics aggregation recommendations, without touching your ingestion pipeline.", + "category": "take-action", + "author": { + "name": "Eve832", + "team": "Grafana Documentation" + }, + "testEnvironment": { + "tier": "cloud" + }, + "depends": ["labeling-strategy-metrics-labeling-strategy"], + "recommends": ["labeling-strategy-logs-labeling-strategy"], + "suggests": ["adaptive-metrics-lj"], + "provides": [] +} diff --git a/labeling-strategy-lj/fix-metrics-cardinality/website.yaml b/labeling-strategy-lj/fix-metrics-cardinality/website.yaml new file mode 100644 index 00000000..3aaf9fdc --- /dev/null +++ b/labeling-strategy-lj/fix-metrics-cardinality/website.yaml @@ -0,0 +1,20 @@ +menuTitle: Fix metrics cardinality +weight: 500 +step: 6 +layout: single-journey +cta: + type: continue +keywords: +- adaptive metrics +- cardinality +- aggregation +- cost-reduction +side_journeys: + title: More to explore (optional) + heading: 'At this point in your journey, you can explore the following paths:' + items: + - title: Introduction to Adaptive Metrics + link: /docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/introduction/ + - title: Manage recommendations + link: /docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/manage-recommendations/ +description: Learn how to reduce existing metrics cardinality with Adaptive Metrics aggregation recommendations, without touching your ingestion pipeline. diff --git a/labeling-strategy-lj/logs-labeling-strategy/content.json b/labeling-strategy-lj/logs-labeling-strategy/content.json new file mode 100644 index 00000000..5502e945 --- /dev/null +++ b/labeling-strategy-lj/logs-labeling-strategy/content.json @@ -0,0 +1,39 @@ +{ + "id": "labeling-strategy-logs-labeling-strategy", + "title": "Apply labeling strategy to logs", + "blocks": [ + { + "type": "markdown", + "content": "Loki takes the fewest-labels principle further than metrics: it indexes only your labels, not the log content, so every label combination splits your logs into another stream to index and store. This milestone gives you the decision framework for what belongs in a label, what belongs in structured metadata, and what stays in the log line." + }, + { + "type": "markdown", + "content": "## Labels tell you where\n\nGood log labels describe the origin or context of the log: application, namespace, environment, cluster, region. These are static, bounded, and what you actually filter by when you start a search. Loki performs best with few labels, and the fewer streams you create, the smaller the index and the faster your queries." + }, + { + "type": "image", + "src": "https://grafana.com/media/docs/learning-journey/labeling-strategy/log-stream-splitting.svg", + "alt": "One application splitting into four log streams because two labels with two values each create four unique label combinations" + }, + { + "type": "markdown", + "content": "## Use dynamic labels sparingly\n\nA dynamic label takes its value from the log content, such as `level`. Use them rarely, and only when all of these are true:\n\n- The value set is low-cardinality, ideally tens of values.\n- The values are long-lived, such as the first segment of an HTTP path, not ephemeral IDs.\n- Users actually query by the label frequently.\n\nOtherwise, don't index it. Loki's filter expressions are fast: `{app=\"api\"} |= \"level=error\"` performs comparably to a `level` label for many workloads, without splitting your streams. Don't add a label until you know you need it." + }, + { + "type": "markdown", + "content": "## The decision framework\n\nFor any piece of log data, ask where it belongs:\n\n- **You filter by it in almost every query, and it's bounded**: make it a label. Example: `env`, `app`.\n- **It's high-cardinality but you query it often**: attach it as [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/). It's queryable with a label filter expression, such as `{job=\"example\"} | pod=\"myservice-abc1234-56789\"`, without being indexed or creating streams. Examples: pod names, process IDs, trace IDs.\n- **Everything else**: leave it in the log line and extract it at query time with parsers like `json` or `logfmt`, or search it with filter expressions." + }, + { + "type": "markdown", + "content": "## Audit what your clients send\n\nCollectors and agents can attach dynamic labels you didn't choose. Use the fewest-labels test on whatever Alloy, Fluentd, or your Docker driver is applying. To see your streams and find high-cardinality labels, run:\n\n```\nlogcli series '{}' --since=1h --analyze-labels\n```\n\nA label with thousands of unique values found in nearly every stream, such as a `requestId`, should be removed from labels and queried with a filter expression instead." + }, + { + "type": "markdown", + "content": "You now have the logs framework: static origin labels, dynamic labels only when bounded and queried, structured metadata for high-cardinality fields, and filter expressions for the rest." + }, + { + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/)\n- [What is structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/)" + } + ] +} diff --git a/labeling-strategy-lj/logs-labeling-strategy/manifest.json b/labeling-strategy-lj/logs-labeling-strategy/manifest.json new file mode 100644 index 00000000..0896895d --- /dev/null +++ b/labeling-strategy-lj/logs-labeling-strategy/manifest.json @@ -0,0 +1,17 @@ +{ + "id": "labeling-strategy-logs-labeling-strategy", + "type": "guide", + "description": "Learn the decision framework for log data in Loki, and when a value belongs in a label, in structured metadata, or in the log line.", + "category": "take-action", + "author": { + "name": "Eve832", + "team": "Grafana Documentation" + }, + "testEnvironment": { + "tier": "cloud" + }, + "depends": ["labeling-strategy-fix-metrics-cardinality"], + "recommends": ["labeling-strategy-correlate-dont-inflate"], + "suggests": [], + "provides": [] +} diff --git a/labeling-strategy-lj/logs-labeling-strategy/website.yaml b/labeling-strategy-lj/logs-labeling-strategy/website.yaml new file mode 100644 index 00000000..03ac2700 --- /dev/null +++ b/labeling-strategy-lj/logs-labeling-strategy/website.yaml @@ -0,0 +1,20 @@ +menuTitle: Logs labeling strategy +weight: 600 +step: 7 +layout: single-journey +cta: + type: continue +keywords: +- logs +- loki +- labels +- structured metadata +side_journeys: + title: More to explore (optional) + heading: 'At this point in your journey, you can explore the following paths:' + items: + - title: Label best practices + link: /docs/loki/latest/get-started/labels/bp-labels/ + - title: What is structured metadata + link: /docs/loki/latest/get-started/labels/structured-metadata/ +description: Learn the decision framework for log data in Loki, and when a value belongs in a label, in structured metadata, or in the log line. diff --git a/labeling-strategy-lj/manifest.json b/labeling-strategy-lj/manifest.json new file mode 100644 index 00000000..96e02cd9 --- /dev/null +++ b/labeling-strategy-lj/manifest.json @@ -0,0 +1,44 @@ +{ + "id": "labeling-strategy-lj", + "type": "path", + "description": "A learning path that teaches you how to design a labeling strategy that controls cardinality, keeps costs predictable, and puts high-cardinality data on the right telemetry signal.", + "category": "take-action", + "author": { + "name": "Eve832", + "team": "Grafana Documentation" + }, + "startingLocation": "/a/grafana-cmab-app", + "targeting": { + "match": { + "and": [ + { + "urlPrefixIn": [ + "/a/grafana-cmab-app", + "/a/grafana-cmab-app/usage", + "/a/grafana-adaptive-metrics-app/overview", + "/a/grafana-adaptive-metrics-app/rule-management" + ] + }, + { + "targetPlatform": "cloud" + } + ] + } + }, + "testEnvironment": { + "tier": "cloud" + }, + "milestones": [ + "labeling-strategy-understand-cardinality", + "labeling-strategy-explore-your-cardinality", + "labeling-strategy-metrics-labeling-strategy", + "labeling-strategy-fix-metrics-cardinality", + "labeling-strategy-logs-labeling-strategy", + "labeling-strategy-correlate-dont-inflate", + "labeling-strategy-end-journey" + ], + "depends": [], + "recommends": [], + "suggests": ["billing-usage-lj", "adaptive-metrics-lj", "adaptive-logs-lj"], + "provides": [] +} diff --git a/labeling-strategy-lj/metrics-labeling-strategy/content.json b/labeling-strategy-lj/metrics-labeling-strategy/content.json new file mode 100644 index 00000000..4ff78ab4 --- /dev/null +++ b/labeling-strategy-lj/metrics-labeling-strategy/content.json @@ -0,0 +1,39 @@ +{ + "id": "labeling-strategy-metrics-labeling-strategy", + "title": "Choose metric labels deliberately", + "blocks": [ + { + "type": "markdown", + "content": "A metric label should be a dimension you aggregate or filter by, with a bounded set of values. This milestone gives you the rules for choosing metric labels and shows where cardinality mistakes usually enter a stack." + }, + { + "type": "markdown", + "content": "## Bounded or it doesn't belong\n\nThe single most important rule: **label values must be bounded**. A label like `env` has a handful of values forever. A label like `user_id` grows without limit, and every new value creates another active series that's billed and indexed.\n\nNever use these as metric label values:\n\n- User, customer, or session IDs\n- Request or trace IDs\n- IP addresses\n- Email addresses\n- Timestamps\n\nEach of these creates a new series per unique value, and the combinations multiply against every other label on the metric." + }, + { + "type": "image", + "src": "https://grafana.com/media/docs/learning-journey/labeling-strategy/unbounded-label-growth.svg", + "alt": "A chart comparing active series over time: a bounded label like env stays flat while an unbounded label like user_id grows without limit" + }, + { + "type": "markdown", + "content": "## Ask before you add\n\nFor every label, ask:\n\n- **Will I aggregate or filter by it?** If nobody queries by this dimension, it's pure cost.\n- **Is the value set bounded?** Think tens of values, not thousands.\n- **Does it multiply badly?** Check the label against the labels already on the metric. Three bounded labels can still combine into thousands of series." + }, + { + "type": "markdown", + "content": "## Where the mistakes happen\n\nCardinality problems usually enter a stack at ingestion setup, not in application code reviews. Enabling an integration with default settings, such as the Kubernetes Monitoring Helm chart across all your clusters, can turn on thousands of metrics and labels at once. That visibility is valuable, but only if your budget expects it. When you set up a new integration:\n\n- Review what metrics the defaults send before pointing them at production.\n- Use allowlists to keep only the metrics and labels you'll use. To reduce what Kubernetes Monitoring collects, refer to [Manage your configuration](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/manage-configuration/).\n- Check the Cardinality tab after enabling anything new, and watch for **New** badges." + }, + { + "type": "markdown", + "content": "## Filter at the source\n\nYou can keep unnecessary series from ever leaving your infrastructure with relabeling rules in Prometheus, Alloy, or your collector:\n\n- **Allowlisting**: keep an explicit set of important metrics and labels and drop everything else, using `keep` and `labelkeep` actions.\n- **Denylisting**: drop an explicit set of high-cardinality metrics and labels and keep everything else, using `drop` and `labeldrop` actions.\n\nFor the full relabeling reference and examples, refer to [Reduce metrics costs by filtering collected and forwarded metrics](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/client-side-filtering/)." + }, + { + "type": "markdown", + "content": "You now have the prevention rules: bounded values only, add labels only when queried, and audit defaults at ingestion setup. Next, you'll fix cardinality you already have." + }, + { + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Client-side filtering](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/client-side-filtering/)\n- [Reduce costs by adjusting data points per minute](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/adjust-data-points-per-minute/)" + } + ] +} diff --git a/labeling-strategy-lj/metrics-labeling-strategy/manifest.json b/labeling-strategy-lj/metrics-labeling-strategy/manifest.json new file mode 100644 index 00000000..51075667 --- /dev/null +++ b/labeling-strategy-lj/metrics-labeling-strategy/manifest.json @@ -0,0 +1,17 @@ +{ + "id": "labeling-strategy-metrics-labeling-strategy", + "type": "guide", + "description": "Learn the rules for choosing metric labels, the unbounded values to avoid, and how to prevent cardinality problems at ingestion setup.", + "category": "take-action", + "author": { + "name": "Eve832", + "team": "Grafana Documentation" + }, + "testEnvironment": { + "tier": "cloud" + }, + "depends": ["labeling-strategy-explore-your-cardinality"], + "recommends": ["labeling-strategy-fix-metrics-cardinality"], + "suggests": [], + "provides": [] +} diff --git a/labeling-strategy-lj/metrics-labeling-strategy/website.yaml b/labeling-strategy-lj/metrics-labeling-strategy/website.yaml new file mode 100644 index 00000000..f7eeb11b --- /dev/null +++ b/labeling-strategy-lj/metrics-labeling-strategy/website.yaml @@ -0,0 +1,20 @@ +menuTitle: Metrics labeling strategy +weight: 400 +step: 5 +layout: single-journey +cta: + type: continue +keywords: +- metrics +- labels +- cardinality +- relabeling +side_journeys: + title: More to explore (optional) + heading: 'At this point in your journey, you can explore the following paths:' + items: + - title: Client-side filtering + link: /docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/client-side-filtering/ + - title: Reduce costs by adjusting data points per minute + link: /docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/adjust-data-points-per-minute/ +description: Learn the rules for choosing metric labels, the unbounded values to avoid, and how to prevent cardinality problems at ingestion setup. diff --git a/labeling-strategy-lj/understand-cardinality/content.json b/labeling-strategy-lj/understand-cardinality/content.json new file mode 100644 index 00000000..e2f59d94 --- /dev/null +++ b/labeling-strategy-lj/understand-cardinality/content.json @@ -0,0 +1,40 @@ +{ + "id": "labeling-strategy-understand-cardinality", + "title": "Understand cardinality and active series", + "blocks": [ + { + "type": "markdown", + "content": "Cardinality is the number of distinct values an attribute can have. In observability, it determines how many unique series and streams your labels create, and that number is what you're billed for. This milestone gives you the mental model that everything else in this path builds on." + }, + { + "type": "markdown", + "content": "## What you're actually billed for\n\nA common misconception is that metrics billing is per metric name. It isn't. A time series is a metric name plus one unique combination of label key-value pairs, and each series that receives data within the last 20 minutes counts as an **active series**. Grafana Cloud bills metrics on the 95th percentile of your active series count over the billing period, so one metric name with many label combinations can cost far more than dozens of well-labeled metrics.\n\nLogs work the same way structurally: each unique combination of label values defines a **log stream**, and Loki indexes, batches, and stores each stream separately." + }, + { + "type": "image", + "src": "https://grafana.com/media/docs/learning-journey/labeling-strategy/series-anatomy.svg", + "alt": "Anatomy of a time series and a log stream, showing that each unique combination of label values creates one series or stream" + }, + { + "type": "markdown", + "content": "## The multiplication\n\nLabels don't add series, they multiply them. A `status` label with 3 values combined with an `action` label with 5 values produces 15 unique series from what looks like one metric. Add an `endpoint` label with 3 values and you're at 45. Every label you add multiplies the count by that label's number of values." + }, + { + "type": "image", + "src": "https://grafana.com/media/docs/learning-journey/labeling-strategy/label-multiplication.svg", + "alt": "Two rows showing label value counts multiplying: 3 status values times 5 action values equals 15 series, and adding a 3-value endpoint label grows the total to 45 series" + }, + { + "type": "markdown", + "content": "## Why this hurts twice\n\nHigh cardinality doesn't only raise your bill:\n\n- **Cost**: every unique series and stream is billed, whether or not anyone queries it.\n- **Performance**: for logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." + }, + { + "type": "markdown", + "content": "You now have the core model: labels multiply into series and streams, and you pay for every unique combination. Next, you'll look at your own stack's cardinality." + }, + { + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Understand metrics usage and cost](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/understand-usage-cost/metrics/)\n- [Cardinality in Loki](https://grafana.com/docs/loki/latest/get-started/labels/cardinality/)" + } + ] +} diff --git a/labeling-strategy-lj/understand-cardinality/manifest.json b/labeling-strategy-lj/understand-cardinality/manifest.json new file mode 100644 index 00000000..fad14d6c --- /dev/null +++ b/labeling-strategy-lj/understand-cardinality/manifest.json @@ -0,0 +1,17 @@ +{ + "id": "labeling-strategy-understand-cardinality", + "type": "guide", + "description": "Learn how label combinations multiply into unique active series and log streams, and why that number drives your Grafana Cloud bill.", + "category": "take-action", + "author": { + "name": "Eve832", + "team": "Grafana Documentation" + }, + "testEnvironment": { + "tier": "cloud" + }, + "depends": [], + "recommends": ["labeling-strategy-explore-your-cardinality"], + "suggests": [], + "provides": [] +} diff --git a/labeling-strategy-lj/understand-cardinality/website.yaml b/labeling-strategy-lj/understand-cardinality/website.yaml new file mode 100644 index 00000000..fb6c23b5 --- /dev/null +++ b/labeling-strategy-lj/understand-cardinality/website.yaml @@ -0,0 +1,20 @@ +menuTitle: Understand cardinality +weight: 200 +step: 3 +layout: single-journey +cta: + type: continue +keywords: +- cardinality +- active series +- log streams +- labels +side_journeys: + title: More to explore (optional) + heading: 'At this point in your journey, you can explore the following paths:' + items: + - title: Understand metrics usage and cost + link: /docs/grafana-cloud/cost-management-and-billing/understand-usage-cost/metrics/ + - title: Cardinality in Loki + link: /docs/loki/latest/get-started/labels/cardinality/ +description: Learn how label combinations multiply into unique active series and log streams, and why that number drives your Grafana Cloud bill. diff --git a/labeling-strategy-lj/website.yaml b/labeling-strategy-lj/website.yaml new file mode 100644 index 00000000..dfcac47a --- /dev/null +++ b/labeling-strategy-lj/website.yaml @@ -0,0 +1,37 @@ +menuTitle: Labeling strategy +weight: 1250 +journey: + group: take-action + skill: Beginner + source: Docs & blog posts + logo: + src: /static/img/menu/grafana2.svg + background: '#FFFFFF' + width: 46 + height: 55 +step: 1 +layout: single-journey +cascade: + layout: single-journey +cta: + type: start + title: Are you ready? + cta_text: Let's go! +keywords: +- labels +- cardinality +- active series +- cost-management +- metrics +- logs +description: Learn how to design a labeling strategy that controls cardinality, keeps your Grafana Cloud costs predictable, and puts high-cardinality data on the right telemetry signal. +related_journeys: + title: Related paths + heading: 'Consider taking the following paths after you complete this journey.' + items: + - title: Understand billing and usage in Grafana Cloud + link: /docs/learning-paths/billing-usage/ + - title: Optimize metrics with Adaptive Metrics + link: /docs/learning-paths/adaptive-metrics/ + - title: Optimize logs with Adaptive Logs + link: /docs/learning-paths/adaptive-logs/ From 7a5fa6e004697441f4d80bc7da290a97aad7b126 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 17 Jul 2026 13:45:48 -0400 Subject: [PATCH 02/17] Add gateway collector enforcement pattern via Fleet Management When many teams send telemetry and the bill payer can't control every client, policy belongs at an admin-owned gateway collector tier managed centrally with Fleet Management. Adds this guidance to the metrics milestone and cross-references it from the logs milestone. --- labeling-strategy-lj/logs-labeling-strategy/content.json | 2 +- labeling-strategy-lj/metrics-labeling-strategy/content.json | 6 +++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/labeling-strategy-lj/logs-labeling-strategy/content.json b/labeling-strategy-lj/logs-labeling-strategy/content.json index 5502e945..f9631264 100644 --- a/labeling-strategy-lj/logs-labeling-strategy/content.json +++ b/labeling-strategy-lj/logs-labeling-strategy/content.json @@ -25,7 +25,7 @@ }, { "type": "markdown", - "content": "## Audit what your clients send\n\nCollectors and agents can attach dynamic labels you didn't choose. Use the fewest-labels test on whatever Alloy, Fluentd, or your Docker driver is applying. To see your streams and find high-cardinality labels, run:\n\n```\nlogcli series '{}' --since=1h --analyze-labels\n```\n\nA label with thousands of unique values found in nearly every stream, such as a `requestId`, should be removed from labels and queried with a filter expression instead." + "content": "## Audit what your clients send\n\nCollectors and agents can attach dynamic labels you didn't choose. Use the fewest-labels test on whatever Alloy, Fluentd, or your Docker driver is applying. To see your streams and find high-cardinality labels, run:\n\n```\nlogcli series '{}' --since=1h --analyze-labels\n```\n\nA label with thousands of unique values found in nearly every stream, such as a `requestId`, should be removed from labels and queried with a filter expression instead.\n\nIf many teams send logs and you can't audit or change every client, apply the same approach you learned for metrics: route logs through gateway collectors your admin team owns, enforce label policy at that layer, and manage those gateway configurations centrally with [Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/)." }, { "type": "markdown", diff --git a/labeling-strategy-lj/metrics-labeling-strategy/content.json b/labeling-strategy-lj/metrics-labeling-strategy/content.json index 4ff78ab4..5915c73b 100644 --- a/labeling-strategy-lj/metrics-labeling-strategy/content.json +++ b/labeling-strategy-lj/metrics-labeling-strategy/content.json @@ -29,7 +29,11 @@ }, { "type": "markdown", - "content": "You now have the prevention rules: bounded values only, add labels only when queried, and audit defaults at ingestion setup. Next, you'll fix cardinality you already have." + "content": "## Enforce your strategy at a gateway\n\nRelabeling rules only help if you control the collectors that apply them. In larger organizations, many teams send telemetry, and the team that pays the bill often can't change every client. In that case, route telemetry through a tier of gateway collectors that your platform or admin team owns, and apply your labeling and filtering policies at that layer. Whatever upstream teams send, your rules run at the gateway before data reaches Grafana Cloud.\n\n[Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/) helps you run this pattern at scale: it gives you centralized control over your collector configurations, so you can roll out and update gateway policies from one place instead of editing collectors one by one." + }, + { + "type": "markdown", + "content": "You now have the prevention rules: bounded values only, add labels only when queried, audit defaults at ingestion setup, and enforce policy at a layer you control. Next, you'll fix cardinality you already have." }, { "type": "markdown", From d0e91efc2d78df352da72d479865d4e91323aac5 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Tue, 21 Jul 2026 10:06:06 -0400 Subject: [PATCH 03/17] content review --- .../assets/images/unbounded-label-growth.svg | 2 +- .../business-value/content.json | 41 +++--- .../correlate-dont-inflate/content.json | 57 ++++---- .../explore-your-cardinality/content.json | 134 +++++++++--------- .../fix-metrics-cardinality/content.json | 133 ++++++++--------- .../logs-labeling-strategy/content.json | 75 +++++----- .../metrics-labeling-strategy/content.json | 83 ++++++----- .../understand-cardinality/content.json | 77 +++++----- 8 files changed, 292 insertions(+), 310 deletions(-) diff --git a/labeling-strategy-lj/assets/images/unbounded-label-growth.svg b/labeling-strategy-lj/assets/images/unbounded-label-growth.svg index 6cd27c28..ee212974 100644 --- a/labeling-strategy-lj/assets/images/unbounded-label-growth.svg +++ b/labeling-strategy-lj/assets/images/unbounded-label-growth.svg @@ -32,7 +32,7 @@ - user_id="...": unbounded **Metrics**.", - "requirements": ["navmenu-open"], - "steps": [ - { - "action": "highlight", - "reftarget": "a[href=\"/a/grafana-adaptive-metrics-app/overview\"]", - "tooltip": "Click Adaptive Metrics" - }, - { - "action": "highlight", - "reftarget": "a[href=\"/a/grafana-adaptive-metrics-app/rule-management\"]", - "tooltip": "Click Metrics" - } - ] + "type": "markdown", + "content": "A good remediation order:\n\n1. **Unused metrics first**: metrics with the **Unused** badge in the Cardinality tab are cost with no value.\n2. **High-cardinality labels on used metrics next**: aggregate away the label dimensions nobody queries, and keep the ones they do.\n3. **Protect what matters**: define exemptions for metrics that must never be aggregated." }, { - "type": "interactive", - "action": "noop", - "content": "Review the recommendations list. Each recommendation shows which metric it affects and which labels it proposes to aggregate away, based on how the metric is actually used." + "type": "markdown", + "content": "For the complete Adaptive Metrics workflow, including rules, exemptions, segmentation, and auto-apply, take the [Optimize metrics with Adaptive Metrics path](https://grafana.com/docs/learning-paths/adaptive-metrics/)." }, { - "type": "interactive", - "action": "noop", - "content": "Expand a recommendation to review its details before applying it, including the recommended rule information and the metric's usage stats.", - "skippable": true + "type": "markdown", + "content": "In the next milestone, you'll switch signals and apply labeling strategy to logs." }, { - "type": "markdown", - "content": "You opened Adaptive Metrics and reviewed the aggregation recommendations for your stack." + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Introduction to Adaptive Metrics](https://grafana.com/docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/introduction/)\n- [Manage recommendations](https://grafana.com/docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/manage-recommendations/)" } - ] - }, - { - "type": "markdown", - "content": "A good remediation order:\n\n1. **Unused metrics first**: metrics with the **Unused** badge in the Cardinality tab are cost with no value.\n2. **High-cardinality labels on used metrics next**: aggregate away the label dimensions nobody queries, and keep the ones they do.\n3. **Protect what matters**: define exemptions for metrics that must never be aggregated." - }, - { - "type": "markdown", - "content": "For the complete Adaptive Metrics workflow, including rules, exemptions, segmentation, and auto-apply, take the [Optimize metrics with Adaptive Metrics path](https://grafana.com/docs/learning-paths/adaptive-metrics/)." - }, - { - "type": "markdown", - "content": "In the next milestone, you'll switch signals and apply labeling strategy to logs." - }, - { - "type": "markdown", - "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Introduction to Adaptive Metrics](https://grafana.com/docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/introduction/)\n- [Manage recommendations](https://grafana.com/docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/manage-recommendations/)" - } - ] -} + ] +} \ No newline at end of file diff --git a/labeling-strategy-lj/logs-labeling-strategy/content.json b/labeling-strategy-lj/logs-labeling-strategy/content.json index f9631264..ec2d993a 100644 --- a/labeling-strategy-lj/logs-labeling-strategy/content.json +++ b/labeling-strategy-lj/logs-labeling-strategy/content.json @@ -1,39 +1,38 @@ { - "id": "labeling-strategy-logs-labeling-strategy", - "title": "Apply labeling strategy to logs", - "blocks": [ - { - "type": "markdown", - "content": "Loki takes the fewest-labels principle further than metrics: it indexes only your labels, not the log content, so every label combination splits your logs into another stream to index and store. This milestone gives you the decision framework for what belongs in a label, what belongs in structured metadata, and what stays in the log line." - }, - { - "type": "markdown", - "content": "## Labels tell you where\n\nGood log labels describe the origin or context of the log: application, namespace, environment, cluster, region. These are static, bounded, and what you actually filter by when you start a search. Loki performs best with few labels, and the fewer streams you create, the smaller the index and the faster your queries." - }, - { - "type": "image", - "src": "https://grafana.com/media/docs/learning-journey/labeling-strategy/log-stream-splitting.svg", - "alt": "One application splitting into four log streams because two labels with two values each create four unique label combinations" - }, - { - "type": "markdown", - "content": "## Use dynamic labels sparingly\n\nA dynamic label takes its value from the log content, such as `level`. Use them rarely, and only when all of these are true:\n\n- The value set is low-cardinality, ideally tens of values.\n- The values are long-lived, such as the first segment of an HTTP path, not ephemeral IDs.\n- Users actually query by the label frequently.\n\nOtherwise, don't index it. Loki's filter expressions are fast: `{app=\"api\"} |= \"level=error\"` performs comparably to a `level` label for many workloads, without splitting your streams. Don't add a label until you know you need it." - }, - { - "type": "markdown", - "content": "## The decision framework\n\nFor any piece of log data, ask where it belongs:\n\n- **You filter by it in almost every query, and it's bounded**: make it a label. Example: `env`, `app`.\n- **It's high-cardinality but you query it often**: attach it as [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/). It's queryable with a label filter expression, such as `{job=\"example\"} | pod=\"myservice-abc1234-56789\"`, without being indexed or creating streams. Examples: pod names, process IDs, trace IDs.\n- **Everything else**: leave it in the log line and extract it at query time with parsers like `json` or `logfmt`, or search it with filter expressions." - }, - { - "type": "markdown", - "content": "## Audit what your clients send\n\nCollectors and agents can attach dynamic labels you didn't choose. Use the fewest-labels test on whatever Alloy, Fluentd, or your Docker driver is applying. To see your streams and find high-cardinality labels, run:\n\n```\nlogcli series '{}' --since=1h --analyze-labels\n```\n\nA label with thousands of unique values found in nearly every stream, such as a `requestId`, should be removed from labels and queried with a filter expression instead.\n\nIf many teams send logs and you can't audit or change every client, apply the same approach you learned for metrics: route logs through gateway collectors your admin team owns, enforce label policy at that layer, and manage those gateway configurations centrally with [Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/)." - }, - { - "type": "markdown", - "content": "You now have the logs framework: static origin labels, dynamic labels only when bounded and queried, structured metadata for high-cardinality fields, and filter expressions for the rest." - }, - { - "type": "markdown", - "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/)\n- [What is structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/)" - } - ] -} + "id": "labeling-strategy-logs-labeling-strategy", + "title": "Apply labeling strategy to logs", + "blocks": [{ + "type": "markdown", + "content": "Loki, which powers Grafana Cloud Logs, takes the fewest labels principle further than metrics: it indexes only your labels, not the log content, so every label combination splits your logs into another stream to index and store. This milestone gives you the decision framework for what belongs in a label, what belongs in structured metadata, and what stays in the log line." + }, + { + "type": "markdown", + "content": "## Labels tell you where\n\nGood log labels describe the origin or context of the log: application, namespace, environment, cluster, region. These are static, bounded, and what you actually filter by when you start a search. Loki performs best with few labels, and the fewer streams you create, the smaller the index and the faster your queries." + }, + { + "type": "image", + "src": "https://interactive-learning.grafana.net/packages/labeling-strategy-lj/assets/images/log-stream-splitting.svg", + "alt": "One application splitting into four log streams because two labels with two values each create four unique label combinations" + }, + { + "type": "markdown", + "content": "## Use dynamic labels sparingly\n\nA dynamic label takes its value from the log content, such as `level`. Use them rarely, and only when all of these are true:\n\n- The value set is low-cardinality, ideally tens of values.\n- The values are long lived, such as the first segment of an HTTP path, not ephemeral IDs.\n- Users actually query by the label frequently.\n\nOtherwise, don't index it. Loki's filter expressions are fast: `{app=\"api\"} |= \"level=error\"` performs comparably to a `level` label for many workloads, without splitting your streams. Don't add a label until you know you need it." + }, + { + "type": "markdown", + "content": "## The decision framework\n\nFor any piece of log data, ask where it belongs:\n\n- **You filter by it in almost every query, and it's bounded**: make it a label. Example: `env`, `app`.\n- **It's high-cardinality but you query it often**: attach it as [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/). It's queryable with a label filter expression, such as `{job=\"example\"} | pod=\"myservice-abc1234-56789\"`, without being indexed or creating streams. Examples: pod names, process IDs, trace IDs.\n- **Everything else**: leave it in the log line and extract it at query time with parsers like `json` or `logfmt`, or search it with filter expressions." + }, + { + "type": "markdown", + "content": "## Audit what your clients send\n\nCollectors and agents can attach dynamic labels you didn't choose. Use the fewest labels test on whatever Alloy, Fluentd, or your Docker driver is applying. To see your streams and find high-cardinality labels, run:\n\n```\nlogcli series '{}' --since=1h --analyze-labels\n```\n\nA label with thousands of unique values found in nearly every stream, such as a `requestId`, should be removed from labels and queried with a filter expression instead.\n\nIf many teams send logs and you can't audit or change every client, apply the same approach you learned for metrics: route logs through gateway collectors your admin team owns, enforce label policy at that layer, and manage those gateway configurations centrally with [Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/)." + }, + { + "type": "markdown", + "content": "You now have the logs framework. This includes static origin labels, dynamic labels only when bounded and queried, structured metadata for high-cardinality fields, and filter expressions for the rest." + }, + { + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/)\n- [What is structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/)" + } + ] +} \ No newline at end of file diff --git a/labeling-strategy-lj/metrics-labeling-strategy/content.json b/labeling-strategy-lj/metrics-labeling-strategy/content.json index 5915c73b..65e577f1 100644 --- a/labeling-strategy-lj/metrics-labeling-strategy/content.json +++ b/labeling-strategy-lj/metrics-labeling-strategy/content.json @@ -1,43 +1,42 @@ { - "id": "labeling-strategy-metrics-labeling-strategy", - "title": "Choose metric labels deliberately", - "blocks": [ - { - "type": "markdown", - "content": "A metric label should be a dimension you aggregate or filter by, with a bounded set of values. This milestone gives you the rules for choosing metric labels and shows where cardinality mistakes usually enter a stack." - }, - { - "type": "markdown", - "content": "## Bounded or it doesn't belong\n\nThe single most important rule: **label values must be bounded**. A label like `env` has a handful of values forever. A label like `user_id` grows without limit, and every new value creates another active series that's billed and indexed.\n\nNever use these as metric label values:\n\n- User, customer, or session IDs\n- Request or trace IDs\n- IP addresses\n- Email addresses\n- Timestamps\n\nEach of these creates a new series per unique value, and the combinations multiply against every other label on the metric." - }, - { - "type": "image", - "src": "https://grafana.com/media/docs/learning-journey/labeling-strategy/unbounded-label-growth.svg", - "alt": "A chart comparing active series over time: a bounded label like env stays flat while an unbounded label like user_id grows without limit" - }, - { - "type": "markdown", - "content": "## Ask before you add\n\nFor every label, ask:\n\n- **Will I aggregate or filter by it?** If nobody queries by this dimension, it's pure cost.\n- **Is the value set bounded?** Think tens of values, not thousands.\n- **Does it multiply badly?** Check the label against the labels already on the metric. Three bounded labels can still combine into thousands of series." - }, - { - "type": "markdown", - "content": "## Where the mistakes happen\n\nCardinality problems usually enter a stack at ingestion setup, not in application code reviews. Enabling an integration with default settings, such as the Kubernetes Monitoring Helm chart across all your clusters, can turn on thousands of metrics and labels at once. That visibility is valuable, but only if your budget expects it. When you set up a new integration:\n\n- Review what metrics the defaults send before pointing them at production.\n- Use allowlists to keep only the metrics and labels you'll use. To reduce what Kubernetes Monitoring collects, refer to [Manage your configuration](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/manage-configuration/).\n- Check the Cardinality tab after enabling anything new, and watch for **New** badges." - }, - { - "type": "markdown", - "content": "## Filter at the source\n\nYou can keep unnecessary series from ever leaving your infrastructure with relabeling rules in Prometheus, Alloy, or your collector:\n\n- **Allowlisting**: keep an explicit set of important metrics and labels and drop everything else, using `keep` and `labelkeep` actions.\n- **Denylisting**: drop an explicit set of high-cardinality metrics and labels and keep everything else, using `drop` and `labeldrop` actions.\n\nFor the full relabeling reference and examples, refer to [Reduce metrics costs by filtering collected and forwarded metrics](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/client-side-filtering/)." - }, - { - "type": "markdown", - "content": "## Enforce your strategy at a gateway\n\nRelabeling rules only help if you control the collectors that apply them. In larger organizations, many teams send telemetry, and the team that pays the bill often can't change every client. In that case, route telemetry through a tier of gateway collectors that your platform or admin team owns, and apply your labeling and filtering policies at that layer. Whatever upstream teams send, your rules run at the gateway before data reaches Grafana Cloud.\n\n[Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/) helps you run this pattern at scale: it gives you centralized control over your collector configurations, so you can roll out and update gateway policies from one place instead of editing collectors one by one." - }, - { - "type": "markdown", - "content": "You now have the prevention rules: bounded values only, add labels only when queried, audit defaults at ingestion setup, and enforce policy at a layer you control. Next, you'll fix cardinality you already have." - }, - { - "type": "markdown", - "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Client-side filtering](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/client-side-filtering/)\n- [Reduce costs by adjusting data points per minute](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/adjust-data-points-per-minute/)" - } - ] -} + "id": "labeling-strategy-metrics-labeling-strategy", + "title": "Choose metric labels deliberately", + "blocks": [{ + "type": "markdown", + "content": "A metric label should be a dimension you aggregate or filter by, with a bounded set of values. This milestone gives you the rules for choosing metric labels and shows where cardinality mistakes usually enter a stack." + }, + { + "type": "markdown", + "content": "## Bounded or it doesn't belong\n\nThe single most important rule: **label values must be bounded**. A label like `env` has a handful of values forever. A label like `user_id` grows without limit, and every new value creates another active series that's billed and indexed.\n\nNever use these as metric label values:\n\n- User, customer, or session IDs\n- Request or trace IDs\n- IP addresses\n- Email addresses\n- Timestamps\n\nEach of these creates a new series per unique value, and the combinations multiply against every other label on the metric." + }, + { + "type": "image", + "src": "https://interactive-learning.grafana.net/packages/labeling-strategy-lj/assets/images/unbounded-label-growth.svg", + "alt": "A chart comparing active series over time: a bounded label like env stays flat while an unbounded label like user_id grows without limit" + }, + { + "type": "markdown", + "content": "## Ask before you add a label\n\nFor every label, ask:\n\n- **Will I aggregate or filter by it?** If nobody queries by this dimension, it's pure cost.\n- **Is the value set bounded?** Think tens of values, not thousands.\n- **How does it multiply?** Check the label against the labels already on the metric. Three bounded labels can still combine into thousands of series." + }, + { + "type": "markdown", + "content": "## Where the mistakes happen\n\nCardinality problems usually enter a stack at ingestion setup, not in application code reviews. Enabling an integration with an app or service with default settings, such as the Kubernetes Monitoring Helm chart across all your clusters, can turn on thousands of metrics and labels at once. That visibility is valuable, but only if your budget expects it. When you set up a new integration:\n\n- Review what metrics the defaults send before pushing them to production.\n- Use allow lists to keep only the metrics and labels you'll use.\n- Check the Cardinality tab after enabling anything new, and watch for **New** badges." + }, + { + "type": "markdown", + "content": "## Filter at the source\n\nYou can keep unnecessary series from ever leaving your infrastructure with relabeling rules in Prometheus, Alloy, or your collector:\n\n- **Allowlisting**: keep an explicit set of important metrics and labels and drop everything else. For example, for Prometheus metrics, use `keep` and `labelkeep` actions.\n- **Denylisting**: drop an explicit set of high-cardinality metrics and labels and keep everything else. For example, for Prometheus metrics, use `drop` and `labeldrop` actions.\n\nFor the full relabeling reference and examples, refer to [Reduce metrics costs by filtering collected and forwarded metrics](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/client-side-filtering/)." + }, + { + "type": "markdown", + "content": "## Enforce your strategy at a gateway\n\nRelabeling rules only help if you control the collectors that apply them. In larger organizations, many teams send telemetry, and the team that pays the bill often can't change every client. In that case, route telemetry through a tier of gateway collectors that your platform or admin team owns, and apply your labeling and filtering policies at that layer. Whatever upstream teams send, your rules run at the gateway before data reaches Grafana Cloud.\n\n[Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/) helps you run this pattern at scale: it gives you centralized control over your collector configurations, so you can roll out and update gateway policies from one place instead of editing collectors one by one." + }, + { + "type": "markdown", + "content": "You now have the prevention rules: bounded values only, add labels only when queried, audit defaults at ingestion setup, and enforce policy at a layer you control. Next, you'll fix cardinality you already have." + }, + { + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Client-side filtering](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/client-side-filtering/)\n- [Reduce costs by adjusting data points per minute](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/adjust-data-points-per-minute/)" + } + ] +} \ No newline at end of file diff --git a/labeling-strategy-lj/understand-cardinality/content.json b/labeling-strategy-lj/understand-cardinality/content.json index e2f59d94..5c9c6a2f 100644 --- a/labeling-strategy-lj/understand-cardinality/content.json +++ b/labeling-strategy-lj/understand-cardinality/content.json @@ -1,40 +1,39 @@ { - "id": "labeling-strategy-understand-cardinality", - "title": "Understand cardinality and active series", - "blocks": [ - { - "type": "markdown", - "content": "Cardinality is the number of distinct values an attribute can have. In observability, it determines how many unique series and streams your labels create, and that number is what you're billed for. This milestone gives you the mental model that everything else in this path builds on." - }, - { - "type": "markdown", - "content": "## What you're actually billed for\n\nA common misconception is that metrics billing is per metric name. It isn't. A time series is a metric name plus one unique combination of label key-value pairs, and each series that receives data within the last 20 minutes counts as an **active series**. Grafana Cloud bills metrics on the 95th percentile of your active series count over the billing period, so one metric name with many label combinations can cost far more than dozens of well-labeled metrics.\n\nLogs work the same way structurally: each unique combination of label values defines a **log stream**, and Loki indexes, batches, and stores each stream separately." - }, - { - "type": "image", - "src": "https://grafana.com/media/docs/learning-journey/labeling-strategy/series-anatomy.svg", - "alt": "Anatomy of a time series and a log stream, showing that each unique combination of label values creates one series or stream" - }, - { - "type": "markdown", - "content": "## The multiplication\n\nLabels don't add series, they multiply them. A `status` label with 3 values combined with an `action` label with 5 values produces 15 unique series from what looks like one metric. Add an `endpoint` label with 3 values and you're at 45. Every label you add multiplies the count by that label's number of values." - }, - { - "type": "image", - "src": "https://grafana.com/media/docs/learning-journey/labeling-strategy/label-multiplication.svg", - "alt": "Two rows showing label value counts multiplying: 3 status values times 5 action values equals 15 series, and adding a 3-value endpoint label grows the total to 45 series" - }, - { - "type": "markdown", - "content": "## Why this hurts twice\n\nHigh cardinality doesn't only raise your bill:\n\n- **Cost**: every unique series and stream is billed, whether or not anyone queries it.\n- **Performance**: for logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." - }, - { - "type": "markdown", - "content": "You now have the core model: labels multiply into series and streams, and you pay for every unique combination. Next, you'll look at your own stack's cardinality." - }, - { - "type": "markdown", - "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Understand metrics usage and cost](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/understand-usage-cost/metrics/)\n- [Cardinality in Loki](https://grafana.com/docs/loki/latest/get-started/labels/cardinality/)" - } - ] -} + "id": "labeling-strategy-understand-cardinality", + "title": "Understand cardinality and active series", + "blocks": [{ + "type": "markdown", + "content": "Cardinality is the number of distinct values an attribute can have. In observability, it determines how many unique series and streams your labels create, and that number is what you're billed for." + }, + { + "type": "markdown", + "content": "## What you're actually billed for\n\nA common misconception is that metrics billing is per metric name. It isn't. A time series is a metric name plus one unique combination of label key-value pairs, and each series that receives data within the last 20 minutes counts as an **active series**. Grafana Cloud bills metrics on the 95th percentile of your active series count over the billing period, so one metric name with many label combinations can cost far more than dozens of well labeled metrics.\n\nLogs work the same way structurally; each unique combination of label values defines a **log stream**, and Loki indexes, batches, and stores each stream separately." + }, + { + "type": "image", + "src": "https://interactive-learning.grafana.net/packages/labeling-strategy-lj/assets/images/series-anatomy.svg", + "alt": "Anatomy of a time series and a log stream, showing that each unique combination of label values creates one series or stream" + }, + { + "type": "markdown", + "content": "## Labels multiply series\n\nLabels don't add series, they multiply them. A `status` label with 3 values combined with an `action` label with 5 values produces 15 unique series from what looks like one metric. Add an `endpoint` label with 3 values and you're at 45. Every label you add multiplies the count by that label's number of values." + }, + { + "type": "image", + "src": "https://interactive-learning.grafana.net/packages/labeling-strategy-lj/assets/images/label-multiplication.svg", + "alt": "Two rows showing label value counts multiplying. 3 status values times 5 action values equals 15 series, and adding a 3-value endpoint label grows the total to 45 series" + }, + { + "type": "markdown", + "content": "## Cardinality effects cost and performance\n\nHigh cardinality directly affects both your bill and can affect performance.\n\n- **Cost**: every unique series and stream is billed, whether or not anyone queries it.\n- **Performance**: for logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." + }, + { + "type": "markdown", + "content": "You now have the core model: labels multiply into series and streams, and you pay for every unique combination. Next, you'll look at your own stack's cardinality." + }, + { + "type": "markdown", + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [Understand metrics usage and cost](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/understand-usage-cost/metrics/)\n- [Cardinality in Loki](https://grafana.com/docs/loki/latest/get-started/labels/cardinality/)" + } + ] +} \ No newline at end of file From fd558217ff64683632f744bce2373180cccfc001 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Tue, 21 Jul 2026 10:13:44 -0400 Subject: [PATCH 04/17] change milestone name --- .../content.json | 4 ++-- .../manifest.json | 2 +- .../website.yaml | 2 +- labeling-strategy-lj/end-journey/manifest.json | 2 +- labeling-strategy-lj/logs-labeling-strategy/manifest.json | 2 +- labeling-strategy-lj/manifest.json | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) rename labeling-strategy-lj/{correlate-dont-inflate => correlate-signals}/content.json (96%) rename labeling-strategy-lj/{correlate-dont-inflate => correlate-signals}/manifest.json (90%) rename labeling-strategy-lj/{correlate-dont-inflate => correlate-signals}/website.yaml (94%) diff --git a/labeling-strategy-lj/correlate-dont-inflate/content.json b/labeling-strategy-lj/correlate-signals/content.json similarity index 96% rename from labeling-strategy-lj/correlate-dont-inflate/content.json rename to labeling-strategy-lj/correlate-signals/content.json index 81abf6d9..ec5deb29 100644 --- a/labeling-strategy-lj/correlate-dont-inflate/content.json +++ b/labeling-strategy-lj/correlate-signals/content.json @@ -1,6 +1,6 @@ { - "id": "labeling-strategy-correlate-dont-inflate", - "title": "Correlate signals instead of inflating labels", + "id": "labeling-strategy-correlate-signals", + "title": "Correlate signals", "blocks": [{ "type": "markdown", "content": "Sometimes you genuinely need a high-cardinality value, like a request ID, when you're debugging. The answer isn't to delete it, and it isn't to put it on a metric label. It's to put the value on the signal designed to hold it, and correlate across signals when you investigate." diff --git a/labeling-strategy-lj/correlate-dont-inflate/manifest.json b/labeling-strategy-lj/correlate-signals/manifest.json similarity index 90% rename from labeling-strategy-lj/correlate-dont-inflate/manifest.json rename to labeling-strategy-lj/correlate-signals/manifest.json index c9023ba1..1d629fc7 100644 --- a/labeling-strategy-lj/correlate-dont-inflate/manifest.json +++ b/labeling-strategy-lj/correlate-signals/manifest.json @@ -1,5 +1,5 @@ { - "id": "labeling-strategy-correlate-dont-inflate", + "id": "labeling-strategy-correlate-signals", "type": "guide", "description": "Learn how to keep metrics low-cardinality by moving high-cardinality detail to correlated logs and querying it with structured metadata.", "category": "take-action", diff --git a/labeling-strategy-lj/correlate-dont-inflate/website.yaml b/labeling-strategy-lj/correlate-signals/website.yaml similarity index 94% rename from labeling-strategy-lj/correlate-dont-inflate/website.yaml rename to labeling-strategy-lj/correlate-signals/website.yaml index b4934c47..e3f2d187 100644 --- a/labeling-strategy-lj/correlate-dont-inflate/website.yaml +++ b/labeling-strategy-lj/correlate-signals/website.yaml @@ -1,4 +1,4 @@ -menuTitle: Correlate, don't inflate +menuTitle: Correlate signals weight: 700 step: 8 layout: single-journey diff --git a/labeling-strategy-lj/end-journey/manifest.json b/labeling-strategy-lj/end-journey/manifest.json index c1420904..762f2a89 100644 --- a/labeling-strategy-lj/end-journey/manifest.json +++ b/labeling-strategy-lj/end-journey/manifest.json @@ -10,7 +10,7 @@ "testEnvironment": { "tier": "cloud" }, - "depends": ["labeling-strategy-correlate-dont-inflate"], + "depends": ["labeling-strategy-correlate-signals"], "recommends": [], "suggests": ["billing-usage-lj", "adaptive-metrics-lj", "adaptive-logs-lj"], "provides": [] diff --git a/labeling-strategy-lj/logs-labeling-strategy/manifest.json b/labeling-strategy-lj/logs-labeling-strategy/manifest.json index 0896895d..76eb4232 100644 --- a/labeling-strategy-lj/logs-labeling-strategy/manifest.json +++ b/labeling-strategy-lj/logs-labeling-strategy/manifest.json @@ -11,7 +11,7 @@ "tier": "cloud" }, "depends": ["labeling-strategy-fix-metrics-cardinality"], - "recommends": ["labeling-strategy-correlate-dont-inflate"], + "recommends": ["labeling-strategy-correlate-signals"], "suggests": [], "provides": [] } diff --git a/labeling-strategy-lj/manifest.json b/labeling-strategy-lj/manifest.json index 96e02cd9..16c47553 100644 --- a/labeling-strategy-lj/manifest.json +++ b/labeling-strategy-lj/manifest.json @@ -34,7 +34,7 @@ "labeling-strategy-metrics-labeling-strategy", "labeling-strategy-fix-metrics-cardinality", "labeling-strategy-logs-labeling-strategy", - "labeling-strategy-correlate-dont-inflate", + "labeling-strategy-correlate-signals", "labeling-strategy-end-journey" ], "depends": [], From 0d6443bdb39968f12c1e49cd9fc907406a8c4ac9 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Tue, 21 Jul 2026 10:26:55 -0400 Subject: [PATCH 05/17] change toc name --- labeling-strategy-lj/website.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/labeling-strategy-lj/website.yaml b/labeling-strategy-lj/website.yaml index dfcac47a..5dbdd886 100644 --- a/labeling-strategy-lj/website.yaml +++ b/labeling-strategy-lj/website.yaml @@ -1,4 +1,4 @@ -menuTitle: Labeling strategy +menuTitle: Design a labeling strategy weight: 1250 journey: group: take-action From 276c22ad737466b078bafb1bec99d8bf0405cca5 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 24 Jul 2026 08:35:35 -0400 Subject: [PATCH 06/17] Update labeling-strategy-lj/understand-cardinality/content.json Co-authored-by: J Stickler --- labeling-strategy-lj/understand-cardinality/content.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/labeling-strategy-lj/understand-cardinality/content.json b/labeling-strategy-lj/understand-cardinality/content.json index 5c9c6a2f..abb7fdc7 100644 --- a/labeling-strategy-lj/understand-cardinality/content.json +++ b/labeling-strategy-lj/understand-cardinality/content.json @@ -25,7 +25,7 @@ }, { "type": "markdown", - "content": "## Cardinality effects cost and performance\n\nHigh cardinality directly affects both your bill and can affect performance.\n\n- **Cost**: every unique series and stream is billed, whether or not anyone queries it.\n- **Performance**: for logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." + "content": "## Cardinality affects cost and performance\n\nHigh cardinality directly affects both your bill and can affect performance.\n\n- **Cost**: every unique series and stream is billed, whether or not anyone queries it.\n- **Performance**: for logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." }, { "type": "markdown", From 5b8c3e155ed68090fc7d114ec919b93a5fc3a283 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 24 Jul 2026 08:35:58 -0400 Subject: [PATCH 07/17] Update labeling-strategy-lj/explore-your-cardinality/content.json Co-authored-by: J Stickler --- labeling-strategy-lj/explore-your-cardinality/content.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/labeling-strategy-lj/explore-your-cardinality/content.json b/labeling-strategy-lj/explore-your-cardinality/content.json index 61e97155..711549df 100644 --- a/labeling-strategy-lj/explore-your-cardinality/content.json +++ b/labeling-strategy-lj/explore-your-cardinality/content.json @@ -62,7 +62,7 @@ }, { "type": "markdown", - "content": "Now that you've used the Cardinality tab to identify which metrics and labels are contributing to your active series count, ask three questions:\n\n- **Which metrics produce the most series?** They're your biggest cost levers.\n- **Which labels have the most values?** They're the multiplication factors to challenge.\n- **Is anything both high-cardinality and unused?** That's cost with no value, and the easiest win." + "content": "Now that you've used the Cardinality tab to identify which metrics and labels are contributing to your active series count, ask three questions:\n\n- **Which metrics produce the most series?** They're your biggest cost levers.\n- **Which labels have the most unique values?** They're the multiplication factors to challenge.\n- **Is anything both high-cardinality and unused?** That's cost with no value, and the easiest win." }, { "type": "markdown", From 7ee3fbd106c59749d5b02c449b4b337b2293cc4c Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 24 Jul 2026 09:02:50 -0400 Subject: [PATCH 08/17] Julie's suggestions --- labeling-strategy-lj/explore-your-cardinality/content.json | 2 +- labeling-strategy-lj/fix-metrics-cardinality/content.json | 2 +- labeling-strategy-lj/metrics-labeling-strategy/content.json | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/labeling-strategy-lj/explore-your-cardinality/content.json b/labeling-strategy-lj/explore-your-cardinality/content.json index 711549df..e804534c 100644 --- a/labeling-strategy-lj/explore-your-cardinality/content.json +++ b/labeling-strategy-lj/explore-your-cardinality/content.json @@ -70,7 +70,7 @@ }, { "type": "markdown", - "content": "In the next milestone, you'll learn how to choose metric labels so this table never becomes a problem." + "content": "In the next milestone, you'll learn how to choose metric labels so cardinality never becomes a problem." }, { "type": "markdown", diff --git a/labeling-strategy-lj/fix-metrics-cardinality/content.json b/labeling-strategy-lj/fix-metrics-cardinality/content.json index d9f31d31..4373713e 100644 --- a/labeling-strategy-lj/fix-metrics-cardinality/content.json +++ b/labeling-strategy-lj/fix-metrics-cardinality/content.json @@ -56,7 +56,7 @@ }, { "type": "markdown", - "content": "A good remediation order:\n\n1. **Unused metrics first**: metrics with the **Unused** badge in the Cardinality tab are cost with no value.\n2. **High-cardinality labels on used metrics next**: aggregate away the label dimensions nobody queries, and keep the ones they do.\n3. **Protect what matters**: define exemptions for metrics that must never be aggregated." + "content": "The recommended order to remediate cardinality:\n\n1. **Unused metrics first**: metrics with the **Unused** badge in the Cardinality tab add cost with no value.\n2. **High-cardinality labels on used metrics next**: aggregate away the label dimensions nobody queries, and keep the ones they do.\n3. **Protect what matters**: define exemptions for metrics that must never be aggregated." }, { "type": "markdown", diff --git a/labeling-strategy-lj/metrics-labeling-strategy/content.json b/labeling-strategy-lj/metrics-labeling-strategy/content.json index 65e577f1..d8f6e5a9 100644 --- a/labeling-strategy-lj/metrics-labeling-strategy/content.json +++ b/labeling-strategy-lj/metrics-labeling-strategy/content.json @@ -32,7 +32,7 @@ }, { "type": "markdown", - "content": "You now have the prevention rules: bounded values only, add labels only when queried, audit defaults at ingestion setup, and enforce policy at a layer you control. Next, you'll fix cardinality you already have." + "content": "You now have the prevention rules: bounded values only, add labels only when they will be used in queries, audit defaults at ingestion setup, and enforce policy at a layer you control. Next, you'll fix cardinality you already have." }, { "type": "markdown", From e62a6666082ba07f343a3897d78314c86e21ec29 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 24 Jul 2026 10:58:07 -0400 Subject: [PATCH 09/17] change dumb language --- labeling-strategy-lj/metrics-labeling-strategy/content.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/labeling-strategy-lj/metrics-labeling-strategy/content.json b/labeling-strategy-lj/metrics-labeling-strategy/content.json index d8f6e5a9..45edbbf5 100644 --- a/labeling-strategy-lj/metrics-labeling-strategy/content.json +++ b/labeling-strategy-lj/metrics-labeling-strategy/content.json @@ -7,7 +7,7 @@ }, { "type": "markdown", - "content": "## Bounded or it doesn't belong\n\nThe single most important rule: **label values must be bounded**. A label like `env` has a handful of values forever. A label like `user_id` grows without limit, and every new value creates another active series that's billed and indexed.\n\nNever use these as metric label values:\n\n- User, customer, or session IDs\n- Request or trace IDs\n- IP addresses\n- Email addresses\n- Timestamps\n\nEach of these creates a new series per unique value, and the combinations multiply against every other label on the metric." + "content": "## Use bounded labels\n\nThe single most important rule: **label values must be bounded**. A label like `env` has a handful of values forever. A label like `user_id` grows without limit, and every new value creates another active series that's billed and indexed.\n\nNever use these as metric label values:\n\n- User, customer, or session IDs\n- Request or trace IDs\n- IP addresses\n- Email addresses\n- Timestamps\n\nEach of these creates a new series per unique value, and the combinations multiply against every other label on the metric." }, { "type": "image", @@ -16,7 +16,7 @@ }, { "type": "markdown", - "content": "## Ask before you add a label\n\nFor every label, ask:\n\n- **Will I aggregate or filter by it?** If nobody queries by this dimension, it's pure cost.\n- **Is the value set bounded?** Think tens of values, not thousands.\n- **How does it multiply?** Check the label against the labels already on the metric. Three bounded labels can still combine into thousands of series." + "content": "## Questions to ask yourself before you add a label\n\nFor every label, ask:\n\n- **Will I aggregate or filter by it?** If nobody queries by this dimension, it's pure cost.\n- **Is the value set bounded?** Think tens of values, not thousands.\n- **How does it multiply?** Check the label against the labels already on the metric. Three bounded labels can still combine into thousands of series." }, { "type": "markdown", From 9f72c1acd7d698ecc46db11bacde25b9f629142f Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 24 Jul 2026 11:16:21 -0400 Subject: [PATCH 10/17] clarify log streams cost --- labeling-strategy-lj/understand-cardinality/content.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/labeling-strategy-lj/understand-cardinality/content.json b/labeling-strategy-lj/understand-cardinality/content.json index abb7fdc7..97c2f13a 100644 --- a/labeling-strategy-lj/understand-cardinality/content.json +++ b/labeling-strategy-lj/understand-cardinality/content.json @@ -25,7 +25,7 @@ }, { "type": "markdown", - "content": "## Cardinality affects cost and performance\n\nHigh cardinality directly affects both your bill and can affect performance.\n\n- **Cost**: every unique series and stream is billed, whether or not anyone queries it.\n- **Performance**: for logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." + "content": "## Cardinality affects cost and performance\n\nHigh cardinality affects both your bill and query performance, but metrics and logs are billed differently.\n\n- **Metrics cost**: Grafana Cloud bills metrics on active series, so every unique label combination is a billing unit, whether or not anyone queries it.\n- **Logs cost**: Grafana Cloud bills logs on GB ingested, not per stream. High stream cardinality doesn't add a direct per-stream charge, but it can inflate query costs because larger indexes and more chunks mean bigger scans.\n- **Performance**: for logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." }, { "type": "markdown", From 29173db056a2670a7f22c99e7099e480cd8e4430 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 24 Jul 2026 11:24:29 -0400 Subject: [PATCH 11/17] logs clarification --- labeling-strategy-lj/understand-cardinality/content.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/labeling-strategy-lj/understand-cardinality/content.json b/labeling-strategy-lj/understand-cardinality/content.json index 97c2f13a..f413afe1 100644 --- a/labeling-strategy-lj/understand-cardinality/content.json +++ b/labeling-strategy-lj/understand-cardinality/content.json @@ -3,11 +3,11 @@ "title": "Understand cardinality and active series", "blocks": [{ "type": "markdown", - "content": "Cardinality is the number of distinct values an attribute can have. In observability, it determines how many unique series and streams your labels create, and that number is what you're billed for." + "content": "Cardinality is the number of distinct values an attribute can have. In observability, it determines how many unique series and streams your labels create, and that number can significantly impact your costs." }, { "type": "markdown", - "content": "## What you're actually billed for\n\nA common misconception is that metrics billing is per metric name. It isn't. A time series is a metric name plus one unique combination of label key-value pairs, and each series that receives data within the last 20 minutes counts as an **active series**. Grafana Cloud bills metrics on the 95th percentile of your active series count over the billing period, so one metric name with many label combinations can cost far more than dozens of well labeled metrics.\n\nLogs work the same way structurally; each unique combination of label values defines a **log stream**, and Loki indexes, batches, and stores each stream separately." + "content": "## What you're actually billed for\n\nA common misconception is that metrics billing is per metric name. It isn't. A time series is a metric name plus one unique combination of label key-value pairs, and each series that receives data within the last 20 minutes counts as an **active series**. Grafana Cloud bills metrics on the 95th percentile of your active series count over the billing period, so one metric name with many label combinations can cost far more than dozens of well labeled metrics.\n\nLogs work the same way structurally, though cost attribution is different. Each unique combination of label values defines a **log stream**, and Loki indexes, batches, and stores each stream separately. Though high-cardinality labels don't add a direct per-stream charge, they can inflate query costs because larger indexes and more chunks mean bigger scans." }, { "type": "image", From 54f17fde77f335c8a82cc7e27cb3571d61b3000e Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 24 Jul 2026 11:26:31 -0400 Subject: [PATCH 12/17] more log stream clarification --- labeling-strategy-lj/understand-cardinality/content.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/labeling-strategy-lj/understand-cardinality/content.json b/labeling-strategy-lj/understand-cardinality/content.json index f413afe1..c6502209 100644 --- a/labeling-strategy-lj/understand-cardinality/content.json +++ b/labeling-strategy-lj/understand-cardinality/content.json @@ -3,7 +3,7 @@ "title": "Understand cardinality and active series", "blocks": [{ "type": "markdown", - "content": "Cardinality is the number of distinct values an attribute can have. In observability, it determines how many unique series and streams your labels create, and that number can significantly impact your costs." + "content": "Cardinality is the number of distinct values an attribute can have. In observability, it determines how many unique series and streams your labels create, and that number can significantly impact your cost and performance." }, { "type": "markdown", From 1d0e0db677c9fe56e067b38706a2a7958b0a5ca1 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 24 Jul 2026 12:02:34 -0400 Subject: [PATCH 13/17] small change --- labeling-strategy-lj/content.json | 49 +++++++++++++++---------------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/labeling-strategy-lj/content.json b/labeling-strategy-lj/content.json index 3da75b37..599bbd8a 100644 --- a/labeling-strategy-lj/content.json +++ b/labeling-strategy-lj/content.json @@ -1,26 +1,25 @@ { - "id": "labeling-strategy-lj", - "title": "Design a labeling strategy to control cardinality", - "blocks": [ - { - "type": "markdown", - "content": "Welcome to the Grafana Cloud learning path that teaches you how to design a labeling strategy that keeps cardinality, cost, and query performance under control.\n\nEvery label you attach to a metric or log multiplies the number of unique series or streams you store and pay for. Most billing surprises in Grafana Cloud trace back to labeling decisions made at instrumentation time, often by accepting defaults without realizing their cost impact. This path teaches you the math behind cardinality, how to inspect your own cardinality, and how to choose labels deliberately for metrics and logs." - }, - { - "type": "markdown", - "content": "## Here's what to expect\n\nWhen you complete this path, you'll be able to:\n\n- Explain how label combinations multiply into unique active series and log streams\n- Inspect your own cardinality with the Cardinality tab in the Cost Management and Billing app\n- Choose metric labels deliberately and avoid unbounded values\n- Reduce existing metrics cardinality with relabeling and Adaptive Metrics\n- Decide when log data belongs in a label, in structured metadata, or in the log line\n- Move high-cardinality context to the right signal instead of deleting it" - }, - { - "type": "markdown", - "content": "## Before you begin\n\nBefore you design your labeling strategy, ensure that you have:\n\n- A Grafana Cloud account. To create an account, refer to [Grafana Cloud](https://grafana.com/signup/cloud/connect-account). A stack that's already receiving metrics and logs shows richer data in the hands-on milestones.\n- The permissions granted by the **Billing and usage reader** role, including cardinality read access. Refer to [Manage access to the Cost Management and Billing app](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/set-up/cost-management-billing-rbac/).\n- Basic familiarity with metrics and logs concepts." - }, - { - "type": "markdown", - "content": "## Troubleshooting\n\nIf you get stuck, we've got your back! Where appropriate, troubleshooting information is just a click away." - }, - { - "type": "markdown", - "content": "## More to explore\n\nWe understand you might want to explore other capabilities not strictly on this path. We'll provide you opportunities where it makes sense." - } - ] -} + "id": "labeling-strategy-lj", + "title": "Design a labeling strategy to control cardinality", + "blocks": [{ + "type": "markdown", + "content": "Welcome to the Grafana Cloud learning path that teaches you how to design a labeling strategy that keeps cardinality, cost, and query performance under control.\n\nEvery label you attach to a metric or log multiplies the number of unique series or streams you generate. Most billing surprises in Grafana Cloud trace back to labeling decisions made at instrumentation time, often by accepting defaults without realizing their cost impact. This path teaches you the math behind cardinality, how to inspect your own cardinality, and how to choose labels deliberately for metrics and logs." + }, + { + "type": "markdown", + "content": "## Here's what to expect\n\nWhen you complete this path, you'll be able to:\n\n- Explain how label combinations multiply into unique active series and log streams\n- Inspect your own cardinality with the Cardinality tab in the Cost Management and Billing app\n- Choose metric labels deliberately and avoid unbounded values\n- Reduce existing metrics cardinality with relabeling and Adaptive Metrics\n- Decide when log data belongs in a label, in structured metadata, or in the log line\n- Move high-cardinality context to the right signal instead of deleting it" + }, + { + "type": "markdown", + "content": "## Before you begin\n\nBefore you design your labeling strategy, ensure that you have:\n\n- A Grafana Cloud account. To create an account, refer to [Grafana Cloud](https://grafana.com/signup/cloud/connect-account). A stack that's already receiving metrics and logs shows richer data in the hands-on milestones.\n- The permissions granted by the **Billing and usage reader** role, including cardinality read access. Refer to [Manage access to the Cost Management and Billing app](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/set-up/cost-management-billing-rbac/).\n- Basic familiarity with metrics and logs concepts." + }, + { + "type": "markdown", + "content": "## Troubleshooting\n\nIf you get stuck, we've got your back! Where appropriate, troubleshooting information is just a click away." + }, + { + "type": "markdown", + "content": "## More to explore\n\nWe understand you might want to explore other capabilities not strictly on this path. We'll provide you opportunities where it makes sense." + } + ] +} \ No newline at end of file From 3c670b523bee85ce29f5925ae6db231a5eb42f4f Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Fri, 24 Jul 2026 12:47:13 -0400 Subject: [PATCH 14/17] autocollapse set to false --- labeling-strategy-lj/content.json | 2 +- labeling-strategy-lj/explore-your-cardinality/content.json | 1 + labeling-strategy-lj/understand-cardinality/content.json | 4 ++-- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/labeling-strategy-lj/content.json b/labeling-strategy-lj/content.json index 599bbd8a..01e7ea0e 100644 --- a/labeling-strategy-lj/content.json +++ b/labeling-strategy-lj/content.json @@ -11,7 +11,7 @@ }, { "type": "markdown", - "content": "## Before you begin\n\nBefore you design your labeling strategy, ensure that you have:\n\n- A Grafana Cloud account. To create an account, refer to [Grafana Cloud](https://grafana.com/signup/cloud/connect-account). A stack that's already receiving metrics and logs shows richer data in the hands-on milestones.\n- The permissions granted by the **Billing and usage reader** role, including cardinality read access. Refer to [Manage access to the Cost Management and Billing app](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/set-up/cost-management-billing-rbac/).\n- Basic familiarity with metrics and logs concepts." + "content": "## Before you begin\n\nBefore you design your labeling strategy, ensure that you have:\n\n- A Grafana Cloud account. To create an account, refer to [Grafana Cloud](https://grafana.com/signup/cloud/connect-account). A stack that's already receiving metrics and logs shows richer data in the interactive milestones.\n- The permissions granted by the **Billing and usage reader** role, including cardinality read access. Refer to [Manage access to the Cost Management and Billing app](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/set-up/cost-management-billing-rbac/).\n- Basic familiarity with metrics and logs concepts." }, { "type": "markdown", diff --git a/labeling-strategy-lj/explore-your-cardinality/content.json b/labeling-strategy-lj/explore-your-cardinality/content.json index e804534c..151c7bcf 100644 --- a/labeling-strategy-lj/explore-your-cardinality/content.json +++ b/labeling-strategy-lj/explore-your-cardinality/content.json @@ -17,6 +17,7 @@ "type": "section", "id": "read-cardinality-tab", "title": "Read the Cardinality tab", + "autoCollapse": false, "blocks": [{ "type": "interactive", "action": "navigate", diff --git a/labeling-strategy-lj/understand-cardinality/content.json b/labeling-strategy-lj/understand-cardinality/content.json index c6502209..237c1f22 100644 --- a/labeling-strategy-lj/understand-cardinality/content.json +++ b/labeling-strategy-lj/understand-cardinality/content.json @@ -7,7 +7,7 @@ }, { "type": "markdown", - "content": "## What you're actually billed for\n\nA common misconception is that metrics billing is per metric name. It isn't. A time series is a metric name plus one unique combination of label key-value pairs, and each series that receives data within the last 20 minutes counts as an **active series**. Grafana Cloud bills metrics on the 95th percentile of your active series count over the billing period, so one metric name with many label combinations can cost far more than dozens of well labeled metrics.\n\nLogs work the same way structurally, though cost attribution is different. Each unique combination of label values defines a **log stream**, and Loki indexes, batches, and stores each stream separately. Though high-cardinality labels don't add a direct per-stream charge, they can inflate query costs because larger indexes and more chunks mean bigger scans." + "content": "## What you're actually billed for\n\nA common misconception is that metrics billing is per metric name. It isn't. A time series is a metric name plus one unique combination of label key-value pairs, and each series that receives data within the last 20 minutes counts as an **active series**. Grafana Cloud bills metrics on the 95th percentile of your active series count over the billing period, so one metric name with many label combinations can cost far more than dozens of well labeled metrics.\n\nLogs work the same way structurally, though cost attribution is different." }, { "type": "image", @@ -25,7 +25,7 @@ }, { "type": "markdown", - "content": "## Cardinality affects cost and performance\n\nHigh cardinality affects both your bill and query performance, but metrics and logs are billed differently.\n\n- **Metrics cost**: Grafana Cloud bills metrics on active series, so every unique label combination is a billing unit, whether or not anyone queries it.\n- **Logs cost**: Grafana Cloud bills logs on GB ingested, not per stream. High stream cardinality doesn't add a direct per-stream charge, but it can inflate query costs because larger indexes and more chunks mean bigger scans.\n- **Performance**: for logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." + "content": "## Cardinality affects cost and performance\n\nHigh cardinality affects both your bill and query performance, but metrics and logs are billed differently.\n\n- **Metrics cost**: Grafana Cloud bills metrics on active series, so every unique label combination is a billing unit, whether or not anyone queries it.\n- **Logs cost**: Grafana Cloud bills logs on GB processed and ingested, not per stream. High stream cardinality doesn't add a direct per-stream charge, but it can inflate query costs because larger indexes and more chunks mean bigger scans.\n- **Performance**: for logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." }, { "type": "markdown", From 08731680e2a6b5b1c8479adafe0d4d3bf586c2d2 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Mon, 27 Jul 2026 10:19:14 -0400 Subject: [PATCH 15/17] remove stupid colons --- labeling-strategy-lj/correlate-signals/content.json | 6 +++--- labeling-strategy-lj/fix-metrics-cardinality/content.json | 4 ++-- labeling-strategy-lj/logs-labeling-strategy/content.json | 6 +++--- labeling-strategy-lj/metrics-labeling-strategy/content.json | 4 ++-- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/labeling-strategy-lj/correlate-signals/content.json b/labeling-strategy-lj/correlate-signals/content.json index ec5deb29..28993a5d 100644 --- a/labeling-strategy-lj/correlate-signals/content.json +++ b/labeling-strategy-lj/correlate-signals/content.json @@ -7,15 +7,15 @@ }, { "type": "markdown", - "content": "## The pattern\n\nWhen a high-cardinality value is attached to a metric label, every unique value multiplies your active series. The same value attached to a log costs a fraction of that, because Loki doesn't index log content:\n\n- **Keep the metric low-cardinality**: the metric answers \"how many\" and \"how fast\" per bounded dimension, such as `service` and `status`.\n- **Put the high-cardinality detail on the correlated log**: attach it as structured metadata, or leave it in the log line. The log answers \"which exact request\".\n- **Correlate at query time**: shared bounded labels, such as `service` and `env`, connect the metric that alerted you to the logs that explain it." + "content": "## The pattern\n\nWhen a high-cardinality value is attached to a metric label, every unique value multiplies your active series. The same value attached to a log costs a fraction of that, because Loki doesn't index log content. Here's how to use structured metadata to keep your metrics low-cardinality and by leveraging logs:\n\n- **Keep the metric low-cardinality.** The metric answers \"how many\" and \"how fast\" per bounded dimension, such as `service` and `status`.\n- **Put the high-cardinality detail on the correlated log.** Attach it as structured metadata, or leave it in the log line. The log answers \"which exact request\".\n- **Correlate at query time.** Shared bounded labels, such as `service` and `env`, connect the metric that alerted you to the logs that explain it." }, { "type": "markdown", - "content": "## A worked example\n\nSuppose you track request latency and want to find slow requests for specific users.\n\n- **Instead of** `http_request_duration_seconds{user_id=\"u-48221\", ...}`, which creates a series per user per label combination,\n- **keep the metric as** `http_request_duration_seconds{service=\"checkout\", env=\"prod\"}`,\n- **and log each request** with `user_id` and `request_id` attached as structured metadata.\n\nWhen latency spikes, the metric tells you where and when. Then you query the logs for that service and window, filtering by structured metadata: `{app=\"checkout\"} | user_id=\"u-48221\"`. You keep full debugging detail without paying for it on every series." + "content": "## A working example\n\nSuppose you track request latency and want to find slow requests for specific users.\n\n- **Instead of** `http_request_duration_seconds{user_id=\"u-48221\", ...}`, which creates a series per user per label combination,\n- **keep the metric as** `http_request_duration_seconds{service=\"checkout\", env=\"prod\"}`,\n- **and log each request** with `user_id` and `request_id` attached as structured metadata.\n\nWhen latency spikes, the metric tells you where and when. Then you query the logs for that service and window, filtering by structured metadata: `{app=\"checkout\"} | user_id=\"u-48221\"`. You keep full debugging detail without paying for it on every series." }, { "type": "markdown", - "content": "## Why this works\n\nEach signal is priced and indexed for a different job. Metrics are cheap per series but multiply by label combinations. Logs carry arbitrary detail, and structured metadata makes high-cardinality fields queryable without indexing them. Putting each value where it belongs is what a labeling strategy actually is: not fewer data points, just the right home for each one." + "content": "## Why this works\n\nEach signal is priced and indexed for a different job. Metrics are cheap per series but multiply by label combinations. Logs carry arbitrary detail, and structured metadata makes high-cardinality fields queryable without indexing them. Putting each value where it belongs is what a labeling strategy actually is; not fewer data points, just the right home for each one." }, { "type": "markdown", diff --git a/labeling-strategy-lj/fix-metrics-cardinality/content.json b/labeling-strategy-lj/fix-metrics-cardinality/content.json index 4373713e..2b9a6dd8 100644 --- a/labeling-strategy-lj/fix-metrics-cardinality/content.json +++ b/labeling-strategy-lj/fix-metrics-cardinality/content.json @@ -7,7 +7,7 @@ }, { "type": "markdown", - "content": "## How Adaptive Metrics helps\n\nAdaptive Metrics analyzes how your organization uses its metrics across dashboards, alerts, recording rules, and queries. It then recommends aggregating underutilized metrics into lower-cardinality versions. Applying a recommendation doesn't delete the metric: it aggregates away unused label dimensions, and your existing dashboards, rules, and queries continue to work without modification. If usage changes later, you can remove the aggregation and the metric is stored at full cardinality again.\n\nThink of it as enforcement at the door: unnecessary series are stopped before they're persisted, without modifying your collector configuration." + "content": "## How Adaptive Metrics helps\n\nAdaptive Metrics analyzes how your organization uses its metrics across dashboards, alerts, recording rules, and queries. It then recommends aggregating underutilized metrics into lower-cardinality versions. Applying a recommendation doesn't delete the metric, rather it aggregates away unused label dimensions, and your existing dashboards, rules, and queries continue to work without modification. If usage changes later, you can remove the aggregation and the metric is stored at full cardinality again.\n\n Unnecessary series are stopped before they're persisted, without modifying your collector configuration." }, { "type": "markdown", @@ -56,7 +56,7 @@ }, { "type": "markdown", - "content": "The recommended order to remediate cardinality:\n\n1. **Unused metrics first**: metrics with the **Unused** badge in the Cardinality tab add cost with no value.\n2. **High-cardinality labels on used metrics next**: aggregate away the label dimensions nobody queries, and keep the ones they do.\n3. **Protect what matters**: define exemptions for metrics that must never be aggregated." + "content": "The recommended order to remediate cardinality:\n\n1. **Address unused metrics.** Metrics with the **Unused** badge in the Cardinality tab add cost with no value.\n2. **Assess high-cardinality labels on used metrics.** Aggregate away the label dimensions nobody queries, and keep the ones they do.\n3. **Protect what matters.** Define exemptions for metrics that must never be aggregated." }, { "type": "markdown", diff --git a/labeling-strategy-lj/logs-labeling-strategy/content.json b/labeling-strategy-lj/logs-labeling-strategy/content.json index ec2d993a..3b755a37 100644 --- a/labeling-strategy-lj/logs-labeling-strategy/content.json +++ b/labeling-strategy-lj/logs-labeling-strategy/content.json @@ -3,7 +3,7 @@ "title": "Apply labeling strategy to logs", "blocks": [{ "type": "markdown", - "content": "Loki, which powers Grafana Cloud Logs, takes the fewest labels principle further than metrics: it indexes only your labels, not the log content, so every label combination splits your logs into another stream to index and store. This milestone gives you the decision framework for what belongs in a label, what belongs in structured metadata, and what stays in the log line." + "content": "Loki, which powers Grafana Cloud Logs, takes the fewest labels principle further than metrics by indexing only your labels, not the log content, so every label combination splits your logs into another stream to index and store. This milestone gives you the decision framework for what belongs in a label, what belongs in structured metadata, and what stays in the log line." }, { "type": "markdown", @@ -16,11 +16,11 @@ }, { "type": "markdown", - "content": "## Use dynamic labels sparingly\n\nA dynamic label takes its value from the log content, such as `level`. Use them rarely, and only when all of these are true:\n\n- The value set is low-cardinality, ideally tens of values.\n- The values are long lived, such as the first segment of an HTTP path, not ephemeral IDs.\n- Users actually query by the label frequently.\n\nOtherwise, don't index it. Loki's filter expressions are fast: `{app=\"api\"} |= \"level=error\"` performs comparably to a `level` label for many workloads, without splitting your streams. Don't add a label until you know you need it." + "content": "## Use dynamic labels sparingly\n\nA dynamic label takes its value from the log content, such as `level`. Use them rarely, and only when all of the following are true:\n\n- The value set is low-cardinality, ideally tens of values.\n- The values are long lived, such as the first segment of an HTTP path, not ephemeral IDs.\n- Users actually query by the label frequently.\n\nOtherwise, don't index it. Loki's filter expressions are fast. `{app=\"api\"} |= \"level=error\"` performs comparably to a `level` label for many workloads, without splitting your streams. Don't add a label until you know you need it." }, { "type": "markdown", - "content": "## The decision framework\n\nFor any piece of log data, ask where it belongs:\n\n- **You filter by it in almost every query, and it's bounded**: make it a label. Example: `env`, `app`.\n- **It's high-cardinality but you query it often**: attach it as [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/). It's queryable with a label filter expression, such as `{job=\"example\"} | pod=\"myservice-abc1234-56789\"`, without being indexed or creating streams. Examples: pod names, process IDs, trace IDs.\n- **Everything else**: leave it in the log line and extract it at query time with parsers like `json` or `logfmt`, or search it with filter expressions." + "content": "## The decision framework\n\nFor any piece of log data, ask where it belongs.\n\n- **If you filter by it in almost every query, and it's bounded,** make it a label. Example: `env`, `app`.\n- **It's high-cardinality but you query it often,** attach it as [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/). It's queryable with a label filter expression, such as `{job=\"example\"} | pod=\"myservice-abc1234-56789\"`, without being indexed or creating streams. Examples: pod names, process IDs, trace IDs.\n- **Everything else,** leave it in the log line and extract it at query time with parsers like `json` or `logfmt`, or search it with filter expressions." }, { "type": "markdown", diff --git a/labeling-strategy-lj/metrics-labeling-strategy/content.json b/labeling-strategy-lj/metrics-labeling-strategy/content.json index 45edbbf5..c79767d3 100644 --- a/labeling-strategy-lj/metrics-labeling-strategy/content.json +++ b/labeling-strategy-lj/metrics-labeling-strategy/content.json @@ -28,11 +28,11 @@ }, { "type": "markdown", - "content": "## Enforce your strategy at a gateway\n\nRelabeling rules only help if you control the collectors that apply them. In larger organizations, many teams send telemetry, and the team that pays the bill often can't change every client. In that case, route telemetry through a tier of gateway collectors that your platform or admin team owns, and apply your labeling and filtering policies at that layer. Whatever upstream teams send, your rules run at the gateway before data reaches Grafana Cloud.\n\n[Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/) helps you run this pattern at scale: it gives you centralized control over your collector configurations, so you can roll out and update gateway policies from one place instead of editing collectors one by one." + "content": "## Enforce your strategy at a gateway\n\nRelabeling rules only help if you control the collectors that apply them. In larger organizations, many teams send telemetry, and the team that pays the bill often can't change every client. In that case, route telemetry through a tier of gateway collectors that your platform or admin team owns, and apply your labeling and filtering policies at that layer. Whatever upstream teams send, your rules run at the gateway before data reaches Grafana Cloud.\n\n[Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/) helps you run this pattern at scale by giving you centralized control over your collector configurations, so you can roll out and update gateway policies from one place instead of editing collectors one by one." }, { "type": "markdown", - "content": "You now have the prevention rules: bounded values only, add labels only when they will be used in queries, audit defaults at ingestion setup, and enforce policy at a layer you control. Next, you'll fix cardinality you already have." + "content": "You now have the prevention rules. Bounded values only, add labels only when they will be used in queries, audit defaults at ingestion setup, and enforce policy at a layer you control. Next, you'll fix cardinality you already have." }, { "type": "markdown", From f5ae3ede2b808309f19899e26bca81726e731c03 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Mon, 27 Jul 2026 10:27:49 -0400 Subject: [PATCH 16/17] add exemplars --- labeling-strategy-lj/content.json | 2 +- labeling-strategy-lj/correlate-signals/content.json | 8 ++++++-- labeling-strategy-lj/end-journey/content.json | 2 +- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/labeling-strategy-lj/content.json b/labeling-strategy-lj/content.json index 01e7ea0e..fcddd930 100644 --- a/labeling-strategy-lj/content.json +++ b/labeling-strategy-lj/content.json @@ -7,7 +7,7 @@ }, { "type": "markdown", - "content": "## Here's what to expect\n\nWhen you complete this path, you'll be able to:\n\n- Explain how label combinations multiply into unique active series and log streams\n- Inspect your own cardinality with the Cardinality tab in the Cost Management and Billing app\n- Choose metric labels deliberately and avoid unbounded values\n- Reduce existing metrics cardinality with relabeling and Adaptive Metrics\n- Decide when log data belongs in a label, in structured metadata, or in the log line\n- Move high-cardinality context to the right signal instead of deleting it" + "content": "## Here's what to expect\n\nWhen you complete this path, you'll be able to:\n\n- Explain how label combinations multiply into unique active series and log streams\n- Inspect your own cardinality with the Cardinality tab in the Cost Management and Billing app\n- Choose metric labels deliberately and avoid unbounded values\n- Reduce existing metrics cardinality with relabeling and Adaptive Metrics\n- Decide when log data belongs in a label, in structured metadata, or in the log line\n- Move high-cardinality context to the right signal instead of deleting it\n- Link a metric spike to the exact trace with exemplars, without adding labels" }, { "type": "markdown", diff --git a/labeling-strategy-lj/correlate-signals/content.json b/labeling-strategy-lj/correlate-signals/content.json index 28993a5d..12b9ea47 100644 --- a/labeling-strategy-lj/correlate-signals/content.json +++ b/labeling-strategy-lj/correlate-signals/content.json @@ -15,7 +15,11 @@ }, { "type": "markdown", - "content": "## Why this works\n\nEach signal is priced and indexed for a different job. Metrics are cheap per series but multiply by label combinations. Logs carry arbitrary detail, and structured metadata makes high-cardinality fields queryable without indexing them. Putting each value where it belongs is what a labeling strategy actually is; not fewer data points, just the right home for each one." + "content": "## Take it further with traces\n\nIf you also collect traces, the same pattern extends to a third signal using [exemplars](https://grafana.com/docs/grafana/latest/fundamentals/exemplars/). An exemplar is a trace ID attached alongside a metric sample, not as a label, so the metric stays low-cardinality while a latency spike becomes one click away from the exact slow trace.\n\n- **Attach the trace ID to the metric as an exemplar**, not a label.\n- **Keep the trace ID on logs as structured metadata**, so logs link to the same trace.\n\nExemplars require a tracing backend such as Tempo, and exemplar support enabled on your metrics data source." + }, + { + "type": "markdown", + "content": "## Why this works\n\nEach signal is priced and indexed for a different job. Metrics are cheap per series but multiply by label combinations. Logs carry arbitrary detail, and structured metadata makes high-cardinality fields queryable without indexing them. Traces hold per-request detail natively, and exemplars link them to your metrics without adding labels. Putting each value where it belongs is what a labeling strategy actually is; not fewer data points, just the right home for each one." }, { "type": "markdown", @@ -23,7 +27,7 @@ }, { "type": "markdown", - "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [What is structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/)\n- [Use telemetry signals together](https://grafana.com/docs/grafana-cloud/telemetry-signals/use-signals-together/key-concepts/)" + "content": "---\n\n### More to explore (optional)\n\nAt this point in your journey, you can explore the following topics:\n\n- [What is structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/)\n- [Use telemetry signals together](https://grafana.com/docs/grafana-cloud/telemetry-signals/use-signals-together/key-concepts/)\n- [Introduction to exemplars](https://grafana.com/docs/grafana/latest/fundamentals/exemplars/)" } ] } \ No newline at end of file diff --git a/labeling-strategy-lj/end-journey/content.json b/labeling-strategy-lj/end-journey/content.json index 256bb2af..c13ff12e 100644 --- a/labeling-strategy-lj/end-journey/content.json +++ b/labeling-strategy-lj/end-journey/content.json @@ -8,7 +8,7 @@ }, { "type": "markdown", - "content": "Throughout this journey, you have:\n\n- Learned how label combinations multiply into unique active series and log streams, and that this is what you're billed for\n- Explored your own stack's cardinality with the Cardinality tab in the Cost Management and Billing app\n- Learned the rules for metric labels: bounded values only, added only when queried, and audited at ingestion setup\n- Reviewed Adaptive Metrics recommendations to reduce cardinality you already have\n- Applied the logs decision framework: labels for origin, structured metadata for high-cardinality fields, filter expressions for the rest\n- Learned to correlate signals so high-cardinality detail lives on logs, not metric labels" + "content": "Throughout this journey, you have:\n\n- Learned how label combinations multiply into unique active series and log streams, and that this is what you're billed for\n- Explored your own stack's cardinality with the Cardinality tab in the Cost Management and Billing app\n- Learned the rules for metric labels: bounded values only, added only when queried, and audited at ingestion setup\n- Reviewed Adaptive Metrics recommendations to reduce cardinality you already have\n- Applied the logs decision framework: labels for origin, structured metadata for high-cardinality fields, filter expressions for the rest\n- Learned to correlate signals so high-cardinality detail lives on logs and traces, not metric labels" }, { "type": "markdown", From 465495771fe6bd548cc10d8d7d7e5d778b923fa7 Mon Sep 17 00:00:00 2001 From: Eve Meelan <81647476+Eve832@users.noreply.github.com> Date: Mon, 27 Jul 2026 10:30:15 -0400 Subject: [PATCH 17/17] moar colons --- labeling-strategy-lj/understand-cardinality/content.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/labeling-strategy-lj/understand-cardinality/content.json b/labeling-strategy-lj/understand-cardinality/content.json index 237c1f22..3474ebc6 100644 --- a/labeling-strategy-lj/understand-cardinality/content.json +++ b/labeling-strategy-lj/understand-cardinality/content.json @@ -25,7 +25,7 @@ }, { "type": "markdown", - "content": "## Cardinality affects cost and performance\n\nHigh cardinality affects both your bill and query performance, but metrics and logs are billed differently.\n\n- **Metrics cost**: Grafana Cloud bills metrics on active series, so every unique label combination is a billing unit, whether or not anyone queries it.\n- **Logs cost**: Grafana Cloud bills logs on GB processed and ingested, not per stream. High stream cardinality doesn't add a direct per-stream charge, but it can inflate query costs because larger indexes and more chunks mean bigger scans.\n- **Performance**: for logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." + "content": "## Cardinality affects cost and performance\n\nHigh cardinality affects both your bill and query performance, but metrics and logs are billed differently.\n\n- **Metrics cost.** Grafana Cloud bills metrics on active series, so every unique label combination is a billing unit, whether or not anyone queries it.\n- **Logs cost.** Grafana Cloud bills logs on GB processed and ingested, not per stream. High stream cardinality doesn't add a direct per-stream charge, but it can inflate query costs because larger indexes and more chunks mean bigger scans.\n- **Performance.** For logs, many streams mean a large index and many small chunks, which slows queries down. Loki is designed for low-cardinality labels and limits index labels to 15 by default.\n\nHigh-cardinality attributes to watch for include user IDs, request or trace IDs, IP addresses, Kubernetes pod names, and timestamps. Any of these used as a label value creates a new series or stream for every unique value." }, { "type": "markdown",