Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -109,11 +109,7 @@ Use the `edit-config` script to safely edit configuration files. It automaticall

3. Define recipients per **role** (see below).

4. Restart the Agent for changes to take effect:

```bash
sudo systemctl restart netdata
```
4. Changes take effect immediately. The next alarm will use the updated configuration, so no restart or reload is required.

## Example: Alert with Role-Based Routing

Expand Down Expand Up @@ -495,7 +491,7 @@ Here are solutions for common alert notification issues:
1. Verify your service key:

```bash
grep -E "PAGERDUTY_SERVICE_KEY" /etc/netdata/health_alarm_notify.conf
grep -E "DEFAULT_RECIPIENT_PD|SEND_PD" /etc/netdata/health_alarm_notify.conf
```

2. Test the PagerDuty API directly:
Expand All @@ -508,9 +504,7 @@ Here are solutions for common alert notification issues:

If notifications seem delayed:

1. Check the `delay` parameter in your alarm configuration
2. Verify your `health.d/*.conf` files for delay settings
3. Check the `ALARM_NOTIFY_DELAY` setting in health_alarm_notify.conf
- Check the `delay` parameter in your alarm definitions in `health.d/*.conf` — this is the only per-alert delay setting for notifications; there is no global delay setting in `health_alarm_notify.conf`.

## Related Docs

Expand Down
2 changes: 1 addition & 1 deletion docs/Exporting Metrics/Connectors/AWS Kinesis.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ The following options can be defined for this exporter.
| password | Password for HTTP authentication | my_password | no |
| data source | Selects the kind of data that will be sent to the external database. (as collected/average/sum) | | no |
| hostname | The hostname to be used for sending data to the external database server. | [global].hostname | no |
| prefix | The prefix to add to all metrics. | Netdata | no |
| prefix | The prefix to add to all metrics. | netdata | no |
| [update every](#option-update-every) | Frequency of sending sending data to the external database, in seconds. | 10 | no |
| [buffer on failures](#option-buffer-on-failures) | The number of iterations (`update every` seconds) to buffer data, when the external database server is not available. | 10 | no |
| timeout ms | The timeout in milliseconds to wait for the external database server to process the data. | 2 * update_every * 1000 | no |
Expand Down
2 changes: 1 addition & 1 deletion docs/Exporting Metrics/Connectors/Google Cloud Pub Sub.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ The following options can be defined for this exporter.
| password | Password for HTTP authentication | my_password | no |
| data source | Selects the kind of data that will be sent to the external database. (as collected/average/sum) | | no |
| hostname | The hostname to be used for sending data to the external database server. | [global].hostname | no |
| prefix | The prefix to add to all metrics. | Netdata | no |
| prefix | The prefix to add to all metrics. | netdata | no |
| [update every](#option-update-every) | Frequency of sending sending data to the external database, in seconds. | 10 | no |
| [buffer on failures](#option-buffer-on-failures) | The number of iterations (`update every` seconds) to buffer data, when the external database server is not available. | 10 | no |
| timeout ms | The timeout in milliseconds to wait for the external database server to process the data. | 2 * update_every * 1000 | no |
Expand Down
2 changes: 1 addition & 1 deletion docs/Exporting Metrics/Connectors/JSON.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ The following options can be defined for this exporter.
| password | Password for HTTP authentication | my_password | no |
| data source | Selects the kind of data that will be sent to the external database. (as collected/average/sum) | | no |
| hostname | The hostname to be used for sending data to the external database server. | [global].hostname | no |
| prefix | The prefix to add to all metrics. | Netdata | no |
| prefix | The prefix to add to all metrics. | netdata | no |
| [update every](#option-update-every) | Frequency of sending sending data to the external database, in seconds. | 10 | no |
| [buffer on failures](#option-buffer-on-failures) | The number of iterations (`update every` seconds) to buffer data, when the external database server is not available. | 10 | no |
| timeout ms | The timeout in milliseconds to wait for the external database server to process the data. | 2 * update_every * 1000 | no |
Expand Down
2 changes: 1 addition & 1 deletion docs/Exporting Metrics/Connectors/MongoDB.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ The following options can be defined for this exporter.
| password | Password for HTTP authentication | my_password | no |
| data source | Selects the kind of data that will be sent to the external database. (as collected/average/sum) | | no |
| hostname | The hostname to be used for sending data to the external database server. | [global].hostname | no |
| prefix | The prefix to add to all metrics. | Netdata | no |
| prefix | The prefix to add to all metrics. | netdata | no |
| [update every](#option-update-every) | Frequency of sending sending data to the external database, in seconds. | 10 | no |
| [buffer on failures](#option-buffer-on-failures) | The number of iterations (`update every` seconds) to buffer data, when the external database server is not available. | 10 | no |
| timeout ms | The timeout in milliseconds to wait for the external database server to process the data. | 2 * update_every * 1000 | no |
Expand Down
2 changes: 1 addition & 1 deletion docs/Exporting Metrics/Connectors/OpenTSDB.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ The following options can be defined for this exporter.
| password | Password for HTTP authentication | my_password | no |
| data source | Selects the kind of data that will be sent to the external database. (as collected/average/sum) | | no |
| hostname | The hostname to be used for sending data to the external database server. | [global].hostname | no |
| prefix | The prefix to add to all metrics. | Netdata | no |
| prefix | The prefix to add to all metrics. | netdata | no |
| [update every](#option-update-every) | Frequency of sending sending data to the external database, in seconds. | 10 | no |
| [buffer on failures](#option-buffer-on-failures) | The number of iterations (`update every` seconds) to buffer data, when the external database server is not available. | 10 | no |
| timeout ms | The timeout in milliseconds to wait for the external database server to process the data. | 2 * update_every * 1000 | no |
Expand Down
56 changes: 55 additions & 1 deletion docs/Exporting Metrics/Exporting reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,60 @@ Netdata provides three data export modes:
| [Prometheus Remote Write](/docs/exporting-metrics/connectors/prometheus-remote-write) | Snappy-compressed protobuf | Binary over HTTP |
| [TimescaleDB](https://github.com/netdata/netdata/blob/master/src/exporting/TIMESCALE.md) | JSON streams | Time-series tables |

## How Metric Names Are Constructed

OpenTSDB and Graphite export dotted metric names. Prometheus scrape and remote write export context-based metric names with labels.

### Flat-name connectors: OpenTSDB and Graphite

These connectors join their name components with dots, so the chart and dimension are visible directly in the metric name:

```text
OpenTSDB: prefix.chart.dimension
Graphite: prefix.hostname.chart.dimension
```

- **OpenTSDB** sends the host as a separate tag (`host=...`).
- **Graphite** embeds the host in the dotted path: `prefix.hostname.chart.dimension`.

Example (OpenTSDB): `netdata.system.cpu.user` with a `host=myhost` tag.

OpenTSDB and Graphite preserve dots in chart and dimension names. They replace other non-alphanumeric characters with underscores.

### Prometheus exports: scrape and remote write

For homogeneous charts, both methods join the prefix and the chart **context** with underscores, and carry the chart, family, and dimension as labels:

```text
prefix_context{chart="...", family="...", dimension="..."}
```

- The chart **context** — the template shared by all charts of the same kind — is part of the metric name, not the individual chart ID or name.
- `average` appends the chart units and `_average`. The scrape endpoint can omit the units with `hideunits=yes`.
- `sum` appends `_sum` without the chart units.
- In `as-collected` mode, incremental and percentage-over-difference counters append `_total`. For charts produced by the Prometheus collector, Netdata does not append `_total`.
- For heterogeneous `as-collected` charts, the dimension moves into the metric name (`prefix_context_dimension`) and is omitted from the labels.
- Netdata sanitizes the context, units, and any dimension embedded in a Prometheus metric name, so dots and other unsupported characters become underscores. Chart, family, and dimension labels retain their label values. The configured prefix is used as provided.

Example (remote write): `netdata_system_cpu_percentage_average{chart="system.cpu", dimension="user", family="cpu", instance="myhost"}`.

For the complete naming rules — contexts, units, suffixes, and how to preview the exact metric names via the `allmetrics` endpoint — see the [Prometheus reference](/docs/exporting-metrics/prometheus).

### Quick comparison

| Aspect | OpenTSDB / Graphite | Prometheus scrape / remote write |
|:-------------------|:-------------------------------------|:-----------------------------------------------------------------------------------|
| OpenTSDB base | `prefix.chart.dimension` | `prefix_context` |
| Graphite base | `prefix.hostname.chart.dimension` | `prefix_context` |
| Dimension | In the metric name | Label; in the metric name for heterogeneous `as-collected` charts |
| Data-source suffix | None | `_total`, `_average`, or `_sum` when applicable |
| Units in name | No | `average` only |
| Host | Tag (OpenTSDB) or path (Graphite) | `instance` label for remote write and all-host scrape; otherwise the scrape target |
| Sanitization | Chart/dimension preserve dots | Context/units/embedded dimensions replace dots |
| Default prefix | `netdata` | `netdata` |

Both approaches respect the `send names instead of ids` setting: when enabled, Netdata uses human-friendly chart and dimension names; when disabled, it uses the raw system IDs. See the [OpenTSDB connector options](/docs/exporting-metrics/connectors/opentsdb) for the prefix and name settings.

## Configuration Structure

Your `exporting.conf` file contains these configuration blocks:
Expand All @@ -86,7 +140,7 @@ Your `exporting.conf` file contains these configuration blocks:
enabled = yes
destination = localhost:2003
data source = average
prefix = Netdata
prefix = netdata
hostname = my-name
update every = 10
buffer on failures = 10
Expand Down
12 changes: 6 additions & 6 deletions docs/Exporting Metrics/Prometheus.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Netdata tracks each Prometheus server's last access time to calculate averages f

Like `average` but sums values instead of averaging them.

**Format:** `CONTEXT_UNITS_sum{chart="CHART",family="FAMILY",dimension="DIMENSION"}`
**Format:** `CONTEXT_sum{chart="CHART",family="FAMILY",dimension="DIMENSION"}`

To change the data source, add the `source` parameter to the URL:

Expand Down Expand Up @@ -265,11 +265,11 @@ Or append `&prefix=netdata` to the URL.

### Metric Units

| Source | Unit Behavior | Control |
|:--------------------|:------------------------------------------|:------------------------------------|
| `average` (default) | Adds units to names (e.g., `_KiB_persec`) | `&hideunits=yes` to hide |
| `as-collected` | No units in names | N/A |
| All sources | v1.12+ standardized units | `&oldunits=yes` for pre-v1.12 names |
| Source | Unit Behavior | Control |
|:--------------------|:------------------------------------------|:--------------------------------------------------------------|
| `average` (default) | Adds units to names (e.g., `_KiB_persec`) | `&hideunits=yes` to hide; `&oldunits=yes` for pre-v1.12 names |
| `as-collected` | No units in names | N/A |
| `sum` | No units in names | N/A |

### Accuracy of Average and Sum Data Sources

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,12 +89,12 @@ Host labels help you:

Netdata automatically generates host labels when it starts, capturing:

| Label Category | Information Captured |
|----------------|-----------------------------------------------------|
| System Info | Kernel version, OS name and version |
| Label Category | Information Captured |
|----------------|---------------------------------------------------------------------------------------------|
| System Info | Kernel version, OS name and version |
| Hardware | CPU architecture, cores, frequency, RAM, disk space, product ID, product name, product type |
| Environment | Container details, Kubernetes node status |
| Infrastructure | Virtualization layer, Parent-child streaming status |
| Environment | Container details, Kubernetes node status |
| Infrastructure | Virtualization layer, Parent-child streaming status |

View your automatic labels at `http://HOST-IP:19999/api/v1/info`:

Expand All @@ -118,7 +118,10 @@ Add your own labels to categorize systems by any criteria you need.
```bash
cd /etc/netdata # Replace with your Netdata config directory
sudo ./edit-config netdata.conf
```
:::note

On Windows, follow [Edit Configuration Files](/docs/netdata-agent/configuration#edit-configuration-files) instead. The `[host labels]` example, naming rules, and environment variable expansion in steps 2 and 3 below work identically on Windows.
:::

2. Add a `[host labels]` section:

Expand All @@ -139,16 +142,16 @@ Add your own labels to categorize systems by any criteria you need.

```text
[host labels]
region = ${REGION}
rack = ${RACK:-unknown}
env = ${DEPLOYMENT_ENV:-production}
location = ${DC}-${RACK:-default}
region = $\{REGION}
rack = $\{RACK:-unknown}
env = $\{DEPLOYMENT_ENV:-production}
location = $\{DC}-$\{RACK:-default}
```

| Syntax | Behavior |
|--------|----------|
| `${VAR}` | Replaced with the value of `VAR`. If unset or empty, the label value becomes `[none]` |
| `${VAR:-default}` | Replaced with the value of `VAR`. If unset or empty, uses `default` |
| Syntax | Behavior |
|-------------------|---------------------------------------------------------------------------------------|
| `${VAR}` | Replaced with the value of `VAR`. If unset or empty, the label value becomes `[none]` |
| `${VAR:-default}` | Replaced with the value of `VAR`. If unset or empty, uses `default` |

Environment variables are resolved when labels are loaded or reloaded. You can mix them with literal text (e.g., `${DC}-${RACK}`).

Expand All @@ -158,10 +161,37 @@ Add your own labels to categorize systems by any criteria you need.
netdatacli reload-labels
```

On Windows, run this through `netdatacli.exe` — see [Using netdatacli](/docs/netdata-agent/maintenance/service-control#windows).

5. Verify your labels at `http://HOST-IP:19999/api/v1/info`

Use custom host labels such as `environment` with [Node Rule-Based Room Assignment](/docs/netdata-cloud/spaces-and-rooms/node-rule-based-room-assignment) to route Kubernetes Nodes into separate Rooms.

### Remove custom labels

Remove a custom label you no longer need.

1. Edit your Netdata configuration:

```bash
cd /etc/netdata # Replace with your Netdata config directory
sudo ./edit-config netdata.conf
```

On Windows, follow [Edit Configuration Files](/docs/netdata-agent/configuration#edit-configuration-files) instead.

2. In the `[host labels]` section, delete the line for the label you want to remove.
3. Save the file.
4. Apply the change without restarting Netdata:

```bash
netdatacli reload-labels
```

On Windows, run this through `netdatacli.exe` — see [Using netdatacli](/docs/netdata-agent/maintenance/service-control#windows).

5. Confirm the label is gone at `http://HOST-IP:19999/api/v1/info`

### Stream labels from Child to Parent

In Parent-Child setups, host labels automatically stream from children to the parent node. Access any child's labels through the parent at:
Expand Down
11 changes: 11 additions & 0 deletions docs/Netdata Agent/Logging.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,17 @@ journalctl MESSAGE_ID=9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de --since "1 hour ago"

The Netdata service's processes execute within the `netdata` journal namespace. Common queries:

:::note
If `journalctl` reports insufficient permissions for the `netdata` namespace, run the query with `sudo`.
For persistent access without `sudo`, add the user to the `systemd-journal` group, then log out and back in.
Membership grants read access to all local journal files, not only Netdata logs.

```bash
sudo usermod -aG systemd-journal "$USER"
```

:::

```bash
# Real-time log monitoring
journalctl -u netdata --namespace=netdata -f
Expand Down
1 change: 1 addition & 0 deletions docs/Netdata Agent/Maintenance/Service Control.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -160,5 +160,6 @@ The `netdatacli` tool is available on Windows at the Netdata installation path (
| **Stop** | `Stop-Service Netdata` | Task Manager > Services > Netdata | — |
| **Restart** | `Restart-Service Netdata` | Task Manager > Services > Netdata | — |
| **Reload Health** | — | — | `netdatacli reload-health` |
| **Reload Labels** | — | — | `netdatacli reload-labels` |
| **Shutdown** | — | — | `netdatacli shutdown-agent` |
| **Ping** | — | — | `netdatacli ping` |
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,20 @@ We recommend using Netdata Cloud to avoid receiving duplicate alert notification
<br/>
</details>

<details>
<summary><strong>Should I configure local collectors (like httpcheck) on both Parents in an HA cluster?</strong></summary><br/>

Local collectors such as `httpcheck` are synthetic checks that run on the Netdata node where they are configured. They do not run against data streamed from child nodes. Each Parent in a cluster is a full Netdata node, so a local collector only runs on the Parent where you configure it.

If you configure a collector on only one Parent and that Parent goes down, the check stops entirely until that Parent recovers, even though your child metrics continue flowing through the other Parent.

To make synthetic checks resilient to a Parent failure, configure the collector on both Parents. In an active-active cluster, each Parent also replicates its local collector metrics to the other, so both Parents maintain a complete copy.

Be aware that each Parent evaluates alerts for its own local collectors independently. Since each Parent is a separate Netdata node, the same check configured on both produces alerts from two different nodes. This differs from child-node alerts, which Netdata Cloud can deduplicate when multiple Parents evaluate the same child (see the question above on duplicate alert notifications). Consider this trade-off when deciding whether to run the same synthetic check on both Parents.

<br/>
</details>

<details>
<summary><strong>When I have only Parents connected to Netdata Cloud, will I be able to use the Functions feature on my child nodes?</strong></summary><br/>

Expand Down