fix(price-pusher): provision Loki + Promtail in sample monitoring stack#3692
fix(price-pusher): provision Loki + Promtail in sample monitoring stack#36920xghost42 wants to merge 2 commits into
Conversation
Closes pyth-network#2520. The bundled grafana-dashboard.sample.json references a Loki data source (uid: ads9ouz3jh4hsa) for three panels — Tx Hash, All Logs, Error Logs — and uses it to populate the `chain` template variable. The sample stack only provisioned Prometheus, so on a fresh docker-compose up the dashboard rendered with an unconfigured-data-source banner across all log panels and an empty chain selector. This change makes the sample stack actually run the dashboard as-is: - datasource.sample.yml: add a Loki entry pinned to the UID the dashboard already expects. - docker-compose.metrics.sample.yaml: add a Loki service and a Promtail sidecar that ships Docker container logs into Loki. - promtail.sample.yml: new file. Scrapes container logs via the Docker socket and labels each stream with `namespace=<container_name>`, which is exactly the label the dashboard's `{namespace=~"$chain"}` filter selects on. JSON pino logs are parsed so that the Error Logs panel's `detected_level = error` matcher works against pino's level field. - README: update the 'Running Locally with Docker' section to describe the new services and the one-container-per-chain pattern the dashboard expects.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 7 Skipped Deployments
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🚩 Dashboard log queries use | logfmt on JSON-formatted pino output
The pre-existing grafana-dashboard.sample.json uses | logfmt as the first parser in several Loki queries (e.g. the All Logs panel at line 1148: {namespace=~"$chain"} | logfmt, and the Tx Hash panel at line 1098 which chains | logfmt | json). Since pino outputs JSON (not logfmt), the logfmt parser will fail to extract any fields. In the Tx Hash panel the subsequent | json stage recovers by parsing the original JSON line, so it still works — but the logfmt step is redundant. In the All Logs panel, | logfmt alone means no structured fields are extracted, though log lines still display. This is a pre-existing issue in the dashboard (not introduced by this PR), but since this PR adds the infrastructure (Loki/Promtail) that makes these panels actually functional for the first time, it may be worth addressing the dashboard queries in a follow-up.
Was this helpful? React with 👍 or 👎 to provide feedback.
Follow-ups from review feedback on the sample monitoring stack: - promtail.sample.yml: add a `template` stage between `json` and `labels` that maps pino's numeric severities (10/20/30/40/50/60) to the string forms Loki's `detected_level` recognises. Without this the Error Logs panel's `detected_level = error` filter never matches because pino emits `level=50`, not `level=error`. - datasource.sample.yml: set an explicit `uid: edryyydtht14wa` on the Prometheus datasource so dashboards that hardcode the UID resolve in a fresh provision. Previously only the Loki UID was pinned, leaving every Prometheus-backed panel to rely on Grafana auto-assignment.
|
Pushed
I left the |
There was a problem hiding this comment.
🚩 Prometheus metrics lack namespace label but dashboard queries filter on it
The Grafana dashboard (pre-existing, not changed in this PR) filters all Prometheus queries with namespace="$chain", e.g. pyth_price_feeds_total{namespace="$chain"}. However, the price pusher's metrics code (src/metrics.ts:35-89) does not include a namespace label on any metric, and prometheus.sample.yml doesn't add one via relabeling. This means in the sample docker-compose setup, Prometheus metric panels will show no data when a specific chain is selected. This is a pre-existing issue — likely the dashboard was designed for a Kubernetes environment where Prometheus adds namespace labels from service discovery. The chain template variable is now sourced from Loki's namespace label (container names), which further highlights the disconnect. Users following the sample setup should be aware that metric panels may not populate without adding a matching namespace label to their Prometheus scrape config.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
Acknowledged on the namespace label finding — confirmed it's a pre-existing dashboard assumption: Treating it the same as the
Happy to bundle that into a separate dashboard-cleanup PR if you'd like — kept this one narrow so the Loki/Promtail provisioning lands cleanly first. |
1 participant
Summary
Closes #2520.
The bundled
grafana-dashboard.sample.jsonreferences a Loki data source (uid: ads9ouz3jh4hsa) for three panels — Tx Hash, All Logs, Error Logs — and uses it to populate thechaintemplate variable. The sample stack only provisioned Prometheus, so on a freshdocker-compose upthe dashboard renders with an unconfigured-data-source banner across all log panels and an empty chain selector. This is exactly what the issue reporter screenshot showed.Change
Make the sample stack actually run the bundled dashboard as-is.
datasource.sample.yml: add a Loki entry pinned to the UID the dashboard already expects (ads9ouz3jh4hsa).docker-compose.metrics.sample.yaml: add a Loki service and a Promtail sidecar that ships Docker container logs into Loki.promtail.sample.yml(new): scrapes container logs via the Docker socket and labels each stream withnamespace=<container_name>— exactly what the dashboard's{namespace=~"$chain"}filter selects on. JSON pino logs are parsed so the Error Logs panel'sdetected_level = errormatcher works against pino'slevelfield.README.md: update the Running Locally with Docker section to describe the new services and the one-container-per-chain pattern the dashboard expects.Verification
docker-compose -f docker-compose.metrics.sample.yaml configis valid YAML and resolves all four services.promtail -check-config -config.file=promtail.sample.ymlparses cleanly (docker_sd_configs,relabel_configs,pipeline_stagesall accepted).pnpm turbo fix --filter @pythnetwork/price-pusheris clean (biome formatting only touches built artefacts).docker-compose -f docker-compose.metrics.sample.yaml up, the imported dashboard now lists every pusher container's name in thechainselector and the log panels populate as soon as the pusher emits a line.Out of scope
I kept the Loki/Promtail/Grafana versions on
:latestto match the existing Prometheus/Grafana pin convention in the same file. Happy to pin to specific tags in a follow-up if the team prefers reproducible versions.