upstream changes#4
Open
mikehash wants to merge 1021 commits into
Open
Conversation
Make the Grafana dashboard a derived artifact of the same source of truth
as the Go metrics, so metric names can never drift between Blockbook and
its dashboard.
- configs/grafana_template.json: the dashboard with every Blockbook metric
name in a PromQL expr replaced by a {{name:<key>}} placeholder keyed by
the stable logical id from configs/metrics.yaml. This is now the hand-
maintained source; the Grafana UI is for preview only.
- contrib/scripts/render_grafana.py: expands {{name:<key>}} / {{help:<key>}}
from metrics.yaml into configs/grafana.json (the file imported into
Grafana). --check fails on staleness for CI / pre-commit.
- configs/grafana.json: the rendered, committed artifact.
- contrib/scripts/templatize_grafana.py: bootstrap/import helper that turns
a Grafana UI export back into the template.
- contrib/scripts/verify_grafana_render.py: round-trip check.
- configs/grafana.md: workflow docs.
Verified: the template carries no literal blockbook_ names (80 exprs fully
placeholdered, 60/60 metrics covered) and render(template) reproduces the
original dashboard exactly. Panel descriptions are left author-curated;
{{help:<key>}} is opt-in because one metric drives several panels with
different descriptions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- render_grafana.py: read metrics/panels/template via context managers instead of leaking file descriptors from bare open() calls - common/metrics.go: detect a duplicate `metric:"<key>"` tag up front and fail with an actionable error, instead of the indirect "unused definition" error that surfaced later Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ection rejections Two websocket abuse-observability metrics, with dashboard panels in the Websocket row, rendered through the metrics.yaml -> template.json/panels.yaml pipeline. - websocket_connection_requests (histogram): requests handled per connection, recorded at disconnect. The p99-vs-p50 spread reveals whether a single connection floods messages, which is the data needed to decide if per-connection message rate limiting is warranted. Until now only the aggregate websocket_requests rate was visible, so one noisy socket was lost among the legitimate Suite connections. - websocket_connection_rejections (counter_vec by reason): connections the per-IP limiter rejects before upgrade (connection_limit / connection_attempt _limit). The limiter previously only logged a warning, so connection-flood abuse being blocked was invisible in Grafana. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n clustering Add two gauges exposing how the open websocket connections are spread across client IPs, the missing signal for spotting a single non-Suite source clustering many connections behind the per-IP limiter: - websocket_unique_ips: distinct IPs currently holding a connection - websocket_max_connections_per_ip: size of the largest single cluster The per-IP state already lives in the connection limiter's clients map, so this only reads its cardinality once per minute from the maintenance loop (renamed from runPeriodicCleanup, now server-owned so it can publish metrics). No per-request cost, and no IP is ever exported as a label, only counts, so there is no cardinality blow-up and no PII in metrics. Two dashboard panels render from configs/metrics.yaml: connections vs unique IPs (the gap between the lines is clustering) and connections per IP (average and busiest). websocket_clients / websocket_unique_ips is the average connections per IP; an average near 1 is a healthy spread across users, while a rising average or a max approaching the 128 per-IP cap points to one source concentrating connections. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
websocket_unique_ips is 0 at startup/idle and only refreshes once a minute, so the average-connections-per-IP query (websocket_clients / websocket_unique_ips) produced NaN (0/0) at idle and +Inf (n/0) in the first minute after traffic resumed. Wrap the denominator in clamp_min(..., 1). clamp_min fixes both cases at the query layer: 0/1 = 0 at idle and n/1 = n (bounded) during the brief stale-gauge window. Updating the gauge on connect/disconnect would only close the staleness window, not the idle 0/0, and would add per-event cardinality recomputation to the hot path the once-a-minute sweep deliberately keeps clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(tickers): verify /api/v2/tickers freshness and positive rates * tests(e2e): add websocket fiat rates tests * tests(e2e): add websocket tests - getblock, balancehistory, tx-specific * tests(e2e): add REST block-filters, balancehistory tests * tests(e2e): add flag to allow e2e tests on unsynced Blockbook * tests(e2e): add machine-readable jUnit XML test report * tests(e2e): upload the jUnit XML e2e tests report in the CI * fix(e2e): block-filter ws test to allow empty filters * fix(e2e): current fiat rates to be maximally 5 minutes in the future * feat(makefile): e2e tests can be simply run locally for debugging reasons
…cate permanent buffer per connection
…very next page request
* update to zebra 5.0.0 * fix(zcash): migrate testnet backend to Zebra 5.0.0 for NU6.2 NU6.2 (consensus branch id 0x5437f330, testnet activation height 4052000) needs a NU6.2-aware node so Blockbook forwards the live consensus.chaintip instead of a stale value. zcashd reached end of support on 2025-05-31 and is being replaced by zebrad/zallet; the only NU6.2-capable zcashd build (6.20.0) ships no debian-bullseye tarball and publishes no sha256, so it cannot pass the existing sha256 build verification unattended. Migrate testnet to Zebra 5.0.0, mirroring the mainnet Zebra backend. Mirror the mainnet docker backend (zfnd/zebra:5.0.0, same registry-verified repo digest) with testnet values via a new zcash_testnet.conf that sets network = "Testnet" and the testnet p2p listen_addr 0.0.0.0:18233. Zebra has no ZeroMQ, so drop backend_message_queue and message_queue_binding_template; the empty binding makes Blockbook fall back to polling. Also drop the zcashd-only zcash-fetch-params postinst and addnode params, and switch the service to simple. Reindent to 4 spaces to match configs/coins/zcash.json. See: https://zfnd.org/zebra-4-5-3-and-5-0-0-emergency-soft-fork-and-nu6-2-activation/ https://github.com/ZcashFoundation/zebra/blob/616fa36ecc06e730f1e2accc9a45bca4eb08b561/CHANGELOG.md?plain=1#L22 https://github.com/ZcashFoundation/zebra/blob/616fa36ecc06e730f1e2accc9a45bca4eb08b561/zebra-chain/src/parameters/network_upgrade.rs#L234 https://github.com/zcash/zcash/blob/v6.20.0/src/chainparams.cpp#L536 * chore(zcash): drop dead backend_message_queue port from mainnet config * chore(zcash-testnet): add additional params from the zcash-mainnet --------- Co-authored-by: Jakub Jerabek <xjerab28@stud.fit.vutbr.cz>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GitHub expression interpolation happens before the shell parses a `run:` line, so `$(...)` / `;cmd;` in the workflow_dispatch `branch_or_tag` input would execute in the prepare jobs *before* validate_branch_or_tag.py ever inspects the value. Pass RESOLVED_BRANCH_OR_TAG (and the trusted repo) via ordinary shell env vars instead, so validation runs against the raw value and unusual-but-legal ref names no longer break the command. Also route inputs.backend_mode and the (allowlisted) coin list through env vars in the build job rather than interpolating them directly into the shell. COINS stays unquoted to preserve the intended per-coin word split. Addresses H3 of the CI/CD security review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A moving tag (`@v4`) lets a compromised or retagged action run with the trust of our self-hosted runners. Pin every third-party action to a full commit SHA (with a trailing `# vX.Y.Z` for readability): actions/checkout -> 34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 actions/setup-node -> 49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 actions/cache -> 0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 actions/upload-artifact -> ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 Pin PyYAML (.github/requirements.txt) so an unexpected release cannot change CI behaviour, and add .github/dependabot.yml for the github-actions and pip ecosystems so these pins are bumped deliberately via PRs. Addresses M4 of the CI/CD security review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
testing.yml declared no workflow-level permissions (inheriting the repo
default, often read/write) and the unit-tests job — which runs `make test`
(building the Docker image and executing PR Makefile/Go code with
--network=host and a source bind-mount) plus the Grafana render scripts —
had no same-repo guard, unlike connectivity/integration. It therefore
relied solely on the repo "require approval for fork PR workflows" setting
to keep untrusted fork code off the sudo-capable self-hosted runner.
* add `permissions: contents: read` at the workflow level;
* add the same same-repo guard the other jobs use to unit-tests, so fork
PRs get no automatic execution on the self-hosted runner;
* set `persist-credentials: false` on every checkout (none of these jobs
push), so the token is not left in the runner's git config.
Moving unit-tests to an ephemeral GitHub-hosted runner (the report's
preferred option) is an infrastructure decision left to the maintainer.
Addresses H1 of the CI/CD security review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Several backend artifacts were fetched from mutable sources or without any
integrity check, so an upstream edit could change package contents (these
packages are then installed and their services restarted on dev hosts):
* omotenashicoin{,_testnet}: binary fetched from a `…/raw/master/…` path
with empty verification fields -> pin to commit 03bde41 and add sha256.
* tron, tron_testnet_nile: extract_command fetched the node config from a
mutable `master`/`refs/heads/master` branch -> pin to a commit SHA and
verify the downloaded .conf with sha256 before use.
* optimism_archive_legacy_geth: source zip from `archive/refs/heads/develop`
-> pin to commit 8205f67, refresh sha256, and fix the extracted source
dir referenced by postinst (was the stale `optimism-legacy-devlop`).
* divi, deeponion: release zip/tarball with no verification -> add sha256.
All checksums were computed by downloading the pinned artifacts.
nuls and unobtanium stay on plaintext HTTP: both are already sha256-verified
(integrity-protected) and neither upstream serves the file over HTTPS. The
new lint flags them as warnings rather than failing.
Add .github/scripts/validate_backend_artifacts.py and a `backend-artifact-lint`
job (ubuntu-latest, no secrets) that FAILS on mutable sources, missing
integrity, or HTTP-without-integrity, and WARNS on HTTP-with-checksum.
Addresses M2 of the CI/CD security review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The build image is rebuilt automatically on CI whenever build/docker
changes, so unpinned toolchain fetches are a supply-chain exposure:
* Verify the Go tarball against its published sha256 after download
(per-arch: amd64/arm64), so a tampered or swapped go$VERSION archive
fails the build instead of silently shipping.
* Pin RocksDB to the immutable commit behind the v9.10.0 tag and assert
`git rev-parse HEAD` matches it, so a moved tag is detected. (Note:
`clone -b <tag>` checks out the commit the annotated tag dereferences
to, not the tag object, hence the commit hash below.)
Checksums/commit were verified by downloading the artifacts:
go1.25.4 linux-amd64 9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec
go1.25.4 linux-arm64 a68e86d4b72c2c2fecf7dfed667680b6c2a071221bbdb6913cf83ce3f80d9ff0
rocksdb v9.10.0 ae8fb3e5000e46d8d4c9dbf3a36019c0aaceebff
The base image digest and the dynamically-versioned Docker CLI tarball are
not pinned here (BASE_IMAGE/DOCKER_VERSION are derived per-host at build
time); those are noted as residual items for the maintainer.
Addresses M3 of the CI/CD security review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deploy pipeline's sync gate connected with an unconditional
ssl._create_unverified_context(), so the "is it synced?" check could be
satisfied over a MITM'd connection.
Make verification opt-in instead of hard-disabled:
* SYNC_CA_FILE=<path> verifies the chain against an internal CA bundle
(preferred for internal/self-signed dev hosts),
* SYNC_TLS_INSECURE=0 verifies against the system trust store,
* otherwise verification stays disabled (default, preserves
the prior behavior for self-signed dev certs).
The selected mode is logged once at startup so a relaxed gate is visible
in the run logs. Default behavior is unchanged, so existing dev deploys
are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-run helper (not invoked by CI). Robustness cleanups from the
security review:
* add `set -euo pipefail`,
* fix the `${UPDATE_VENDOR:0}` typo (substring expansion) to the
intended `${UPDATE_VENDOR:-0}` default,
* quote all expansions and carry the coin list in an array so unusual
args/hostnames don't word-split,
* track per-coin status separately so a mid-loop failure isn't masked
by a later success, and use POSIX `=` instead of the `==` bashism.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.