TraceFlow

TraceFlow is a Go observability backend for distributed traces. It accepts standard OTLP/gRPC span traffic, aggregates traces through a sharded in-memory index, applies integrated tail sampling, persists accepted spans to ClickHouse, exposes trace list/detail/stat APIs, and ships with a browser waterfall UI.

The project is benchmarked with standard ecosystem tools: OpenTelemetry Collector Contrib telemetrygen for ingestion and clickhouse-benchmark for query latency. No performance claim depends on a custom span generator.

Highlights

• Standard OTLP/gRPC TraceService receiver on port 4317.
• Legacy JSON span API for local demo workflows.
• 64-shard in-memory trace index for low-contention ingestion.
• Integrated tail-decision buffer for keeping errors and slow traces while sampling healthy traffic.
• Batched ClickHouse persistence with acknowledgement-preserving group commit.
• ClickHouse-backed trace list, trace detail, and service statistics queries.
• Prometheus metrics for accepted spans, collector errors, and runtime state.
• Browser waterfall UI for local inspection.
• Reproducible benchmark evidence directories with generator logs, collector logs, manifest metadata, and pass/fail claim files.

Verified performance

Ingestion

Reference run: benchmark-results/20260622T163246Z

Metric	Result
Harness	OpenTelemetry telemetrygen v0.154.0
Transport	OTLP/gRPC insecure
Duration	10 seconds
Client processes	7
Workers per process	8
Offered load	560,000 spans/sec
Generated spans	1,383,300
Accepted spans	1,371,166
Accepted throughput	128,833.35 spans/sec
Delivery ratio	99.1228%
Collector errors	0

Claim gate:

Gate	Result
Sustained >= 80,000 accepted spans/sec	Passed
>= 99% telemetrygen-to-collector delivery	Passed
Collector errors == 0	Passed

ClickHouse trace lookup

Reference run: benchmark-results/clickhouse-20260624T175147Z

Metric	Result
Harness	clickhouse-benchmark
Dataset	30,000,000 spans
Time range	30 days
Query	trace-by-ID lookup
Iterations	1,000
Concurrency	4
p50 latency	5 ms
p95 latency	7 ms
p99 latency	9 ms
p99.9 latency	42 ms
QPS	621.1

Claim gate:

Gate	Result
p99 trace lookup < 10 ms over 30M spans / 30 days	Passed

Concurrency 8 is not claimed on this laptop. The retained concurrency-8 run reached higher QPS but measured 13 ms p99, which fails the sub-10 ms latency gate.

Architecture

flowchart LR
    subgraph Producers
        SDK["OpenTelemetry SDKs"]
        TG["telemetrygen"]
        Demo["demo services"]
    end

    SDK -->|"OTLP/gRPC"| OTLP["TraceService receiver"]
    TG -->|"OTLP/gRPC"| OTLP
    Demo -->|"traceparent + JSON demo API"| HTTP["HTTP collector API"]

    OTLP --> Normalize["span normalization"]
    HTTP --> Normalize
    Normalize --> Tail["tail-decision buffer"]
    Tail --> Memory[("64-shard memory index")]
    Tail --> ClickHouse[("ClickHouse MergeTree store")]
    ClickHouse --> Query["trace query API"]
    Memory --> Query
    Query --> UI["waterfall UI"]
    Normalize --> Metrics["Prometheus metrics"]

Loading

Ingestion flow:

1. Producers send spans over OTLP/gRPC or the demo JSON endpoint.
2. TraceFlow normalizes resource, scope, trace, span, status, timing, service name, and scalar attributes into its query model.
3. The tail sampler keeps priority traces and deterministically samples healthy traffic.
4. Accepted spans are written to the memory index and, when configured, ClickHouse.
5. Query APIs serve trace lists, trace details, and aggregate service statistics.

Quickstart

Requirements:

• Go 1.24 or newer
• Docker, if using ClickHouse through Compose

Run locally with the in-memory backend:

go test ./...
go run ./cmd/collector -addr :9411 -otlp-grpc-addr :4317

Open:

• UI: http://127.0.0.1:9411/
• OTLP/gRPC: 127.0.0.1:4317
• trace API: http://127.0.0.1:9411/api/traces
• stats API: http://127.0.0.1:9411/api/stats
• metrics: http://127.0.0.1:9411/metrics

Run with ClickHouse:

docker compose up --build

Compose starts the collector and ClickHouse with persistent storage. It exposes:

• 4317 for OTLP/gRPC;
• 9411 for the UI and HTTP APIs;
• 8123 for ClickHouse HTTP;
• 9000 for ClickHouse native protocol.

Sending spans

OTLP/gRPC is the benchmarked ingestion path. The HTTP endpoint remains useful for local demos:

curl -X POST http://127.0.0.1:9411/api/spans \
  -H 'content-type: application/json' \
  -d '{"spans":[{"trace_id":"trace-1","span_id":"span-1","service":"checkout","operation":"checkout.request","start_unix_ms":1710000000000,"duration_ms":42,"status":"ok"}]}'

Demo workflow:

./scripts/start_collector.sh
./scripts/start_demo_services.sh
./scripts/demo.sh
./scripts/stop.sh

The demo propagates W3C traceparent across a checkout flow:

checkout.request
├── inventory.lookup
└── payment.charge

Reproducing ingestion evidence

Windows PowerShell:

.\bench\telemetrygen\run.ps1

Linux:

bash bench/telemetrygen/run.sh

The harness:

• installs or reuses telemetrygen v0.154.0;
• starts a clean collector;
• launches seven official telemetrygen processes;
• records every generator and collector log;
• reads accepted-span and error counters from the collector;
• computes delivery against telemetrygen output;
• writes claim-verification.json.

The gate requires at least 80,000 accepted spans/sec, at least 99% delivery, and zero collector errors.

Reproducing ClickHouse evidence

.\bench\clickhouse\run.ps1 -Rows 30000000 -Iterations 1000 -Concurrency 4

The query gate uses ClickHouse's own benchmark client against a 30-day synthetic span dataset. The retained passing run is benchmark-results/clickhouse-20260624T175147Z/.

Tail sampling

Tail sampling is configured on the collector process:

go run ./cmd/collector \
  -tail-sample-rate 0.06 \
  -tail-slow-threshold-ms 500 \
  -tail-decision-wait 2s

The end-to-end test sends 10,200 traces through the processor:

• 100 error traces retained;
• 100 slow traces retained;
• 9,376 of 10,000 healthy traces dropped;
• healthy-drop rate: 93.76%.

Storage model

The ClickHouse backend stores spans in a MergeTree table partitioned by month and ordered for trace lookup. The schema keeps core identifiers, service/operation names, timestamps, duration, status, scalar attributes, and a 90-day retention window.

Set these variables to use ClickHouse from a custom deployment:

TRACEFLOW_CLICKHOUSE_URL=http://127.0.0.1:8123
TRACEFLOW_QUERY_BACKEND=clickhouse

Repository layout

cmd/collector/          collector process and runtime flags
internal/otlp/          standard TraceService receiver and conversion
internal/collector/     HTTP APIs, UI, and metrics
internal/store/         memory index and ClickHouse read/write backends
internal/sampler/       trace-completion buffer and deterministic tail policy
pkg/traceflow/          demo tracing SDK and traceparent propagation
bench/telemetrygen/     official ingestion benchmark and evidence gate
bench/clickhouse/       ClickHouse query-latency benchmark
scripts/                local demo workflow
web/                    browser waterfall UI assets

Scope

TraceFlow is a production-style observability systems project, not a managed tracing service. It does not yet include multi-tenant authentication, OpenTelemetry Collector pipeline plugins, distributed sharding across multiple collector nodes, cold-storage tiering, or hosted operations. The verified claims are limited to the retained local benchmark evidence and disclosed configuration.