Skip to content

Commit 00ce72c

Browse files
wrn14897wrn14897
authored
[HDX-4029] Add commonly-used core and contrib components to OTel Collector builder-config (#2121)
## Summary Update `packages/otel-collector/builder-config.yaml` to include commonly-used components from the upstream [opentelemetry-collector](https://github.com/open-telemetry/opentelemetry-collector) core and [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distributions. This gives users more flexibility in their custom OTel configs without pulling in the entire contrib distribution (which causes very long compile times). Also adds Go module and build cache mounts to the OCB Docker build stage for faster rebuilds, and bumps CI timeouts for integration and smoke test jobs to account for the larger binary. ### Core extensions added (2) - `memorylimiterextension` — memory-based limiting at the extension level - `zpagesextension` — zPages debugging endpoints ### Contrib receivers added (4) - `dockerstatsreceiver` — container metrics from Docker - `filelogreceiver` — tail log files - `k8sclusterreceiver` — Kubernetes cluster-level metrics - `kubeletstatsreceiver` — node/pod/container metrics from kubelet ### Contrib processors added (12) - `attributesprocessor` — insert/update/delete/hash attributes - `cumulativetodeltaprocessor` — convert cumulative metrics to delta - `filterprocessor` — drop unwanted telemetry - `groupbyattrsprocessor` — reassign resource attributes - `k8sattributesprocessor` — enrich telemetry with k8s metadata - `logdedupprocessor` — deduplicate repeated log entries - `metricstransformprocessor` — rename/aggregate/transform metrics - `probabilisticsamplerprocessor` — percentage-based sampling - `redactionprocessor` — mask/remove sensitive data - `resourceprocessor` — modify resource attributes - `spanprocessor` — rename spans, extract attributes - `tailsamplingprocessor` — sample traces based on policies ### Contrib extensions added (1) - `filestorage` — persistent file-based storage (used by clickhouse exporter sending queue in EE OpAMP controller) ### Other changes - **Docker cache mounts**: Added `--mount=type=cache` for Go module and build caches in the OCB builder stage of both `docker/otel-collector/Dockerfile` and `docker/hyperdx/Dockerfile` - **CI timeouts**: Bumped `integration` and `otel-smoke-test` jobs from 8 to 16 minutes in `.github/workflows/main.yml` All existing HyperDX-specific components are preserved unchanged. ### How to test locally or on Vercel 1. Build the OTel Collector Docker image — verify OCB resolves all listed modules 2. Provide a custom OTel config that uses one of the newly-added components and verify it loads 3. Verify existing HyperDX OTel pipeline still functions ### References - Linear Issue: https://linear.app/clickhouse/issue/HDX-4029 - Upstream core builder-config: https://github.com/open-telemetry/opentelemetry-collector/blob/main/cmd/otelcorecol/builder-config.yaml - Upstream contrib builder-config: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/otelcontribcol/builder-config.yaml
1 parent 06b4ac9 commit 00ce72c

6 files changed

Lines changed: 116 additions & 20 deletions

File tree

‎.changeset/otel-collector-add-core-extensions.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
'@hyperdx/otel-collector': minor
3+
---
4+
5+
feat: Add missing core extensions, commonly-used contrib processors/receivers, and filestorage extension
6+
7+
Add the two missing core extensions (memorylimiterextension, zpagesextension),
8+
12 commonly-used contrib processors (attributes, filter, resource, k8sattributes,
9+
tailsampling, probabilisticsampler, span, groupbyattrs, redaction, logdedup,
10+
metricstransform, cumulativetodelta), 4 commonly-used contrib receivers
11+
(filelog, dockerstats, k8scluster, kubeletstats), and the filestorage extension
12+
(used for persistent sending queue in the clickhouse exporter) to
13+
builder-config.yaml.

‎.github/workflows/main.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ jobs:
4747
- name: Run unit tests
4848
run: make ci-unit
4949
integration:
50-
timeout-minutes: 8
50+
timeout-minutes: 16
5151
runs-on: ubuntu-24.04
5252
steps:
5353
- name: Checkout
@@ -93,7 +93,7 @@ jobs:
9393
working-directory: ./packages/otel-collector
9494
run: go test ./...
9595
otel-smoke-test:
96-
timeout-minutes: 8
96+
timeout-minutes: 16
9797
runs-on: ubuntu-24.04
9898
steps:
9999
- name: Checkout

‎docker/hyperdx/Dockerfile‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,9 @@ ARG OTEL_COLLECTOR_CORE_VERSION
2929
COPY --from=ocb-bin /usr/local/bin/ocb /usr/local/bin/ocb
3030
WORKDIR /build
3131
COPY packages/otel-collector/builder-config.yaml .
32-
RUN sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
32+
RUN --mount=type=cache,target=/go/pkg/mod,id=ocb-gomod \
33+
--mount=type=cache,target=/root/.cache/go-build,id=ocb-gobuild \
34+
sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
3335
CGO_ENABLED=0 ocb --config=builder-config.yaml
3436

3537
# Build the Go migration tool with full TLS support for ClickHouse

‎docker/otel-collector/Dockerfile‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,9 @@ ARG OTEL_COLLECTOR_CORE_VERSION
1919
COPY --from=ocb-bin /usr/local/bin/ocb /usr/local/bin/ocb
2020
WORKDIR /build
2121
COPY packages/otel-collector/builder-config.yaml .
22-
RUN sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
22+
RUN --mount=type=cache,target=/go/pkg/mod,id=ocb-gomod \
23+
--mount=type=cache,target=/root/.cache/go-build,id=ocb-gobuild \
24+
sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
2325
CGO_ENABLED=0 ocb --config=builder-config.yaml
2426

2527
# Build the Go migration tool with full TLS support for ClickHouse

‎packages/otel-collector/README.md‎

Lines changed: 37 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
Custom-built OpenTelemetry Collector for HyperDX, compiled via
44
[OCB (OpenTelemetry Collector Builder)](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder).
55
This replaces the pre-built `otel/opentelemetry-collector-contrib` image with a
6-
binary that includes only the components HyperDX needs, plus any custom
7-
receivers/processors added in this package.
6+
binary that includes the components HyperDX needs plus commonly-used core and
7+
contrib components, and any custom receivers/processors added in this package.
88

99
## Architecture
1010

@@ -50,26 +50,44 @@ builds.
5050

5151
## Included components
5252

53-
All components referenced in config files and the OpAMP controller:
53+
Components used by HyperDX internally are marked with their config references.
54+
Components marked "user configs" are included so users can reference them in
55+
custom OTel configurations without rebuilding the collector.
5456

5557
### Receivers
5658

57-
| Component | Module | Used in |
58-
| --------------- | ------- | ------------------------------------------------- |
59-
| `nop` | core | OpAMP controller |
60-
| `otlp` | core | standalone configs, OpAMP controller, smoke tests |
61-
| `fluentforward` | contrib | standalone configs, OpAMP controller, smoke tests |
62-
| `hostmetrics` | contrib | custom.config.yaml |
63-
| `prometheus` | contrib | OpAMP controller, smoke tests |
59+
| Component | Module | Used in |
60+
| ---------------- | ------- | ------------------------------------------------- |
61+
| `nop` | core | OpAMP controller |
62+
| `otlp` | core | standalone configs, OpAMP controller, smoke tests |
63+
| `dockerstats` | contrib | user configs |
64+
| `filelog` | contrib | user configs |
65+
| `fluentforward` | contrib | standalone configs, OpAMP controller, smoke tests |
66+
| `hostmetrics` | contrib | custom.config.yaml |
67+
| `k8scluster` | contrib | user configs |
68+
| `kubeletstats` | contrib | user configs |
69+
| `prometheus` | contrib | OpAMP controller, smoke tests |
6470

6571
### Processors
6672

67-
| Component | Module | Used in |
68-
| ------------------- | ------- | ------------------------------------------------- |
69-
| `batch` | core | config.yaml, standalone configs, OpAMP controller |
70-
| `memory_limiter` | core | config.yaml, standalone configs, OpAMP controller |
71-
| `resourcedetection` | contrib | config.yaml |
72-
| `transform` | contrib | config.yaml, standalone configs, OpAMP controller |
73+
| Component | Module | Used in |
74+
| ---------------------- | ------- | ------------------------------------------------- |
75+
| `batch` | core | config.yaml, standalone configs, OpAMP controller |
76+
| `memory_limiter` | core | config.yaml, standalone configs, OpAMP controller |
77+
| `attributes` | contrib | user configs |
78+
| `cumulativetodelta` | contrib | user configs |
79+
| `filter` | contrib | user configs |
80+
| `groupbyattrs` | contrib | user configs |
81+
| `k8sattributes` | contrib | user configs |
82+
| `logdedup` | contrib | user configs |
83+
| `metricstransform` | contrib | user configs |
84+
| `probabilisticsampler` | contrib | user configs |
85+
| `redaction` | contrib | user configs |
86+
| `resourcedetection` | contrib | config.yaml |
87+
| `resource` | contrib | user configs |
88+
| `span` | contrib | user configs |
89+
| `tailsampling` | contrib | user configs |
90+
| `transform` | contrib | config.yaml, standalone configs, OpAMP controller |
7391

7492
### Exporters
7593

@@ -92,7 +110,10 @@ All components referenced in config files and the OpAMP controller:
92110

93111
| Component | Module | Used in |
94112
| ----------------- | ------- | ---------------------------------------- |
113+
| `memorylimiter` | core | user configs |
114+
| `zpages` | core | user configs |
95115
| `bearertokenauth` | contrib | standalone-auth config, OpAMP controller |
116+
| `file_storage` | contrib | OpAMP controller (sending queue storage) |
96117
| `health_check` | contrib | config.yaml, standalone-auth config |
97118
| `opamp` | contrib | used by OpAMP supervisor |
98119
| `pprof` | contrib | included for debugging/profiling |

‎packages/otel-collector/builder-config.yaml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,24 @@ receivers:
2727
go.opentelemetry.io/collector/receiver/otlpreceiver
2828
v__OTEL_COLLECTOR_VERSION__
2929
# Contrib
30+
- gomod:
31+
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver
32+
v__OTEL_COLLECTOR_VERSION__
33+
- gomod:
34+
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver
35+
v__OTEL_COLLECTOR_VERSION__
3036
- gomod:
3137
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/fluentforwardreceiver
3238
v__OTEL_COLLECTOR_VERSION__
3339
- gomod:
3440
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver
3541
v__OTEL_COLLECTOR_VERSION__
42+
- gomod:
43+
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver
44+
v__OTEL_COLLECTOR_VERSION__
45+
- gomod:
46+
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver
47+
v__OTEL_COLLECTOR_VERSION__
3648
- gomod:
3749
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver
3850
v__OTEL_COLLECTOR_VERSION__
@@ -46,9 +58,45 @@ processors:
4658
go.opentelemetry.io/collector/processor/memorylimiterprocessor
4759
v__OTEL_COLLECTOR_VERSION__
4860
# Contrib
61+
- gomod:
62+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor
63+
v__OTEL_COLLECTOR_VERSION__
64+
- gomod:
65+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/cumulativetodeltaprocessor
66+
v__OTEL_COLLECTOR_VERSION__
67+
- gomod:
68+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor
69+
v__OTEL_COLLECTOR_VERSION__
70+
- gomod:
71+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/groupbyattrsprocessor
72+
v__OTEL_COLLECTOR_VERSION__
73+
- gomod:
74+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor
75+
v__OTEL_COLLECTOR_VERSION__
76+
- gomod:
77+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/logdedupprocessor
78+
v__OTEL_COLLECTOR_VERSION__
79+
- gomod:
80+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/metricstransformprocessor
81+
v__OTEL_COLLECTOR_VERSION__
82+
- gomod:
83+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor
84+
v__OTEL_COLLECTOR_VERSION__
85+
- gomod:
86+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/redactionprocessor
87+
v__OTEL_COLLECTOR_VERSION__
4988
- gomod:
5089
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor
5190
v__OTEL_COLLECTOR_VERSION__
91+
- gomod:
92+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor
93+
v__OTEL_COLLECTOR_VERSION__
94+
- gomod:
95+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/spanprocessor
96+
v__OTEL_COLLECTOR_VERSION__
97+
- gomod:
98+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor
99+
v__OTEL_COLLECTOR_VERSION__
52100
- gomod:
53101
github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor
54102
v__OTEL_COLLECTOR_VERSION__
@@ -83,10 +131,20 @@ connectors:
83131
v__OTEL_COLLECTOR_VERSION__
84132

85133
extensions:
134+
# Core
135+
- gomod:
136+
go.opentelemetry.io/collector/extension/memorylimiterextension
137+
v__OTEL_COLLECTOR_VERSION__
138+
- gomod:
139+
go.opentelemetry.io/collector/extension/zpagesextension
140+
v__OTEL_COLLECTOR_VERSION__
86141
# Contrib
87142
- gomod:
88143
github.com/open-telemetry/opentelemetry-collector-contrib/extension/bearertokenauthextension
89144
v__OTEL_COLLECTOR_VERSION__
145+
- gomod:
146+
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage
147+
v__OTEL_COLLECTOR_VERSION__
90148
- gomod:
91149
github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension
92150
v__OTEL_COLLECTOR_VERSION__

0 commit comments

Comments
 (0)