Skip to content

Commit 13cebf5

Browse files
wrn14897wrn14897
committed
feat: add core extensions, commonly-used contrib processors and receivers
Add the two missing core extensions (memorylimiterextension, zpagesextension), 12 commonly-used contrib processors, and 4 commonly-used contrib receivers to builder-config.yaml. New processors: attributesprocessor, cumulativetodeltaprocessor, filterprocessor, groupbyattrsprocessor, k8sattributesprocessor, logdedupprocessor, metricstransformprocessor, probabilisticsamplerprocessor, redactionprocessor, resourceprocessor, spanprocessor, tailsamplingprocessor New receivers: dockerstatsreceiver, filelogreceiver, k8sclusterreceiver, kubeletstatsreceiver Refs: HDX-4029
1 parent 6e9a553 commit 13cebf5

6 files changed

Lines changed: 116 additions & 20 deletions

File tree

‎.changeset/otel-collector-add-core-extensions.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
'@hyperdx/otel-collector': minor
3+
---
4+
5+
feat: Add missing core extensions, commonly-used contrib processors/receivers, and filestorage extension
6+
7+
Add the two missing core extensions (memorylimiterextension, zpagesextension),
8+
12 commonly-used contrib processors (attributes, filter, resource, k8sattributes,
9+
tailsampling, probabilisticsampler, span, groupbyattrs, redaction, logdedup,
10+
metricstransform, cumulativetodelta), 4 commonly-used contrib receivers
11+
(filelog, dockerstats, k8scluster, kubeletstats), and the filestorage extension
12+
(used for persistent sending queue in the clickhouse exporter) to
13+
builder-config.yaml.

‎.github/workflows/main.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ jobs:
4747
- name: Run unit tests
4848
run: make ci-unit
4949
integration:
50-
timeout-minutes: 8
50+
timeout-minutes: 16
5151
runs-on: ubuntu-24.04
5252
steps:
5353
- name: Checkout
@@ -93,7 +93,7 @@ jobs:
9393
working-directory: ./packages/otel-collector
9494
run: go test ./...
9595
otel-smoke-test:
96-
timeout-minutes: 8
96+
timeout-minutes: 16
9797
runs-on: ubuntu-24.04
9898
steps:
9999
- name: Checkout

‎docker/hyperdx/Dockerfile‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,9 @@ ARG OTEL_COLLECTOR_CORE_VERSION
2929
COPY --from=ocb-bin /usr/local/bin/ocb /usr/local/bin/ocb
3030
WORKDIR /build
3131
COPY packages/otel-collector/builder-config.yaml .
32-
RUN sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
32+
RUN --mount=type=cache,target=/go/pkg/mod,id=ocb-gomod \
33+
--mount=type=cache,target=/root/.cache/go-build,id=ocb-gobuild \
34+
sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
3335
CGO_ENABLED=0 ocb --config=builder-config.yaml
3436

3537
# Build the Go migration tool with full TLS support for ClickHouse

‎docker/otel-collector/Dockerfile‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,9 @@ ARG OTEL_COLLECTOR_CORE_VERSION
1919
COPY --from=ocb-bin /usr/local/bin/ocb /usr/local/bin/ocb
2020
WORKDIR /build
2121
COPY packages/otel-collector/builder-config.yaml .
22-
RUN sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
22+
RUN --mount=type=cache,target=/go/pkg/mod,id=ocb-gomod \
23+
--mount=type=cache,target=/root/.cache/go-build,id=ocb-gobuild \
24+
sed -i "s/__OTEL_COLLECTOR_VERSION__/${OTEL_COLLECTOR_VERSION}/g; s/__OTEL_COLLECTOR_CORE_VERSION__/${OTEL_COLLECTOR_CORE_VERSION}/g" builder-config.yaml && \
2325
CGO_ENABLED=0 ocb --config=builder-config.yaml
2426

2527
# Build the Go migration tool with full TLS support for ClickHouse

‎packages/otel-collector/README.md‎

Lines changed: 37 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
Custom-built OpenTelemetry Collector for HyperDX, compiled via
44
[OCB (OpenTelemetry Collector Builder)](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder).
55
This replaces the pre-built `otel/opentelemetry-collector-contrib` image with a
6-
binary that includes only the components HyperDX needs, plus any custom
7-
receivers/processors added in this package.
6+
binary that includes the components HyperDX needs plus commonly-used core and
7+
contrib components, and any custom receivers/processors added in this package.
88

99
## Architecture
1010

@@ -50,26 +50,44 @@ builds.
5050

5151
## Included components
5252

53-
All components referenced in config files and the OpAMP controller:
53+
Components used by HyperDX internally are marked with their config references.
54+
Components marked "user configs" are included so users can reference them in
55+
custom OTel configurations without rebuilding the collector.
5456

5557
### Receivers
5658

57-
| Component | Module | Used in |
58-
| --------------- | ------- | ------------------------------------------------- |
59-
| `nop` | core | OpAMP controller |
60-
| `otlp` | core | standalone configs, OpAMP controller, smoke tests |
61-
| `fluentforward` | contrib | standalone configs, OpAMP controller, smoke tests |
62-
| `hostmetrics` | contrib | custom.config.yaml |
63-
| `prometheus` | contrib | OpAMP controller, smoke tests |
59+
| Component | Module | Used in |
60+
| ---------------- | ------- | ------------------------------------------------- |
61+
| `nop` | core | OpAMP controller |
62+
| `otlp` | core | standalone configs, OpAMP controller, smoke tests |
63+
| `dockerstats` | contrib | user configs |
64+
| `filelog` | contrib | user configs |
65+
| `fluentforward` | contrib | standalone configs, OpAMP controller, smoke tests |
66+
| `hostmetrics` | contrib | custom.config.yaml |
67+
| `k8scluster` | contrib | user configs |
68+
| `kubeletstats` | contrib | user configs |
69+
| `prometheus` | contrib | OpAMP controller, smoke tests |
6470

6571
### Processors
6672

67-
| Component | Module | Used in |
68-
| ------------------- | ------- | ------------------------------------------------- |
69-
| `batch` | core | config.yaml, standalone configs, OpAMP controller |
70-
| `memory_limiter` | core | config.yaml, standalone configs, OpAMP controller |
71-
| `resourcedetection` | contrib | config.yaml |
72-
| `transform` | contrib | config.yaml, standalone configs, OpAMP controller |
73+
| Component | Module | Used in |
74+
| ---------------------- | ------- | ------------------------------------------------- |
75+
| `batch` | core | config.yaml, standalone configs, OpAMP controller |
76+
| `memory_limiter` | core | config.yaml, standalone configs, OpAMP controller |
77+
| `attributes` | contrib | user configs |
78+
| `cumulativetodelta` | contrib | user configs |
79+
| `filter` | contrib | user configs |
80+
| `groupbyattrs` | contrib | user configs |
81+
| `k8sattributes` | contrib | user configs |
82+
| `logdedup` | contrib | user configs |
83+
| `metricstransform` | contrib | user configs |
84+
| `probabilisticsampler` | contrib | user configs |
85+
| `redaction` | contrib | user configs |
86+
| `resourcedetection` | contrib | config.yaml |
87+
| `resource` | contrib | user configs |
88+
| `span` | contrib | user configs |
89+
| `tailsampling` | contrib | user configs |
90+
| `transform` | contrib | config.yaml, standalone configs, OpAMP controller |
7391

7492
### Exporters
7593

@@ -92,7 +110,10 @@ All components referenced in config files and the OpAMP controller:
92110

93111
| Component | Module | Used in |
94112
| ----------------- | ------- | ---------------------------------------- |
113+
| `memorylimiter` | core | user configs |
114+
| `zpages` | core | user configs |
95115
| `bearertokenauth` | contrib | standalone-auth config, OpAMP controller |
116+
| `file_storage` | contrib | OpAMP controller (sending queue storage) |
96117
| `health_check` | contrib | config.yaml, standalone-auth config |
97118
| `opamp` | contrib | used by OpAMP supervisor |
98119
| `pprof` | contrib | included for debugging/profiling |

‎packages/otel-collector/builder-config.yaml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,24 @@ receivers:
2727
go.opentelemetry.io/collector/receiver/otlpreceiver
2828
v__OTEL_COLLECTOR_VERSION__
2929
# Contrib
30+
- gomod:
31+
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver
32+
v__OTEL_COLLECTOR_VERSION__
33+
- gomod:
34+
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver
35+
v__OTEL_COLLECTOR_VERSION__
3036
- gomod:
3137
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/fluentforwardreceiver
3238
v__OTEL_COLLECTOR_VERSION__
3339
- gomod:
3440
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver
3541
v__OTEL_COLLECTOR_VERSION__
42+
- gomod:
43+
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver
44+
v__OTEL_COLLECTOR_VERSION__
45+
- gomod:
46+
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver
47+
v__OTEL_COLLECTOR_VERSION__
3648
- gomod:
3749
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver
3850
v__OTEL_COLLECTOR_VERSION__
@@ -46,9 +58,45 @@ processors:
4658
go.opentelemetry.io/collector/processor/memorylimiterprocessor
4759
v__OTEL_COLLECTOR_VERSION__
4860
# Contrib
61+
- gomod:
62+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor
63+
v__OTEL_COLLECTOR_VERSION__
64+
- gomod:
65+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/cumulativetodeltaprocessor
66+
v__OTEL_COLLECTOR_VERSION__
67+
- gomod:
68+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor
69+
v__OTEL_COLLECTOR_VERSION__
70+
- gomod:
71+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/groupbyattrsprocessor
72+
v__OTEL_COLLECTOR_VERSION__
73+
- gomod:
74+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor
75+
v__OTEL_COLLECTOR_VERSION__
76+
- gomod:
77+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/logdedupprocessor
78+
v__OTEL_COLLECTOR_VERSION__
79+
- gomod:
80+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/metricstransformprocessor
81+
v__OTEL_COLLECTOR_VERSION__
82+
- gomod:
83+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor
84+
v__OTEL_COLLECTOR_VERSION__
85+
- gomod:
86+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/redactionprocessor
87+
v__OTEL_COLLECTOR_VERSION__
4988
- gomod:
5089
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor
5190
v__OTEL_COLLECTOR_VERSION__
91+
- gomod:
92+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor
93+
v__OTEL_COLLECTOR_VERSION__
94+
- gomod:
95+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/spanprocessor
96+
v__OTEL_COLLECTOR_VERSION__
97+
- gomod:
98+
github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor
99+
v__OTEL_COLLECTOR_VERSION__
52100
- gomod:
53101
github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor
54102
v__OTEL_COLLECTOR_VERSION__
@@ -83,10 +131,20 @@ connectors:
83131
v__OTEL_COLLECTOR_VERSION__
84132

85133
extensions:
134+
# Core
135+
- gomod:
136+
go.opentelemetry.io/collector/extension/memorylimiterextension
137+
v__OTEL_COLLECTOR_VERSION__
138+
- gomod:
139+
go.opentelemetry.io/collector/extension/zpagesextension
140+
v__OTEL_COLLECTOR_VERSION__
86141
# Contrib
87142
- gomod:
88143
github.com/open-telemetry/opentelemetry-collector-contrib/extension/bearertokenauthextension
89144
v__OTEL_COLLECTOR_VERSION__
145+
- gomod:
146+
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage
147+
v__OTEL_COLLECTOR_VERSION__
90148
- gomod:
91149
github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension
92150
v__OTEL_COLLECTOR_VERSION__

0 commit comments

Comments
 (0)