Skip to content

Commit cb84145

Browse files
wrn14897wrn14897
authored
[HDX-3994] Deprecate clickhouse.json feature gate in favor of per-exporter json config (#2119)
## Summary Deprecate the upstream-deprecated `--feature-gates=clickhouse.json` CLI flag in favor of the per-exporter `json: true` config option, as recommended by the OpenTelemetry ClickHouse exporter v0.149.0. This introduces a new env var `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE` that controls JSON mode at the exporter config level. The old `OTEL_AGENT_FEATURE_GATE_ARG` env var remains backward-compatible — when it contains `clickhouse.json`, the entrypoint strips that gate, maps it to the new env var, and prints a deprecation warning. Other feature gates are preserved and passed through to the collector. **Key changes:** - **`docker/otel-collector/entrypoint.sh`** — Detects `clickhouse.json` in `OTEL_AGENT_FEATURE_GATE_ARG`, strips it, sets `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE=true`, and prints a deprecation warning. Remaining feature gates are still passed through to the collector in both standalone and supervisor modes. - **`docker/otel-collector/config.standalone.yaml`** — Added `json: ${env:HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE:-false}` to both ClickHouse exporter configs - **`packages/api/src/opamp/controllers/opampController.ts`** — Added `json` field to the `CollectorConfig` type and both ClickHouse exporter configs for OpAMP-managed collectors - **`docker/otel-collector/supervisor_docker.yaml.tmpl`** — Feature gate pass-through preserved for non-`clickhouse.json` gates (entrypoint strips the deprecated gate before supervisor template renders) - **`smoke-tests/otel-collector/`** — Added a JSON-enabled otel-collector service and smoke tests verifying: - `ResourceAttributes` and `LogAttributes` columns in `otel_logs` are `JSON` type (not `Map`) - Log data with various attribute types (string, int, boolean) is inserted and queryable via JSON path access ### How to test locally or on Vercel 1. Run `yarn dev` to start the dev stack 2. Verify the `otel-collector-json` container starts without errors (the `clickhouse.json` feature gate is stripped, not passed to the collector) 3. Check container logs for the deprecation warning when `OTEL_AGENT_FEATURE_GATE_ARG` contains `clickhouse.json` 4. Verify the non-JSON `otel-collector` service continues to work normally (json defaults to false) 5. Run smoke tests: `cd smoke-tests/otel-collector && bats json-exporter.bats` ### References - Linear Issue: https://linear.app/hyperdx/issue/HDX-3994 - Upstream deprecation: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter#experimental-json-support
1 parent 4ca1d47 commit cb84145

14 files changed

Lines changed: 187 additions & 6 deletions

File tree

‎.changeset/deprecate-clickhouse-json-feature-gate.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
'@hyperdx/otel-collector': patch
3+
---
4+
5+
refactor: Deprecate clickhouse.json feature gate in favor of per-exporter json config
6+
7+
Replace the upstream-deprecated `--feature-gates=clickhouse.json` CLI flag with
8+
the per-exporter `json: true` config option controlled by
9+
`HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE`. The old
10+
`OTEL_AGENT_FEATURE_GATE_ARG` is still supported for backward compatibility but
11+
prints a deprecation warning when `clickhouse.json` is detected.

‎docker-compose.dev.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -80,9 +80,9 @@ services:
8080
CUSTOM_OTELCOL_CONFIG_FILE: '/etc/otelcol-contrib/custom.config.yaml'
8181
# Uncomment to enable stdout logging for the OTel collector
8282
OTEL_SUPERVISOR_LOGS: 'true'
83-
# Uncomment to enable JSON schema in ClickHouse
83+
# Enable JSON schema in the ClickHouse exporter (per-exporter config)
8484
# Be sure to also set BETA_CH_OTEL_JSON_SCHEMA_ENABLED to 'true' in ch-server
85-
OTEL_AGENT_FEATURE_GATE_ARG: '--feature-gates=clickhouse.json'
85+
HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE: 'true'
8686
volumes:
8787
- ./docker/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml
8888
- ./docker/otel-collector/supervisor_docker.yaml.tmpl:/etc/otel/supervisor.yaml.tmpl
@@ -109,7 +109,7 @@ services:
109109
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
110110
HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE: ${HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE}
111111
# Set to 'true' to allow for proper OTel JSON Schema creation
112-
# Be sure to also set the OTEL_AGENT_FEATURE_GATE_ARG env in otel-collector
112+
# Be sure to also set HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE in otel-collector
113113
# BETA_CH_OTEL_JSON_SCHEMA_ENABLED: 'true'
114114
volumes:
115115
- ./docker/clickhouse/local/config.xml:/etc/clickhouse-server/config.xml

‎docker/otel-collector/config.standalone.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ exporters:
3333
logs_table_name: hyperdx_sessions
3434
timeout: 5s
3535
create_schema: ${env:HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA:-false}
36+
json: ${env:HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE:-false}
3637
retry_on_failure:
3738
enabled: true
3839
initial_interval: 5s
@@ -46,6 +47,7 @@ exporters:
4647
ttl: 720h
4748
timeout: 5s
4849
create_schema: ${env:HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA:-false}
50+
json: ${env:HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE:-false}
4951
retry_on_failure:
5052
enabled: true
5153
initial_interval: 5s

‎docker/otel-collector/entrypoint.sh‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,26 @@
11
#!/bin/sh
22
set -e
33

4-
# Fall back to legacy schema when the ClickHouse JSON feature gate is enabled
4+
# DEPRECATED: The clickhouse.json feature gate has been removed upstream.
5+
# When OTEL_AGENT_FEATURE_GATE_ARG contains clickhouse.json, strip it and
6+
# map it to HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE instead. Other feature gates
7+
# are preserved and passed through to the collector.
58
if echo "$OTEL_AGENT_FEATURE_GATE_ARG" | grep -q "clickhouse.json"; then
9+
echo "WARNING: '--feature-gates=clickhouse.json' is deprecated and no longer supported by the collector."
10+
echo "WARNING: Use HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE=true instead. This flag will be removed in a future release."
11+
export HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE=true
12+
13+
# Strip clickhouse.json from the feature gates, keeping any other gates
14+
REMAINING_GATES=$(echo "$OTEL_AGENT_FEATURE_GATE_ARG" | sed 's/--feature-gates=//' | tr ',' '\n' | grep -v 'clickhouse.json' | tr '\n' ',' | sed 's/,$//')
15+
if [ -n "$REMAINING_GATES" ]; then
16+
export OTEL_AGENT_FEATURE_GATE_ARG="--feature-gates=$REMAINING_GATES"
17+
else
18+
unset OTEL_AGENT_FEATURE_GATE_ARG
19+
fi
20+
fi
21+
22+
# Fall back to legacy schema when ClickHouse JSON exporter mode is enabled
23+
if [ "$HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE" = "true" ]; then
624
export HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA=true
725
fi
826

@@ -39,7 +57,7 @@ if [ -z "$OPAMP_SERVER_URL" ]; then
3957
COLLECTOR_ARGS="$COLLECTOR_ARGS --config $CUSTOM_OTELCOL_CONFIG_FILE"
4058
fi
4159

42-
# Pass feature gates to the collector in standalone mode
60+
# Pass remaining feature gates to the collector in standalone mode
4361
if [ -n "$OTEL_AGENT_FEATURE_GATE_ARG" ]; then
4462
COLLECTOR_ARGS="$COLLECTOR_ARGS $OTEL_AGENT_FEATURE_GATE_ARG"
4563
fi

‎docker/otel-collector/supervisor_docker.yaml.tmpl‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,8 @@ agent:
2323
{{- if getenv "CUSTOM_OTELCOL_CONFIG_FILE" }}
2424
- {{ getenv "CUSTOM_OTELCOL_CONFIG_FILE" }}
2525
{{- end }}
26-
args:
2726
{{- if getenv "OTEL_AGENT_FEATURE_GATE_ARG" }}
27+
args:
2828
- {{ getenv "OTEL_AGENT_FEATURE_GATE_ARG" }}
2929
{{- end }}
3030

‎packages/api/src/opamp/controllers/opampController.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ type CollectorConfig = {
8282
logs_table_name: string;
8383
timeout: string;
8484
create_schema: string;
85+
json: string;
8586
retry_on_failure: {
8687
enabled: boolean;
8788
initial_interval: string;
@@ -97,6 +98,7 @@ type CollectorConfig = {
9798
ttl: string;
9899
timeout: string;
99100
create_schema: string;
101+
json: string;
100102
retry_on_failure: {
101103
enabled: boolean;
102104
initial_interval: string;
@@ -205,6 +207,7 @@ export const buildOtelCollectorConfig = (
205207
timeout: '5s',
206208
create_schema:
207209
'${env:HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA:-false}',
210+
json: '${env:HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE:-false}',
208211
retry_on_failure: {
209212
enabled: true,
210213
initial_interval: '5s',
@@ -221,6 +224,7 @@ export const buildOtelCollectorConfig = (
221224
timeout: '5s',
222225
create_schema:
223226
'${env:HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA:-false}',
227+
json: '${env:HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE:-false}',
224228
retry_on_failure: {
225229
enabled: true,
226230
initial_interval: '5s',

‎smoke-tests/otel-collector/data/json-exporter/basic-insert/assert_query.sql‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
SELECT
2+
Body,
3+
toString(LogAttributes.`user.id`),
4+
toString(LogAttributes.`request.method`),
5+
toString(LogAttributes.`http.status_code`),
6+
toString(LogAttributes.error)
7+
FROM otel_json.otel_logs
8+
WHERE toString(ResourceAttributes.`suite-id`) = 'json-exporter'
9+
AND toString(ResourceAttributes.`test-id`) = 'basic-insert'
10+
ORDER BY Timestamp
11+
FORMAT CSV

‎smoke-tests/otel-collector/data/json-exporter/basic-insert/expected.snap‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
"JSON exporter log with string attributes","user-123","GET","",""
2+
"JSON exporter log with integer attribute","","","200",""
3+
"JSON exporter log with boolean attribute","","","","false"

‎smoke-tests/otel-collector/data/json-exporter/basic-insert/input.json‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
{
2+
"resourceLogs": [
3+
{
4+
"resource": {
5+
"attributes": [
6+
{
7+
"key": "suite-id",
8+
"value": {
9+
"stringValue": "json-exporter"
10+
}
11+
},
12+
{
13+
"key": "test-id",
14+
"value": {
15+
"stringValue": "basic-insert"
16+
}
17+
},
18+
{
19+
"key": "service.name",
20+
"value": {
21+
"stringValue": "test-service"
22+
}
23+
}
24+
]
25+
},
26+
"scopeLogs": [
27+
{
28+
"scope": {},
29+
"logRecords": [
30+
{
31+
"timeUnixNano": "1901999590000000000",
32+
"body": {
33+
"stringValue": "JSON exporter log with string attributes"
34+
},
35+
"attributes": [
36+
{
37+
"key": "user.id",
38+
"value": {
39+
"stringValue": "user-123"
40+
}
41+
},
42+
{
43+
"key": "request.method",
44+
"value": {
45+
"stringValue": "GET"
46+
}
47+
}
48+
]
49+
},
50+
{
51+
"timeUnixNano": "1901999590000000001",
52+
"body": {
53+
"stringValue": "JSON exporter log with integer attribute"
54+
},
55+
"attributes": [
56+
{
57+
"key": "http.status_code",
58+
"value": {
59+
"intValue": "200"
60+
}
61+
}
62+
]
63+
},
64+
{
65+
"timeUnixNano": "1901999590000000002",
66+
"body": {
67+
"stringValue": "JSON exporter log with boolean attribute"
68+
},
69+
"attributes": [
70+
{
71+
"key": "error",
72+
"value": {
73+
"boolValue": false
74+
}
75+
}
76+
]
77+
}
78+
]
79+
}
80+
]
81+
}
82+
]
83+
}

‎smoke-tests/otel-collector/data/json-exporter/column-types/assert_query.sql‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
SELECT name, type
2+
FROM system.columns
3+
WHERE database = 'otel_json'
4+
AND table = 'otel_logs'
5+
AND name IN ('ResourceAttributes', 'LogAttributes')
6+
ORDER BY name
7+
FORMAT CSV

0 commit comments

Comments
 (0)