Skip to content

Add OTEL K8s events integration tests#702

Open
louisall wants to merge 1 commit into
mainfrom
feat/otel-events-integration-tests
Open

Add OTEL K8s events integration tests#702
louisall wants to merge 1 commit into
mainfrom
feat/otel-events-integration-tests

Conversation

@louisall
Copy link
Copy Markdown
Contributor

Description of the issue

Adds integration test coverage for the K8s events pipeline that was added in:

These were the OTEL events tests originally written in an internal scratchpad repo. Porting them into the agent test framework here (alongside the existing PR #671 standard-cluster metrics tests and the in-flight PR #697 logs framework) so they run as part of the standard cluster CI run.

Description of changes

Ports K8s events integration tests into the OTEL standard cluster test suite. Tests validate that K8s events ingested via the k8s_events receiver flow through the cluster-scraper logs pipeline, land in CloudWatch Logs, and carry the expected resource/scope/log attributes set by the pipeline processors (k8sattributes, nodemetadataenricher, transform).

Components:

  • util/otellogs/: CW Logs Insights client with query cache and OTLP JSON parser, mirroring the structure of util/otelmetrics. Differs from the in-flight PR feat(otel): Add OTLP logs integration test framework #697 util/otellogs only in that the query cache uses @message like substring filtering — events JSON has dots in the attribute keys (e.g. "k8s.cluster.name" is a literal key) which Insights cannot navigate structurally.
  • test/otel/standard/events_test.go: 7 integration tests — log-group existence, resource attributes, cluster identity, scope attributes, log attributes, pod enrichment (count-based majority assertion), and event-kind diversity.
  • test/otel/standard/events_setup_test.go: log-group helper, pipeline constant, and logs client init invoked from TestMain.
  • terraform/eks/daemon/otel/main.tf: pre-creates the events log group + stream (the OTLP x-aws-log-group-create header is not honored by the backend), enables otelContainerInsights.events.enabled in the helm release, and bumps the test-runner sleep from 3 to 5 minutes for log propagation.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Validated end-to-end against a standard EKS cluster with an agent build incorporating the three feature PRs above and the helm-charts #301 events config. Followed the exact Terraform-driven sequence: log group + stream creation → helm install with events enabled → 5-minute propagation wait → go test -tags integration. Result: 7/7 PASS.

=== RUN   TestEventsLogGroupExists                              PASS
=== RUN   TestEventsResourceAttributes                          PASS (10/10 sub)
=== RUN   TestEventsClusterIdentity                             PASS
=== RUN   TestEventsScopeAttributes                             PASS (3/3 sub)
=== RUN   TestEventsLogAttributes                               PASS (6/6 sub)
=== RUN   TestEventsPodEnrichment                               PASS
    receiver_attrs                                              PASS
    enrichment_attrs (k8s.node.name, host.id, host.name,
                      host.type, host.image.id, cloud.az)       PASS (6/6 sub)
    workload_derivation                                         PASS
=== RUN   TestEventsKindDiversity                               PASS
PASS

Port K8s events integration tests into the agent test framework for the
OTEL standard cluster. Tests validate that K8s events ingested via the
k8s_events receiver flow through the cluster-scraper logs pipeline,
land in CloudWatch Logs, and carry the expected resource/scope/log
attributes set by the pipeline processors (k8sattributes,
nodemetadataenricher, transform).

Components:
- util/otellogs/: CW Logs Insights client with query cache and OTLP
  JSON parser, mirroring the util/otelmetrics structure
- test/otel/standard/events_test.go: 7 integration tests covering log
  group existence, resource attributes, cluster identity, scope
  attributes, log attributes, pod enrichment, and event kind diversity
- test/otel/standard/events_setup_test.go: log group helper, pipeline
  constant, and logs client init invoked from TestMain
- terraform/eks/daemon/otel/: pre-create the events log group and
  stream (OTLP x-aws-log-group-create header is not honored by the
  backend), enable otelContainerInsights.events.enabled in helm,
  bump test-runner sleep from 3 to 5 minutes for log propagation

Validated locally against an existing standard cluster: 7/7 PASS.
@louisall louisall requested a review from a team as a code owner May 19, 2026 13:43
@louisall louisall requested a review from miconeilaws May 20, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant