Add OTEL K8s events integration tests#702
Open
louisall wants to merge 1 commit into
Open
Conversation
Port K8s events integration tests into the agent test framework for the OTEL standard cluster. Tests validate that K8s events ingested via the k8s_events receiver flow through the cluster-scraper logs pipeline, land in CloudWatch Logs, and carry the expected resource/scope/log attributes set by the pipeline processors (k8sattributes, nodemetadataenricher, transform). Components: - util/otellogs/: CW Logs Insights client with query cache and OTLP JSON parser, mirroring the util/otelmetrics structure - test/otel/standard/events_test.go: 7 integration tests covering log group existence, resource attributes, cluster identity, scope attributes, log attributes, pod enrichment, and event kind diversity - test/otel/standard/events_setup_test.go: log group helper, pipeline constant, and logs client init invoked from TestMain - terraform/eks/daemon/otel/: pre-create the events log group and stream (OTLP x-aws-log-group-create header is not honored by the backend), enable otelContainerInsights.events.enabled in helm, bump test-runner sleep from 3 to 5 minutes for log propagation Validated locally against an existing standard cluster: 7/7 PASS.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the issue
Adds integration test coverage for the K8s events pipeline that was added in:
These were the OTEL events tests originally written in an internal scratchpad repo. Porting them into the agent test framework here (alongside the existing PR #671 standard-cluster metrics tests and the in-flight PR #697 logs framework) so they run as part of the standard cluster CI run.
Description of changes
Ports K8s events integration tests into the OTEL standard cluster test suite. Tests validate that K8s events ingested via the
k8s_eventsreceiver flow through the cluster-scraper logs pipeline, land in CloudWatch Logs, and carry the expected resource/scope/log attributes set by the pipeline processors (k8sattributes, nodemetadataenricher, transform).Components:
util/otellogs/: CW Logs Insights client with query cache and OTLP JSON parser, mirroring the structure ofutil/otelmetrics. Differs from the in-flight PR feat(otel): Add OTLP logs integration test framework #697util/otellogsonly in that the query cache uses@message likesubstring filtering — events JSON has dots in the attribute keys (e.g."k8s.cluster.name"is a literal key) which Insights cannot navigate structurally.test/otel/standard/events_test.go: 7 integration tests — log-group existence, resource attributes, cluster identity, scope attributes, log attributes, pod enrichment (count-based majority assertion), and event-kind diversity.test/otel/standard/events_setup_test.go: log-group helper, pipeline constant, and logs client init invoked fromTestMain.terraform/eks/daemon/otel/main.tf: pre-creates the events log group + stream (the OTLPx-aws-log-group-createheader is not honored by the backend), enablesotelContainerInsights.events.enabledin the helm release, and bumps the test-runner sleep from 3 to 5 minutes for log propagation.License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
Validated end-to-end against a standard EKS cluster with an agent build incorporating the three feature PRs above and the helm-charts #301 events config. Followed the exact Terraform-driven sequence: log group + stream creation → helm install with events enabled → 5-minute propagation wait →
go test -tags integration. Result: 7/7 PASS.