Skip to content

pubsub: panic when using tracing due to nil Span interface #14277

@magnusewe

Description

@magnusewe

Client

PubSub v2.4.0

Environment

Go v1.25.8

Code and Dependencies

// Creating a client with otel tracing enabled:
client, err := pubsub.NewClientWithConfig(
	ctx,
	projectID,
	&pubsub.ClientConfig{
		EnableOpenTelemetryTracing: true,
	},
)

// Pulling messages...

Expected behavior

No panics in the pubsub library.

Actual behavior

Nil interface panic when the v2 subscriber calls ccSpan.End().
This started happening after the client had been running for a while.

panic: runtime error: invalid memory address or nil pointer dereference                                                                               
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xdfd902]                                                                                                                                                                                       
                                                                                                                                                                                                                                                              
goroutine 408 [running]:                                                                                                                                                                                                                                      
cloud.google.com/go/pubsub/v2.(*Subscriber).Receive.func2()                                                                                                                                                                                                   
    cloud.google.com/go/pubsub/v2@v2.4.0/subscriber.go:397 +0x682                                                                                                                                                                                             
golang.org/x/sync/errgroup.(*Group).Go.func1()                                                                                                                                                                                                                
    golang.org/x/sync@v0.19.0/errgroup/errgroup.go:93 +0x50                                                                                                                                                                                                   
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 312                                                                                                                                                                                            
    golang.org/x/sync@v0.19.0/errgroup/errgroup.go:78 +0x95

This happens because ccSpan is never assigned when iter.activeSpans.Load(ackh.ackID) returns false.

var ccSpan trace.Span
if iter.enableTracing {
c, ok := iter.activeSpans.Load(ackh.ackID)
if ok {
sc := c.(trace.Span)
otelCtx = trace.ContextWithSpanContext(otelCtx, sc.SpanContext())
// Don't override otelCtx here since the parent of subsequent spans
// should be the subscribe span still.
_, ccSpan = startSpan(otelCtx, ccSpanName, "")
}
}
// Use the original user defined ctx for this operation so the acquire operation can be cancelled.
if err := fc.acquire(ctx, len(msg.Data)); err != nil {
// TODO(jba): test that these "orphaned" messages are nacked immediately when ctx is done.
for _, m := range msgs[i:] {
m.Nack()
}
// Return nil if the context is done, not err.
return nil
}
if iter.enableTracing {
ccSpan.End()
}

Metadata

Metadata

Assignees

Labels

api: pubsubIssues related to the Pub/Sub API.priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions