device/telemetry: samples silently dropped when submission fails at buffer capacity

[elitegreg]

Tracked by #4130.

Problem

Submitter.Tick discards samples with no log and no metric when submission failed and the
partition is at capacity:

// controlplane/telemetry/internal/telemetry/submitter.go:246
// If submission failed and the buffer is not at capacity, prepend the samples back to the
// buffer. If the buffer is at capacity and we have failed all attempts, don't prepend the
// samples back to the buffer.
overCapacity := s.cfg.Buffer.Len(partitionKey)+len(tmp) >= s.cfg.Buffer.Capacity(partitionKey)
if !success && !overCapacity {
	s.cfg.Buffer.PriorityPrepend(partitionKey, tmp)
}

// Always recycle the slice for reuse
s.cfg.Buffer.Recycle(partitionKey, tmp)

When !success && overCapacity, tmp is recycled and the samples are gone. The
SubmitterRetriesExhausted counter fires for the failed submission, so you learn a submission
failed, but nothing anywhere records that the samples were discarded rather than requeued.

This is the only path in the submitter that permanently destroys measurement data with zero
signal.

Partition capacity is 4096 (collector.go:17, partitionBufferCapacity), which at 6 samples per
minute is roughly 11 hours of backlog before this triggers.

Proposed change

Log and count the drop.

if !success {
	if overCapacity {
		metrics.Errors.WithLabelValues(metrics.ErrorTypeSubmitterBufferFull).Inc()
		metrics.SamplesDropped.WithLabelValues(metrics.DropReasonBufferFull).Add(float64(len(tmp)))
		log.Warn("partition buffer at capacity after failed submission, dropping samples",
			"droppedSamples", len(tmp),
			"bufferLen", s.cfg.Buffer.Len(partitionKey),
			"capacity", s.cfg.Buffer.Capacity(partitionKey))
	} else {
		s.cfg.Buffer.PriorityPrepend(partitionKey, tmp)
	}
}

New identifiers needed in internal/metrics/metrics.go:

• ErrorTypeSubmitterBufferFull = "submitter_buffer_full"
• a doublezero_device_telemetry_agent_samples_dropped_total counter with a reason label

The reason label lets the account-full path (#4127) use the same counter.

Noise

One line per partition per submission interval, and only while a partition has been backed up for
~11 hours. In steady state this never fires.

Acceptance

• A test that fails submission and fills the partition to capacity asserts the warning is emitted
  and the counter increments by len(tmp).
• Existing requeue behavior below capacity is unchanged.