feat: replace all Prometheus metrics with OTel instruments (7+2) (#98)

* feat: replace all Prometheus metrics with OTel instruments (7+2)

Phase 2 of the OTel migration. Atomic swap of all 12 prometheus/client_golang
metrics with 7 core + 2 deployment OTel instruments mapped to the Four
Golden Signals.

LATENCY:
- sei.controller.seinode.phase.duration — time in each phase (histogram)
- sei.controller.plan.duration — plan wall-clock time (histogram)

TRAFFIC:
- sei.controller.seinode.phase — observable gauge with PhaseTracker callback
- sei.controller.seinodedeployment.phase — same pattern

ERRORS:
- sei.controller.plan.failures — terminal plan failure counter
- sei.controller.reconcile.errors — namespace-scoped error counter

SATURATION:
- sei.controller.plan.active — UpDownCounter for concurrent plans

DEPLOYMENT:
- sei.controller.seinodedeployment.replicas — replica_state dimension

CRD change: PhaseTransitionTime added to SeiNodeStatus for phase duration
computation. Stamped in setTargetPhase (executor) and ResolvePlan
(Pending→Initializing transition).

Cut: phase_transitions_total (combinatorial), last_init_duration (per-node),
reconcile_substep_duration (internal), reconcile_errors per-name (unbounded),
condition gauge (CRD queryable), timeSubstep wrapper.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: address OTel expert review — per-package meters, no redundant counter

Three fixes from the OpenTelemetry expert review:

1. Per-package meters: each package now creates its own meter via
   observability.NewMeter("pkg") for proper scope identification in
   backends. Replaces the shared global Meter.

2. Remove redundant planFailures counter: planDuration histogram with
   an outcome attribute (complete/failed) provides both duration AND
   count via _count. The histogram's outcome dimension replaces the
   separate failure counter.

3. Document context.Background() usage in handleTerminalPlan as a
   TODO for when tracing is added. Acceptable for metrics-only
   recording in a controller reconcile loop.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: thread context through metric emission sites

Thread the reconcile ctx to all metric emission call sites instead of
using context.Background(). This ensures proper context propagation
when tracing is added.

- ResolvePlan now accepts ctx, passes it to handleTerminalPlan
- emitGroupReplicas now accepts ctx from the reconciler
- No remaining context.Background() in production metric code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: use t.Context() in planner tests instead of context.Background()

Per OTel expert guide: use t.Context() in tests for automatic
cancellation when the test ends.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: address PR comments — gauge for plan.active, cleanup, constant

1. planActiveCount: switched from UpDownCounter to Int64Gauge backed
   by an atomic counter. Records the absolute count on each change.
2. Remove dead CleanupPlanMetrics no-op function.
3. Use seiNodeControllerName constant instead of "seinode" string.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* revert: use UpDownCounter for plan.active — designed for increment/decrement

Int64UpDownCounter is the correct OTel instrument for "number of active X"
patterns. It has Add(+1)/Add(-1) built in, maps to a Prometheus gauge,
and doesn't need a separate atomic counter. Reverts the gauge change.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: bdchatham

api/v1alpha1/seinode_types.go

@@ -229,6 +229,11 @@ type SeiNodeStatus struct {
 	// +optional
 	Conditions []metav1.Condition `json:"conditions,omitempty"`
 
+	// PhaseTransitionTime is when the node last changed phases.
+	// Used to compute phase duration metrics.
+	// +optional
+	PhaseTransitionTime *metav1.Time `json:"phaseTransitionTime,omitempty"`
+
 	// Plan tracks the active task sequence for this node. A planner generates
 	// the plan based on the node's current state and conditions.
 	// +optional

api/v1alpha1/zz_generated.deepcopy.go

@@ -582,6 +582,12 @@ spec:
                 - Failed
                 - Terminating
                 type: string
+              phaseTransitionTime:
+                description: |-
+                  PhaseTransitionTime is when the node last changed phases.
+                  Used to compute phase duration metrics.
+                format: date-time
+                type: string
               plan:
                 description: |-
                   Plan tracks the active task sequence for this node. A planner generates

config/crd/sei.io_seinodes.yaml

@@ -8,12 +8,12 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/s3 v1.97.2
 	github.com/google/uuid v1.6.0
 	github.com/onsi/gomega v1.38.2
-	github.com/prometheus/client_golang v1.23.2
 	github.com/sei-protocol/sei-config v0.0.11
 	github.com/sei-protocol/seictl v0.0.30
 	go.opentelemetry.io/otel v1.43.0
 	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.43.0
 	go.opentelemetry.io/otel/exporters/prometheus v0.65.0
+	go.opentelemetry.io/otel/metric v1.43.0
 	go.opentelemetry.io/otel/sdk v1.43.0
 	go.opentelemetry.io/otel/sdk/metric v1.43.0
 	k8s.io/api v0.35.0
@@ -78,6 +78,7 @@ require (
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/oapi-codegen/runtime v1.2.0 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
+	github.com/prometheus/client_golang v1.23.2 // indirect
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.67.5 // indirect
 	github.com/prometheus/otlptranslator v1.0.0 // indirect
@@ -92,7 +93,6 @@ require (
 	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0 // indirect
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.37.0 // indirect
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.37.0 // indirect
-	go.opentelemetry.io/otel/metric v1.43.0 // indirect
 	go.opentelemetry.io/otel/trace v1.43.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.10.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect

go.mod

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"time"
 
+	"go.opentelemetry.io/otel/metric"
 	appsv1 "k8s.io/api/apps/v1"
 	batchv1 "k8s.io/api/batch/v1"
 	corev1 "k8s.io/api/core/v1"
@@ -20,6 +21,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/predicate"
 
 	seiv1alpha1 "github.com/sei-protocol/sei-k8s-controller/api/v1alpha1"
+	"github.com/sei-protocol/sei-k8s-controller/internal/controller/observability"
 	"github.com/sei-protocol/sei-k8s-controller/internal/noderesource"
 	"github.com/sei-protocol/sei-k8s-controller/internal/planner"
 	"github.com/sei-protocol/sei-k8s-controller/internal/platform"
@@ -101,7 +103,7 @@ func (r *SeiNodeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct
 	// builds a new one based on the node's phase, stamping it onto
 	// node.Status.Plan (and transitioning Pending → Initializing).
 	planAlreadyActive := node.Status.Plan != nil && node.Status.Plan.Phase == seiv1alpha1.TaskPlanActive
-	if err := planner.ResolvePlan(node); err != nil {
+	if err := planner.ResolvePlan(ctx, node); err != nil {
 		return ctrl.Result{}, fmt.Errorf("resolving plan: %w", err)
 	}
 
@@ -136,15 +138,28 @@ func (r *SeiNodeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct
 	// Emit metrics/events if the phase changed.
 	if node.Status.Phase != observedPhase {
 		ns, name := node.Namespace, node.Name
-		nodePhaseTransitions.WithLabelValues(ns, string(observedPhase), string(node.Status.Phase)).Inc()
+		nodePhaseTransitions.Add(ctx, 1,
+			metric.WithAttributes(
+				observability.AttrController.String(seiNodeControllerName),
+				observability.AttrNamespace.String(ns),
+				observability.AttrFromPhase.String(string(observedPhase)),
+				observability.AttrToPhase.String(string(node.Status.Phase)),
+			),
+		)
 		emitNodePhase(ns, name, node.Status.Phase)
 		r.Recorder.Eventf(node, corev1.EventTypeNormal, "PhaseTransition",
 			"Phase changed from %s to %s", observedPhase, node.Status.Phase)
 
-		if node.Status.Phase == seiv1alpha1.PhaseRunning {
-			dur := time.Since(node.CreationTimestamp.Time).Seconds()
-			nodeInitDuration.WithLabelValues(ns, node.Spec.ChainID).Observe(dur)
-			nodeLastInitDuration.WithLabelValues(ns, name).Set(dur)
+		// Record time spent in the previous phase.
+		if node.Status.PhaseTransitionTime != nil && observedPhase != "" {
+			dur := time.Since(node.Status.PhaseTransitionTime.Time).Seconds()
+			nodePhaseDuration.Record(ctx, dur,
+				metric.WithAttributes(
+					observability.AttrNamespace.String(ns),
+					observability.AttrChainID.String(node.Spec.ChainID),
+					observability.AttrPhase.String(string(observedPhase)),
+				),
+			)
 		}
 	}
 

internal/controller/node/controller.go

@@ -1,12 +1,10 @@
 package node
 
 import (
-	"github.com/prometheus/client_golang/prometheus"
-	"sigs.k8s.io/controller-runtime/pkg/metrics"
+	"go.opentelemetry.io/otel/metric"
 
 	seiv1alpha1 "github.com/sei-protocol/sei-k8s-controller/api/v1alpha1"
 	"github.com/sei-protocol/sei-k8s-controller/internal/controller/observability"
-	"github.com/sei-protocol/sei-k8s-controller/internal/planner"
 )
 
 var allNodePhases = []string{
@@ -17,57 +15,54 @@ var allNodePhases = []string{
 	string(seiv1alpha1.PhaseTerminating),
 }
 
+var nodePhaseTracker = observability.NewPhaseTracker(allNodePhases)
+
 var (
-	nodePhaseGauge = prometheus.NewGaugeVec(
-		prometheus.GaugeOpts{
-			Name: "sei_controller_seinode_phase",
-			Help: "Current phase of each SeiNode (1=active, 0=inactive)",
-		},
-		[]string{"namespace", "name", "phase"},
-	)
+	// nodePhaseTransitions counts phase transitions.
+	nodePhaseTransitions metric.Int64Counter
 
-	nodePhaseTransitions = prometheus.NewCounterVec(
-		prometheus.CounterOpts{
-			Name: "sei_controller_seinode_phase_transitions_total",
-			Help: "Phase state machine transitions",
-		},
-		[]string{"namespace", "from", "to"},
-	)
+	// nodePhaseDuration records time spent in each phase when transitioning out.
+	nodePhaseDuration metric.Float64Histogram
+)
+
+var meter = observability.NewMeter("node")
 
-	nodeInitDuration = prometheus.NewHistogramVec(
-		prometheus.HistogramOpts{
-			Name:    "sei_controller_seinode_init_duration_seconds",
-			Help:    "Time from Pending to Running",
-			Buckets: observability.InitBuckets,
-		},
-		[]string{"namespace", "chain_id"},
+func init() {
+	var err error
+
+	// The observable gauge is registered for its callback side effect.
+	_, err = meter.Float64ObservableGauge(
+		"sei.controller.seinode.phase",
+		metric.WithDescription("Current phase of each SeiNode (1=active, 0=inactive)"),
+		metric.WithFloat64Callback(nodePhaseTracker.Observe),
 	)
+	handleInitErr(err)
 
-	nodeLastInitDuration = prometheus.NewGaugeVec(
-		prometheus.GaugeOpts{
-			Name: "sei_controller_seinode_last_init_duration_seconds",
-			Help: "Per-node init duration, set once when node reaches Running",
-		},
-		[]string{"namespace", "name"},
+	nodePhaseTransitions, err = meter.Int64Counter(
+		"sei.controller.seinode.phase.transitions",
+		metric.WithDescription("Phase state machine transitions"),
 	)
-)
+	handleInitErr(err)
 
-func init() {
-	metrics.Registry.MustRegister(
-		nodePhaseGauge,
-		nodePhaseTransitions,
-		nodeInitDuration,
-		nodeLastInitDuration,
+	nodePhaseDuration, err = meter.Float64Histogram(
+		"sei.controller.seinode.phase.duration",
+		metric.WithDescription("Time spent in each phase before transitioning"),
+		metric.WithUnit("s"),
+		metric.WithExplicitBucketBoundaries(observability.InitBuckets...),
 	)
+	handleInitErr(err)
+}
+
+func handleInitErr(err error) {
+	if err != nil {
+		panic("otel metric init: " + err.Error())
+	}
 }
 
 func emitNodePhase(ns, name string, phase seiv1alpha1.SeiNodePhase) {
-	observability.EmitPhaseGauge(nodePhaseGauge, ns, name, string(phase), allNodePhases)
+	nodePhaseTracker.Set(ns, name, string(phase))
 }
 
 func cleanupNodeMetrics(namespace, name string) {
-	observability.DeletePhaseGauge(nodePhaseGauge, namespace, name, allNodePhases)
-	nodeLastInitDuration.DeleteLabelValues(namespace, name)
-	observability.ReconcileErrorsTotal.DeleteLabelValues(seiNodeControllerName, namespace, name)
-	planner.CleanupPlanMetrics("seinode", namespace, name)
+	nodePhaseTracker.Delete(namespace, name)
 }

internal/controller/node/metrics.go

@@ -36,7 +36,7 @@ const (
 
 func mustBuildPlan(t *testing.T, node *seiv1alpha1.SeiNode) *seiv1alpha1.TaskPlan {
 	t.Helper()
-	if err := planner.ResolvePlan(node); err != nil {
+	if err := planner.ResolvePlan(context.Background(), node); err != nil {
 		t.Fatalf("ResolvePlan: %v", err)
 	}
 	return node.Status.Plan
@@ -550,7 +550,7 @@ func TestExecutePlan_NilPlan_ReturnsError(t *testing.T) {
 
 func TestResolvePlan_FullNode(t *testing.T) {
 	node := snapshotNode()
-	if err := planner.ResolvePlan(node); err != nil {
+	if err := planner.ResolvePlan(context.Background(), node); err != nil {
 		t.Fatal(err)
 	}
 	if node.Status.Plan == nil {
@@ -560,7 +560,7 @@ func TestResolvePlan_FullNode(t *testing.T) {
 
 func TestResolvePlan_Archive(t *testing.T) {
 	node := snapshotterNode()
-	if err := planner.ResolvePlan(node); err != nil {
+	if err := planner.ResolvePlan(context.Background(), node); err != nil {
 		t.Fatal(err)
 	}
 	if node.Status.Plan == nil {
@@ -570,7 +570,7 @@ func TestResolvePlan_Archive(t *testing.T) {
 
 func TestResolvePlan_Validator(t *testing.T) {
 	node := genesisNode()
-	if err := planner.ResolvePlan(node); err != nil {
+	if err := planner.ResolvePlan(context.Background(), node); err != nil {
 		t.Fatal(err)
 	}
 	if node.Status.Plan == nil {
@@ -580,7 +580,7 @@ func TestResolvePlan_Validator(t *testing.T) {
 
 func TestResolvePlan_Replayer(t *testing.T) {
 	node := replayerNode()
-	if err := planner.ResolvePlan(node); err != nil {
+	if err := planner.ResolvePlan(context.Background(), node); err != nil {
 		t.Fatal(err)
 	}
 	if node.Status.Plan == nil {
@@ -595,7 +595,7 @@ func TestResolvePlan_NoSubSpec(t *testing.T) {
 			Image:   "sei:latest",
 		},
 	}
-	err := planner.ResolvePlan(node)
+	err := planner.ResolvePlan(context.Background(), node)
 	if err == nil {
 		t.Error("expected error for node with no sub-spec")
 	}
@@ -607,7 +607,7 @@ func TestResolvePlan_ResumesActivePlan(t *testing.T) {
 		ID:    "existing-plan",
 		Phase: seiv1alpha1.TaskPlanActive,
 	}
-	if err := planner.ResolvePlan(node); err != nil {
+	if err := planner.ResolvePlan(context.Background(), node); err != nil {
 		t.Fatal(err)
 	}
 	if node.Status.Plan.ID != "existing-plan" {
@@ -684,7 +684,7 @@ func TestExecutePlan_CompletedPlan_StaysForPlannerCleanup(t *testing.T) {
 	node.Status.Phase = seiv1alpha1.PhaseRunning
 	node.Status.Plan = nil
 	// Build a NodeUpdate plan for a Running node with drift.
-	if err := planner.ResolvePlan(node); err != nil {
+	if err := planner.ResolvePlan(context.Background(), node); err != nil {
 		t.Fatal(err)
 	}
 	g.Expect(node.Status.Plan).NotTo(BeNil())

internal/controller/node/plan_execution_test.go

@@ -18,7 +18,6 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/predicate"
 
 	seiv1alpha1 "github.com/sei-protocol/sei-k8s-controller/api/v1alpha1"
-	"github.com/sei-protocol/sei-k8s-controller/internal/controller/observability"
 	"github.com/sei-protocol/sei-k8s-controller/internal/planner"
 )
 
@@ -82,13 +81,8 @@ func (r *SeiNodeDeploymentReconciler) Reconcile(ctx context.Context, req ctrl.Re
 	statusBase := client.MergeFromWithOptions(group.DeepCopy(), client.MergeFromWithOptimisticLock{})
 	ns, name := group.Namespace, group.Name
 
-	// Networking runs before node creation so the DNS readiness gate
-	// can block until public routes are resolvable.
-	if err := timeSubstep("reconcileNetworking", func() error {
-		return r.reconcileNetworking(ctx, group)
-	}); err != nil {
+	if err := r.reconcileNetworking(ctx, group); err != nil {
 		logger.Error(err, "reconciling networking")
-		observability.ReconcileErrorsTotal.WithLabelValues(controllerName, ns, name).Inc()
 		return ctrl.Result{}, fmt.Errorf("reconciling networking: %w", err)
 	}
 
@@ -101,18 +95,14 @@ func (r *SeiNodeDeploymentReconciler) Reconcile(ctx context.Context, req ctrl.Re
 		return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
 	}
 
-	if err := timeSubstep("reconcileSeiNodes", func() error {
-		return r.reconcileSeiNodes(ctx, group)
-	}); err != nil {
+	if err := r.reconcileSeiNodes(ctx, group); err != nil {
 		logger.Error(err, "reconciling SeiNodes")
-		observability.ReconcileErrorsTotal.WithLabelValues(controllerName, ns, name).Inc()
 		return ctrl.Result{}, fmt.Errorf("reconciling SeiNodes: %w", err)
 	}
 
 	planResult, planErr := r.reconcilePlan(ctx, group)
 	if planErr != nil {
 		logger.Error(planErr, "reconciling plan")
-		observability.ReconcileErrorsTotal.WithLabelValues(controllerName, ns, name).Inc()
 		return ctrl.Result{}, fmt.Errorf("reconciling plan: %w", planErr)
 	}
 	if shouldRequeue(planResult) {
@@ -122,11 +112,8 @@ func (r *SeiNodeDeploymentReconciler) Reconcile(ctx context.Context, req ctrl.Re
 		return planResult, nil
 	}
 
-	if err := timeSubstep("reconcileMonitoring", func() error {
-		return r.reconcileMonitoring(ctx, group)
-	}); err != nil {
+	if err := r.reconcileMonitoring(ctx, group); err != nil {
 		logger.Error(err, "reconciling monitoring")
-		observability.ReconcileErrorsTotal.WithLabelValues(controllerName, ns, name).Inc()
 		return ctrl.Result{}, fmt.Errorf("reconciling monitoring: %w", err)
 	}
 
@@ -135,8 +122,7 @@ func (r *SeiNodeDeploymentReconciler) Reconcile(ctx context.Context, req ctrl.Re
 	}
 
 	emitGroupPhase(ns, name, group.Status.Phase)
-	emitGroupReplicas(ns, name, group.Spec.Replicas, group.Status.ReadyReplicas)
-	emitGroupConditions(ns, name, group.Status.Conditions)
+	emitGroupReplicas(ctx, ns, name, group.Spec.Replicas, group.Status.ReadyReplicas)
 
 	return ctrl.Result{RequeueAfter: statusPollInterval}, nil
 }