Skip to content

Commit 20a1a90

Browse files
bdchathamclaude
andauthored
feat: per-deployment internal RPC ClusterIP Service + status.rpcService (#99)
* feat: per-deployment internal RPC ClusterIP Service + status.rpcService Adds a cluster-internal ClusterIP Service to every SeiNodeDeployment so in-cluster consumers can dial a single stable DNS name ({deployment}-rpc.{namespace}.svc) rather than chasing ordinals on per-node headless Services. kube-proxy L4 load-balances across ready child pods via the existing sei.io/nodedeployment pod label. Reconciled unconditionally — lives alongside (not replacing) the .spec.networking / HTTPRoute path. - API: new RpcServiceStatus / RpcServicePorts types; additive pointer field .status.rpcService on SeiNodeDeployment. - Generator: pure generateInternalRpcService with named ports (rpc/evm-http/evm-ws/rest/grpc) per the milestone interface contract. - Reconcile: new reconcileInternalRpcService invoked from the deployment reconcile loop; populates status.rpcService in-memory for the existing single Status().Patch() flush. - Orphan path: retain-policy now strips the internal Service's ownerRef alongside the external one. - Tests: pure-generator and fake-client reconcile coverage (status stamping, ownerRef shape, idempotency, orphan path). Ports use "evm-http" in the Service (not seiconfig's "evm-rpc") because the milestone interface contract fixes those names for kube-native tools. Refs: platform#96 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: rename rpcService → internalService, drop stateful ports Three concerns folded into one follow-up: 1. Rename `rpcService` → `internalService` (status field, types, Service name suffix, tests, godoc). The Service is the single internal access point; naming it mode-neutral ages better than "RPC"-scoped naming, especially with the stateful ports dropped below. 2. Drop stateful ports (evm-ws 8546, grpc 9090) from the Service and status schema. A kube-proxy L4 LB spreads connections across pods, which breaks WebSocket subscriptions and pins HTTP/2 gRPC per-connection — neither load-balances correctly. Remaining ports: rpc (26657), evm-http (8545), rest (1317) — all stateless HTTP request/response. Stateful consumers use per-node headless Services. 3. Move internal Service orphan handling out of `orphanNetworkingResources` into a new `orphanInternalService` method. The internal Service's lifecycle is unconditional; it should not be bundled with the networking-resources teardown. Added a test for `.spec.networking → nil` transitions confirming the internal Service survives. All tests green (lint + test). CRD + DeepCopy regenerated via `make manifests generate`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent fd8708a commit 20a1a90

7 files changed

Lines changed: 597 additions & 0 deletions

File tree

api/v1alpha1/seinodedeployment_types.go

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -248,12 +248,48 @@ type SeiNodeDeploymentStatus struct {
248248
// +optional
249249
NetworkingStatus *NetworkingStatus `json:"networkingStatus,omitempty"`
250250

251+
// InternalService reports the in-cluster ClusterIP Service that kube-proxy
252+
// load-balances across healthy child pods. Populated unconditionally —
253+
// this path is independent of .spec.networking.
254+
// +optional
255+
InternalService *InternalServiceStatus `json:"internalService,omitempty"`
256+
251257
// +listType=map
252258
// +listMapKey=type
253259
// +optional
254260
Conditions []metav1.Condition `json:"conditions,omitempty"`
255261
}
256262

263+
// InternalServiceStatus reports the resolved in-cluster ClusterIP Service
264+
// exposed for a SeiNodeDeployment. Consumers resolve the service at
265+
// {name}.{namespace}.svc and dial the named port from {ports}.
266+
type InternalServiceStatus struct {
267+
// Name is the Kubernetes Service name (always "{deployment-name}-internal").
268+
Name string `json:"name"`
269+
270+
// Namespace is the Service's namespace (always equal to the deployment's
271+
// namespace).
272+
Namespace string `json:"namespace"`
273+
274+
// Ports enumerates the named ports on the Service.
275+
Ports InternalServicePorts `json:"ports"`
276+
}
277+
278+
// InternalServicePorts is the set of named ports advertised on the internal
279+
// ClusterIP Service. Only stateless HTTP request/response protocols are
280+
// exposed here — stateful protocols (EVM WebSocket, gRPC streaming, P2P
281+
// gossip) do not load-balance correctly behind a kube-proxy L4 LB, and
282+
// consumers needing those use the per-node headless Services instead.
283+
// Field names are part of the public interface contract.
284+
type InternalServicePorts struct {
285+
// Rpc is the Tendermint / CometBFT RPC port (26657).
286+
Rpc int32 `json:"rpc"`
287+
// EvmHttp is the EVM JSON-RPC HTTP port (8545).
288+
EvmHttp int32 `json:"evmHttp"`
289+
// Rest is the Cosmos REST (LCD) port (1317).
290+
Rest int32 `json:"rest"`
291+
}
292+
257293
// GroupNodeStatus is a summary of a child SeiNode's state.
258294
type GroupNodeStatus struct {
259295
// Name is the SeiNode resource name.

api/v1alpha1/zz_generated.deepcopy.go

Lines changed: 36 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

config/crd/sei.io_seinodedeployments.yaml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -795,6 +795,45 @@ spec:
795795
items:
796796
type: string
797797
type: array
798+
internalService:
799+
description: |-
800+
InternalService reports the in-cluster ClusterIP Service that kube-proxy
801+
load-balances across healthy child pods. Populated unconditionally —
802+
this path is independent of .spec.networking.
803+
properties:
804+
name:
805+
description: Name is the Kubernetes Service name (always "{deployment-name}-internal").
806+
type: string
807+
namespace:
808+
description: |-
809+
Namespace is the Service's namespace (always equal to the deployment's
810+
namespace).
811+
type: string
812+
ports:
813+
description: Ports enumerates the named ports on the Service.
814+
properties:
815+
evmHttp:
816+
description: EvmHttp is the EVM JSON-RPC HTTP port (8545).
817+
format: int32
818+
type: integer
819+
rest:
820+
description: Rest is the Cosmos REST (LCD) port (1317).
821+
format: int32
822+
type: integer
823+
rpc:
824+
description: Rpc is the Tendermint / CometBFT RPC port (26657).
825+
format: int32
826+
type: integer
827+
required:
828+
- evmHttp
829+
- rest
830+
- rpc
831+
type: object
832+
required:
833+
- name
834+
- namespace
835+
- ports
836+
type: object
798837
networkingStatus:
799838
description: NetworkingStatus reports the observed state of networking
800839
resources.

internal/controller/nodedeployment/controller.go

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,11 @@ func (r *SeiNodeDeploymentReconciler) Reconcile(ctx context.Context, req ctrl.Re
8181
statusBase := client.MergeFromWithOptions(group.DeepCopy(), client.MergeFromWithOptimisticLock{})
8282
ns, name := group.Namespace, group.Name
8383

84+
if err := r.reconcileInternalService(ctx, group); err != nil {
85+
logger.Error(err, "reconciling internal RPC service")
86+
return ctrl.Result{}, fmt.Errorf("reconciling internal RPC service: %w", err)
87+
}
88+
8489
if err := r.reconcileNetworking(ctx, group); err != nil {
8590
logger.Error(err, "reconciling networking")
8691
return ctrl.Result{}, fmt.Errorf("reconciling networking: %w", err)
@@ -160,6 +165,9 @@ func (r *SeiNodeDeploymentReconciler) handleDeletion(ctx context.Context, group
160165
if err := r.orphanNetworkingResources(ctx, group); err != nil {
161166
return ctrl.Result{}, fmt.Errorf("orphaning networking resources: %w", err)
162167
}
168+
if err := r.orphanInternalService(ctx, group); err != nil {
169+
return ctrl.Result{}, fmt.Errorf("orphaning internal Service: %w", err)
170+
}
163171
} else {
164172
if err := r.deleteNetworkingResources(ctx, group); err != nil {
165173
r.Recorder.Eventf(group, corev1.EventTypeWarning, "DeleteFailed", "Failed to clean up networking resources: %v", err)
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
package nodedeployment
2+
3+
import (
4+
"context"
5+
"fmt"
6+
7+
seiconfig "github.com/sei-protocol/sei-config"
8+
corev1 "k8s.io/api/core/v1"
9+
apierrors "k8s.io/apimachinery/pkg/api/errors"
10+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
11+
"k8s.io/apimachinery/pkg/types"
12+
"k8s.io/apimachinery/pkg/util/intstr"
13+
ctrl "sigs.k8s.io/controller-runtime"
14+
"sigs.k8s.io/controller-runtime/pkg/client"
15+
16+
seiv1alpha1 "github.com/sei-protocol/sei-k8s-controller/api/v1alpha1"
17+
)
18+
19+
// internalServiceName returns the deterministic name of the ClusterIP
20+
// Service that publishes in-cluster access for a SeiNodeDeployment.
21+
func internalServiceName(group *seiv1alpha1.SeiNodeDeployment) string {
22+
return fmt.Sprintf("%s-internal", group.Name)
23+
}
24+
25+
// reconcileInternalService applies the cluster-internal ClusterIP Service
26+
// that kube-proxy load-balances across healthy child pods and stamps
27+
// status.internalService on the group. This runs unconditionally — it is
28+
// independent of .spec.networking and lives alongside the external Service /
29+
// HTTPRoute reconciliation.
30+
func (r *SeiNodeDeploymentReconciler) reconcileInternalService(ctx context.Context, group *seiv1alpha1.SeiNodeDeployment) error {
31+
desired := generateInternalService(group)
32+
desired.SetGroupVersionKind(corev1.SchemeGroupVersion.WithKind("Service"))
33+
if err := ctrl.SetControllerReference(group, desired, r.Scheme); err != nil {
34+
return fmt.Errorf("setting owner reference on internal Service: %w", err)
35+
}
36+
//nolint:staticcheck // migrating to typed ApplyConfiguration is a separate effort
37+
if err := r.Patch(ctx, desired, client.Apply, fieldOwner, client.ForceOwnership); err != nil {
38+
return fmt.Errorf("applying internal Service: %w", err)
39+
}
40+
41+
group.Status.InternalService = &seiv1alpha1.InternalServiceStatus{
42+
Name: desired.Name,
43+
Namespace: desired.Namespace,
44+
Ports: seiv1alpha1.InternalServicePorts{
45+
Rpc: seiconfig.PortRPC,
46+
EvmHttp: seiconfig.PortEVMHTTP,
47+
Rest: seiconfig.PortREST,
48+
},
49+
}
50+
return nil
51+
}
52+
53+
// orphanInternalService strips the owner reference on the internal Service
54+
// so the resource survives parent deletion under DeletionPolicy=Retain. The
55+
// internal Service's lifecycle is unconditional — it does not participate
56+
// in the networking-resources teardown path — so its retain handling lives
57+
// alongside its create path, not alongside orphanNetworkingResources.
58+
func (r *SeiNodeDeploymentReconciler) orphanInternalService(ctx context.Context, group *seiv1alpha1.SeiNodeDeployment) error {
59+
svc := &corev1.Service{}
60+
err := r.Get(ctx, types.NamespacedName{Name: internalServiceName(group), Namespace: group.Namespace}, svc)
61+
if apierrors.IsNotFound(err) {
62+
return nil
63+
}
64+
if err != nil {
65+
return fmt.Errorf("fetching internal Service for orphan: %w", err)
66+
}
67+
if err := r.removeOwnerRef(ctx, svc, group); err != nil {
68+
return fmt.Errorf("orphaning internal Service: %w", err)
69+
}
70+
return nil
71+
}
72+
73+
// generateInternalService is a pure function that produces the desired
74+
// ClusterIP Service for in-cluster consumers. The Service is always created
75+
// regardless of .spec.networking. kube-proxy load-balances traffic across
76+
// ready child pods using the deployment-scoped pod label.
77+
//
78+
// Only stateless HTTP request/response ports are exposed (rpc, evm-http,
79+
// rest). Stateful protocols — EVM WebSocket (8546), gRPC / HTTP/2 streaming
80+
// (9090), P2P gossip — do not work correctly behind a kube-proxy L4 LB
81+
// (WebSocket subscriptions pin state to a pod; HTTP/2 connections pin
82+
// per-connection). Consumers needing those use the per-node headless
83+
// Services for deterministic per-pod addressing.
84+
func generateInternalService(group *seiv1alpha1.SeiNodeDeployment) *corev1.Service {
85+
return &corev1.Service{
86+
ObjectMeta: metav1.ObjectMeta{
87+
Name: internalServiceName(group),
88+
Namespace: group.Namespace,
89+
Labels: resourceLabels(group),
90+
Annotations: managedByAnnotations(),
91+
},
92+
Spec: corev1.ServiceSpec{
93+
Type: corev1.ServiceTypeClusterIP,
94+
// Select by deployment membership only (not revision): during a
95+
// rollout, kube-proxy should keep routing to whichever pods are
96+
// Ready, irrespective of which generation they belong to.
97+
Selector: groupOnlySelector(group),
98+
// kube-proxy must exclude not-ready pods so clients never see
99+
// partially-bootstrapped nodes.
100+
PublishNotReadyAddresses: false,
101+
Ports: []corev1.ServicePort{
102+
{
103+
Name: "rpc",
104+
Port: seiconfig.PortRPC,
105+
TargetPort: intstr.FromInt32(seiconfig.PortRPC),
106+
Protocol: corev1.ProtocolTCP,
107+
},
108+
{
109+
Name: "evm-http",
110+
Port: seiconfig.PortEVMHTTP,
111+
TargetPort: intstr.FromInt32(seiconfig.PortEVMHTTP),
112+
Protocol: corev1.ProtocolTCP,
113+
},
114+
{
115+
Name: "rest",
116+
Port: seiconfig.PortREST,
117+
TargetPort: intstr.FromInt32(seiconfig.PortREST),
118+
Protocol: corev1.ProtocolTCP,
119+
},
120+
},
121+
},
122+
}
123+
}

0 commit comments

Comments
 (0)