diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index ab9ded873..b559524fc 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -1862,8 +1862,8 @@ jobs: job_timeout: 75 # Boots a kind cluster, installs the chart, runs the bucket's suite. - bucket: optional-components parallel: 1 - test_timeout: 20m # one lifecycle test: helm upgrade, then mcp, syslog, frp tunnel and core checks in order - job_timeout: 60 # Boots a kind cluster, installs the chart, runs the bucket's suite. + test_timeout: 35m # lifecycle plus two-GameServer TCP/UDP relay identity, outage and recovery checks + job_timeout: 75 # Boots a kind cluster, installs the chart, runs the bucket's suite. env: CLUSTER: gameplane-e2e CTX: kind-gameplane-e2e @@ -1971,8 +1971,8 @@ jobs: job_timeout: 75 # Boots a kind cluster, installs the chart, runs the bucket's suite. - bucket: optional-components parallel: 1 - test_timeout: 20m # one lifecycle test: helm upgrade, then mcp, syslog, frp tunnel and core checks in order - job_timeout: 60 # Boots a kind cluster, installs the chart, runs the bucket's suite. + test_timeout: 35m # lifecycle plus two-GameServer TCP/UDP relay identity, outage and recovery checks + job_timeout: 75 # Boots a kind cluster, installs the chart, runs the bucket's suite. env: CLUSTER: gameplane-e2e CTX: kind-gameplane-e2e diff --git a/docs/relay-health.md b/docs/relay-health.md new file mode 100644 index 000000000..f44c4267f --- /dev/null +++ b/docs/relay-health.md @@ -0,0 +1,117 @@ +# Relay readiness and registration identity + +The frp/Playit wrapper's process health and its provider connection are separate. +On Pod TCP port 8091, `/healthz` reports that the wrapper's HTTP server is alive, +`/readyz` returns 200 only when the provider connection and configured forwards +are confirmed, and `/status` returns a small JSON state without credentials or +provider response text. The operator installs a startup probe on `/healthz` and +a readiness probe on `/readyz` for frp. Playit uses `/healthz` for Pod readiness +because its pinned provider cannot supply live connection evidence. An external relay outage does not trigger a +liveness restart of the wrapper. + +Provider checks run every two seconds with a two-second query deadline. Local +observations expire after ten seconds, and child exit immediately clears them. +The operator also requires the current Deployment generation, updated replicas +and the expected readiness probe; an old Pod without the probe cannot satisfy +the new readiness contract. Configured addresses remain visible while the +connection is unconfirmed, but `TunnelReady` is not true. + +## frp + +The generated frpc TOML sets `user` to the immutable GameServer UID. The wire +registration is `.`; the GameTemplate's port names and +`remotePorts[].name` do not change. Pod and container restarts reuse the UID. +Deleting and recreating a GameServer creates a new identity, including when its +namespace and name are reused. Distinct clusters' UIDs also separate their +registrations on a shared frps. + +frpc's authenticated status API listens only on `127.0.0.1:7400`. It uses the +mounted token as its local admin password; it is never exposed by a Service or +forwarded from the public relay. The wrapper does not use an environment HTTP +proxy or follow redirects for this request. Every configured TCP/UDP proxy must +appear with the matching name and protocol, a `running` status and no error. +Authentication failure, an occupied registration, absent proxy or unavailable +local API cannot produce readiness. This confirms the relay registrations, not +the health of a sleeping game backend. + +Application heartbeat settings are five seconds between requests and fifteen +seconds before a silent control connection is closed; TCP multiplex keepalive +also uses five seconds. Allow the provider detection interval, local check and +Kubernetes probe/reconcile intervals for an outage to reach the dashboard. + +### Upgrade and replacement + +Upgrade the operator and tunnel image together. The new image requires +`GAMESERVER_UID`, so a mixed old-operator/new-image installation fails closed. +The upgraded operator reconciles existing tunnel Deployments in place, replaces +their Pod templates with the UID environment variable and current probes, and +uses Recreate to retire the old client before starting its replacement. No +manual Deployment environment edit is required; upgrade the operator first or +together with its configured tunnel image. +Existing unprefixed proxy names change to UID-prefixed names. Stop the old +client, wait for frps to unregister its old proxies and release their public +ports, or remove those registrations using the frps administrator's normal +procedure. Readiness stays false while a port is still occupied. Do not remove +an unrelated GameServer's registration to make an upgrade succeed. + +The operator uses one replica and the Recreate strategy for relay Deployments. +It never deliberately rolls out two clients with the same provider identity. +Replacement has a short forwarding interruption while the old connection +closes and the new one registers. When a deleted old Pod's node is unreachable, +wait for frps to expire its old session before expecting the replacement ready. +Do not manually scale a relay Deployment above one replica. + +## Playit connection evidence + +The pinned Playit v1.0.10 IPC v2 contract exposes lifecycle and address +assignment, but no authenticated relay heartbeat or connection timestamp. +Its daemon caches `Running` after a successful API refresh and leaves that +cache unchanged when later refreshes fail. Receiving another `get_state` +response therefore cannot prove that a relay is connected or refresh a health +lease. Process age and the presence of an assigned address do not prove it +either. + +Gameplane keeps reporting assigned addresses, but Playit's relay `/readyz` returns +503 and `TunnelReady` reports `RelayConnectionUnknown`; before assignment it +reports `AwaitingAddress`. Its Kubernetes readiness probe uses `/healthz`, so a +running wrapper can become Pod Ready while its connection evidence remains +unknown. The child keeps running and forwarding when it can, and Kubernetes does +not restart it solely for this unknown connection state. The JSON status distinguishes known assignment from +unknown connection. This is a deliberate limitation of the pinned provider, +not a claim that every Playit tunnel is disconnected. + +Positive Playit readiness requires a future provider interface that exposes +fresh, authenticated relay connection evidence, or a separately designed +end-to-end probe. The relevant upstream definitions are +[the IPC model](https://github.com/playit-cloud/playit-agent/blob/v1.0.10/packages/playit-ipc/src/model.rs) +and [the daemon's cached lifecycle publisher](https://github.com/playit-cloud/playit-agent/blob/v1.0.10/packages/playitd/src/daemon.rs). + +## Retry and resource policy + +Unexpected child exits, including clean exits, wait 1, 2, 4, 8 seconds and so on, +up to 300 seconds, before another attempt. Cancellation interrupts the wait. +There is no jitter, as requested by the maintainer. Retry escalation resets only +after one uninterrupted minute of confirmed provider readiness. A failed or +stale observation breaks that interval. A long-lived disconnected process does +not reset it. Playit's unsupported connection evidence cannot reset it either. +The embedded Tailscale loop uses the same interval, fed by confirmed online +tailnet status and active forwarding listeners rather than process lifetime. + +Resources use the existing `operator.tunnelResources` Helm value and +`--tunnel-resources` operator flag from the workload-resource policy. Omitted or +null configuration defaults to requests of 25m CPU and 64Mi memory, with no +limits. A non-null empty object explicitly clears defaults; supplied requests +and limits replace them. These are scheduling reservations, not a fleet +capacity guarantee. Readiness and replacement changes preserve this contract. + +## Regression coverage + +The CI optional-components lifecycle test runs two GameServers from the same +TCP/UDP echo template against one real frps, with different remote ports. An +independent client Pod checks replies identify the intended backend through +both protocols. It also covers registration rejection, replacement isolation, +invalid credentials, a shared relay outage and recovery, and credential-free +logs. Unit tests cover UID validation/stability, namespace/recreation identity, +status decoding, stale expiry, bounded retries/cancellation, healthy-interval +reset, old-deployment readiness, Playit's cached-running limitation, and +resource override removal. These regression tests are executed in CI. diff --git a/docs/tunnels.md b/docs/tunnels.md index 66f965301..504d98d96 100644 --- a/docs/tunnels.md +++ b/docs/tunnels.md @@ -509,3 +509,8 @@ changes. the same security practices as you would for a public-facing LoadBalancer (e.g., disable admin consoles, run security updates, restrict in-game permissions). + +See [relay readiness and registration identity](relay-health.md) for the frp +UID migration, replacement policy, health probes, retry reset and the pinned +Playit version's connection-status limitation. An assigned address alone does +not make a relay ready. diff --git a/i18n/locales/en/app.json b/i18n/locales/en/app.json index eeaeda262..d450aa1e1 100644 --- a/i18n/locales/en/app.json +++ b/i18n/locales/en/app.json @@ -567,6 +567,10 @@ "defaultMessage": "These settings apply to the backing Service, not to tunnel traffic: {settings}.", "description": "Informational condition when hostname, source ranges or node ports are set on a server with a tunnel. settings is a comma-separated list of field names." }, + "condition.tunnelready.awaitingaddress": { + "defaultMessage": "Waiting for the provider to assign a tunnel endpoint; the relay connection is unconfirmed.", + "description": "TunnelReady condition before Playit assigns an endpoint." + }, "condition.tunnelready.deploymentnotready": { "defaultMessage": "Waiting for the tunnel deployment to become ready.", "description": "TunnelReady condition while the tunnel pod is not ready." @@ -587,6 +591,10 @@ "defaultMessage": "{provider, select, playit {The tunnel deployment is ready. Waiting for the playit endpoint assignment.} tailscale {The tunnel deployment is ready and authenticated.} other {The tunnel deployment is ready and all ports are mapped.}}", "description": "TunnelReady condition when the tunnel works." }, + "condition.tunnelready.relayconnectionunknown": { + "defaultMessage": "The relay connection cannot be confirmed. Check the provider's connection status and the tunnel deployment's readiness probe.", + "description": "TunnelReady condition when provider connection evidence is missing or unsupported." + }, "condition.tunnelready.tunnelcredentialrefused": { "defaultMessage": "The tunnel credentials secret ''{secret}'' is not owned by this GameServer. {remedy, select, defaultName {Add an ownerReference to this GameServer to the Secret, or delete the Secret and save the credential again from the dashboard.} other {Save the credential from the dashboard, or add an ownerReference to this GameServer to the Secret.}}", "description": "TunnelReady condition when the operator refuses to mount a credentials secret. secret is the secret name, remedy is defaultName or other." diff --git a/operator/hack/condition-ids/main.go b/operator/hack/condition-ids/main.go index ef0c4fa69..364afe085 100644 --- a/operator/hack/condition-ids/main.go +++ b/operator/hack/condition-ids/main.go @@ -125,6 +125,8 @@ var rows = []row{ {types: []string{"TunnelReady"}, reason: "InvalidTailscaleTags", english: "Correct the Tailscale tags: use an optional tag: prefix, a letter first, then only letters, digits or hyphens.", description: "TunnelReady condition when Tailscale access-control tags have invalid syntax."}, {types: []string{"TunnelReady"}, reason: "InvalidConfig", english: "{provider, select, none {The tunnel is enabled but the server has no tunnel configuration.} other {The ''{provider}'' tunnel provider is selected but its configuration is missing.}}", description: "TunnelReady condition for an incomplete tunnel setup. provider is the selected provider, or none."}, {types: []string{"TunnelReady"}, reason: "DeploymentNotReady", english: "Waiting for the tunnel deployment to become ready.", description: "TunnelReady condition while the tunnel pod is not ready."}, + {types: []string{"TunnelReady"}, reason: "RelayConnectionUnknown", english: "The relay connection cannot be confirmed. Check the provider's connection status and the tunnel deployment's readiness probe.", description: "TunnelReady condition when provider connection evidence is missing or unsupported."}, + {types: []string{"TunnelReady"}, reason: "AwaitingAddress", english: "Waiting for the provider to assign a tunnel endpoint; the relay connection is unconfirmed.", description: "TunnelReady condition before Playit assigns an endpoint."}, {types: []string{"TunnelReady"}, reason: "PortNotMapped", english: "No frp remote port is configured for: {ports}.", description: "TunnelReady condition for frp when advertised ports have no remote port. ports is a comma-separated list of port names."}, {types: []string{"TunnelReady"}, reason: "NoCredentials", english: "No credentials secret was provided, so the ''{provider}'' tunnel cannot authenticate.", description: "TunnelReady condition when a tunnel provider needs credentials that are missing."}, {types: []string{"TunnelReady"}, reason: "Ready", english: "{provider, select, playit {The tunnel deployment is ready. Waiting for the playit endpoint assignment.} tailscale {The tunnel deployment is ready and authenticated.} other {The tunnel deployment is ready and all ports are mapped.}}", description: "TunnelReady condition when the tunnel works."}, diff --git a/operator/internal/controller/gameserver_relay_health_test.go b/operator/internal/controller/gameserver_relay_health_test.go new file mode 100644 index 000000000..9b48f40ae --- /dev/null +++ b/operator/internal/controller/gameserver_relay_health_test.go @@ -0,0 +1,173 @@ +package controller + +import ( + "reflect" + "testing" + + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + + gameplanev1alpha1 "github.com/GameplanePanel/gameplane/operator/api/v1alpha1" +) + +func TestRelayDeploymentIdentityProbesAndResourceRemoval(t *testing.T) { + t.Parallel() + s := wipeScheme(t) + for _, identity := range []struct{ namespace, uid string }{{"one", "uid-one"}, {"two", "uid-two"}, {"one", "replacement-uid"}} { + gs := &gameplanev1alpha1.GameServer{ObjectMeta: metav1.ObjectMeta{Name: "same", Namespace: identity.namespace, UID: types.UID(identity.uid)}, Spec: gameplanev1alpha1.GameServerSpec{Networking: gameplanev1alpha1.GameServerNetworking{Tunnel: &gameplanev1alpha1.GameServerTunnel{Enabled: true, Provider: "frp", Frp: &gameplanev1alpha1.FrpTunnelSpec{ServerAddr: "relay.example"}}}}} + c := fake.NewClientBuilder().WithScheme(s).WithObjects(gs).Build() + r := &GameServerReconciler{Client: c, Scheme: s} + for _, resources := range []*corev1.ResourceRequirements{nil, {Requests: corev1.ResourceList{corev1.ResourceCPU: resource.MustParse("90m")}, Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("128Mi")}}, {}, nil} { + r.TunnelResources = resources + if err := r.reconcileTunnel(t.Context(), gs, singlePortTemplate(true, "minecraft"), true); err != nil { + t.Fatal(err) + } + var dep appsv1.Deployment + if err := c.Get(t.Context(), client.ObjectKey{Namespace: gs.Namespace, Name: gs.Name + "-tunnel"}, &dep); err != nil { + t.Fatal(err) + } + if dep.Spec.Strategy.Type != appsv1.RecreateDeploymentStrategyType || dep.Spec.Strategy.RollingUpdate != nil { + t.Fatal("relay permits overlapping identities") + } + container := dep.Spec.Template.Spec.Containers[0] + if container.StartupProbe == nil || container.StartupProbe.HTTPGet == nil || container.StartupProbe.HTTPGet.Path != "/healthz" || !hasRelayReadinessProbe(&dep) || container.LivenessProbe != nil { + t.Fatal("provider outage must affect readiness without restarting the supervisor") + } + if !reflect.DeepEqual(container.Resources, auxiliaryResources(resources, "25m", "64Mi")) { + t.Fatal("resource defaults or override removal changed") + } + found := false + for _, env := range container.Env { + if env.Name == "GAMESERVER_UID" { + found = env.Value == identity.uid + } + } + if !found { + t.Fatal("UID identity missing or unstable") + } + } + } +} + +func TestRelayConditionsRequireCurrentProviderProbe(t *testing.T) { + t.Parallel() + s := wipeScheme(t) + gs := &gameplanev1alpha1.GameServer{ObjectMeta: metav1.ObjectMeta{Name: "relay", Namespace: "games", UID: "uid"}, Spec: gameplanev1alpha1.GameServerSpec{Networking: gameplanev1alpha1.GameServerNetworking{Tunnel: &gameplanev1alpha1.GameServerTunnel{Provider: "frp", Frp: &gameplanev1alpha1.FrpTunnelSpec{ServerAddr: "relay"}}}}} + c := fake.NewClientBuilder().WithScheme(s).WithObjects(gs).Build() + r := &GameServerReconciler{Client: c, Scheme: s} + if err := r.reconcileTunnel(t.Context(), gs, singlePortTemplate(true, "minecraft"), true); err != nil { + t.Fatal(err) + } + var dep appsv1.Deployment + if err := c.Get(t.Context(), client.ObjectKey{Namespace: gs.Namespace, Name: "relay-tunnel"}, &dep); err != nil { + t.Fatal(err) + } + dep.Generation = 2 + dep.Status = appsv1.DeploymentStatus{ReadyReplicas: 1, UpdatedReplicas: 1, ObservedGeneration: 2} + assert := func(want metav1.ConditionStatus) { + t.Helper() + cond := meta.FindStatusCondition(computeTunnelConditions(gs, tunnelPlan{wantTunnel: true}, &dep), "TunnelReady") + if cond == nil || cond.Status != want { + t.Fatalf("condition=%+v want=%s", cond, want) + } + } + assert(metav1.ConditionTrue) + dep.Status.ObservedGeneration = 1 + assert(metav1.ConditionUnknown) + dep.Status.ObservedGeneration = 2 + dep.Status.ReadyReplicas = 0 + assert(metav1.ConditionFalse) + dep.Status.ReadyReplicas = 1 + dep.Spec.Template.Spec.Containers[0].ReadinessProbe = nil + assert(metav1.ConditionUnknown) +} + +func TestExistingFrpDeploymentGainsUIDDuringReconciliation(t *testing.T) { + t.Parallel() + s := wipeScheme(t) + gs := &gameplanev1alpha1.GameServer{ObjectMeta: metav1.ObjectMeta{Name: "legacy", Namespace: "games", UID: "legacy-uid"}, Spec: gameplanev1alpha1.GameServerSpec{Networking: gameplanev1alpha1.GameServerNetworking{Tunnel: &gameplanev1alpha1.GameServerTunnel{Enabled: true, Provider: "frp", Frp: &gameplanev1alpha1.FrpTunnelSpec{ServerAddr: "relay"}}}}} + old := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: "legacy-tunnel", Namespace: "games", UID: "deployment-uid"}, Spec: appsv1.DeploymentSpec{Strategy: appsv1.DeploymentStrategy{Type: appsv1.RollingUpdateDeploymentStrategyType, RollingUpdate: &appsv1.RollingUpdateDeployment{}}, Template: corev1.PodTemplateSpec{Spec: corev1.PodSpec{Containers: []corev1.Container{{Name: "tunnel", Image: "old", Env: []corev1.EnvVar{{Name: "TUNNEL_TYPE", Value: "frp"}}}}}}}} + c := fake.NewClientBuilder().WithScheme(s).WithObjects(gs, old).Build() + r := &GameServerReconciler{Client: c, Scheme: s} + for i := 0; i < 2; i++ { + if err := r.reconcileTunnel(t.Context(), gs, singlePortTemplate(true, "minecraft"), true); err != nil { + t.Fatal(err) + } + var updated appsv1.Deployment + if err := c.Get(t.Context(), client.ObjectKeyFromObject(old), &updated); err != nil { + t.Fatal(err) + } + if updated.UID != old.UID || updated.Spec.Strategy.Type != appsv1.RecreateDeploymentStrategyType || updated.Spec.Strategy.RollingUpdate != nil { + t.Fatalf("legacy Deployment was not migrated in place: %+v", updated.Spec.Strategy) + } + found := false + for _, env := range updated.Spec.Template.Spec.Containers[0].Env { + if env.Name == "GAMESERVER_UID" { + found = env.Value == string(gs.UID) + } + } + if !found || !hasRelayReadinessProbe(&updated) { + t.Fatal("existing Deployment lacks UID or forwarding probe") + } + } +} + +func TestPlayitPodReadinessUsesAvailableProcessSignal(t *testing.T) { + t.Parallel() + s := wipeScheme(t) + gs := &gameplanev1alpha1.GameServer{ObjectMeta: metav1.ObjectMeta{Name: "playit", Namespace: "games", UID: "playit-uid"}, Spec: gameplanev1alpha1.GameServerSpec{Networking: gameplanev1alpha1.GameServerNetworking{Tunnel: &gameplanev1alpha1.GameServerTunnel{Enabled: true, Provider: "playit", Playit: &gameplanev1alpha1.PlayitTunnelSpec{}}}}} + c := fake.NewClientBuilder().WithScheme(s).WithObjects(gs).Build() + r := &GameServerReconciler{Client: c, Scheme: s} + if err := r.reconcileTunnel(t.Context(), gs, singlePortTemplate(true, "minecraft"), true); err != nil { + t.Fatal(err) + } + var dep appsv1.Deployment + if err := c.Get(t.Context(), client.ObjectKey{Namespace: gs.Namespace, Name: gs.Name + "-tunnel"}, &dep); err != nil { + t.Fatal(err) + } + probe := dep.Spec.Template.Spec.Containers[0].ReadinessProbe + if probe == nil || probe.HTTPGet == nil || probe.HTTPGet.Path != "/healthz" || probe.HTTPGet.Port.IntValue() != 8091 { + t.Fatalf("Playit requires unsupported connection evidence: %+v", probe) + } + condition := meta.FindStatusCondition(computeTunnelConditions(gs, tunnelPlan{wantTunnel: true}, &dep), "TunnelReady") + if condition == nil || condition.Status == metav1.ConditionTrue { + t.Fatalf("process readiness implied relay readiness: %+v", condition) + } +} + +func TestPlayitAssignmentDoesNotImplyRelayReady(t *testing.T) { + t.Parallel() + gs := &gameplanev1alpha1.GameServer{Spec: gameplanev1alpha1.GameServerSpec{Networking: gameplanev1alpha1.GameServerNetworking{Tunnel: &gameplanev1alpha1.GameServerTunnel{Provider: "playit", Playit: &gameplanev1alpha1.PlayitTunnelSpec{}, CredentialsSecretRef: &gameplanev1alpha1.SecretNameRef{Name: "relay-credentials"}}}}} + dep := tunnelReadyDeployment() + // A legacy deployment without the provider probe remains unknown even + // when its Pod is Ready; assignment alone cannot bypass that requirement. + legacy := meta.FindStatusCondition(computeTunnelConditions(gs, tunnelPlan{wantTunnel: true}, dep), "TunnelReady") + if legacy == nil || legacy.Status != metav1.ConditionUnknown || legacy.Reason != "RelayConnectionUnknown" { + t.Fatalf("legacy playit readiness=%+v, want unknown connection", legacy) + } + dep.Generation = 1 + dep.Status.ObservedGeneration = 1 + dep.Status.UpdatedReplicas = 1 + dep.Spec.Template.Spec.Containers = []corev1.Container{{Name: "tunnel", ReadinessProbe: &corev1.Probe{ProbeHandler: corev1.ProbeHandler{HTTPGet: &corev1.HTTPGetAction{Path: "/readyz", Port: intstr.FromInt32(8091)}}}}} + for _, probePath := range []string{"/readyz", "/healthz"} { + dep.Spec.Template.Spec.Containers[0].ReadinessProbe.HTTPGet.Path = probePath + for _, endpoints := range [][]gameplanev1alpha1.GameServerEndpoint{nil, {{Name: "game", Host: "example.playit.gg", Port: 30000}}} { + gs.Status.TunnelEndpoints = endpoints + cond := meta.FindStatusCondition(computeTunnelConditions(gs, tunnelPlan{wantTunnel: true}, dep), "TunnelReady") + wantStatus, wantReason := metav1.ConditionUnknown, "RelayConnectionUnknown" + if len(endpoints) == 0 { + wantStatus, wantReason = metav1.ConditionFalse, "AwaitingAddress" + } + if cond == nil || cond.Status != wantStatus || cond.Reason != wantReason { + t.Fatalf("playit readiness=%+v, want %s/%s", cond, wantStatus, wantReason) + } + } + } +} diff --git a/operator/internal/controller/gameserver_status.go b/operator/internal/controller/gameserver_status.go index c43c3d613..dcc1454ec 100644 --- a/operator/internal/controller/gameserver_status.go +++ b/operator/internal/controller/gameserver_status.go @@ -1128,6 +1128,15 @@ func computeTunnelConditions( tunnelReady.Message = "Tailscale supports only TCP and UDP; unsupported advertised ports: " + strings.Join(plan.unsupportedPorts, ", ") return upsertCondition(conds, tunnelReady) } + if tunnel.Provider == "playit" && (tunnel.Playit == nil || tunnel.CredentialsSecretRef == nil || tunnel.CredentialsSecretRef.Name == "") { + tunnelReady.Status = metav1.ConditionFalse + tunnelReady.Reason = "NoCredentials" + tunnelReady.Message = "playit configuration and credentials are required before relay connection can be confirmed" + if tunnel.Playit == nil { + tunnelReady.Reason = "InvalidConfig" + } + return upsertCondition(conds, tunnelReady) + } if tunnelDep == nil { tunnelReady.Status = metav1.ConditionFalse tunnelReady.Reason = "DeploymentNotReady" @@ -1135,11 +1144,34 @@ func computeTunnelConditions( conds = upsertCondition(conds, tunnelReady) return conds } + if tunnel.Provider == "frp" || tunnel.Provider == "playit" { + probePresent := hasRelayReadinessProbe(tunnelDep) + if tunnel.Provider == "playit" { + probePresent = probePresent || hasTunnelReadinessProbe(tunnelDep, "/healthz") + } + if !probePresent || tunnelDep.Status.ObservedGeneration < tunnelDep.Generation || tunnelDep.Status.UpdatedReplicas == 0 { + tunnelReady.Status = metav1.ConditionUnknown + tunnelReady.Reason = "RelayConnectionUnknown" + tunnelReady.Message = "waiting for the current relay deployment and its connection readiness probe" + return upsertCondition(conds, tunnelReady) + } + if tunnel.Provider == "playit" { + tunnelReady.Status = metav1.ConditionUnknown + tunnelReady.Reason = "RelayConnectionUnknown" + tunnelReady.Message = "playit v1.0.10 IPC reports cached assignment but cannot confirm relay connection; check provider connectivity separately" + if len(gs.Status.TunnelEndpoints) == 0 { + tunnelReady.Status = metav1.ConditionFalse + tunnelReady.Reason = "AwaitingAddress" + tunnelReady.Message = "waiting for playit endpoint assignment; relay connection is unconfirmed" + } + return upsertCondition(conds, tunnelReady) + } + } if tunnelDep.Status.ReadyReplicas == 0 { tunnelReady.Status = metav1.ConditionFalse tunnelReady.Reason = "DeploymentNotReady" - tunnelReady.Message = "tunnel deployment has no ready replicas" + tunnelReady.Message = "tunnel deployment has no ready replicas; relay connection or forward registration is unconfirmed" conds = upsertCondition(conds, tunnelReady) return conds } @@ -1170,7 +1202,7 @@ func computeTunnelConditions( // All endpoints known: deployment ready and mappings complete. tunnelReady.Status = metav1.ConditionTrue tunnelReady.Reason = "Ready" - tunnelReady.Message = "tunnel deployment ready and all ports mapped" + tunnelReady.Message = "relay connection and all configured frp forward registrations confirmed" case "tailscale": if tunnel.Tailscale == nil { @@ -1208,33 +1240,6 @@ func computeTunnelConditions( tunnelReady.Reason = "Ready" tunnelReady.Message = "tailnet online and all advertised TCP/UDP forwarding listeners active; client access remains subject to tailnet policy" - case "playit": - if tunnel.Playit == nil { - tunnelReady.Status = metav1.ConditionFalse - tunnelReady.Reason = "InvalidConfig" - tunnelReady.Message = "playit provider selected but no playit config" - conds = upsertCondition(conds, tunnelReady) - return conds - } - - // Check if credentials are provided (playit requires authentication). - if tunnel.CredentialsSecretRef == nil { - tunnelReady.Status = metav1.ConditionFalse - tunnelReady.Reason = "NoCredentials" - tunnelReady.Message = "no credentials secret provided; playit requires authentication" - conds = upsertCondition(conds, tunnelReady) - return conds - } - - // Deployment is ready; playit endpoint will arrive asynchronously. - // Report AwaitingAddress while the tunnel pod hasn't yet reported an address. - // This is optimistic: the condition flips based on whether the tunnel pod - // has written to status.tunnelEndpoint. The agent or operator reconcile must - // populate it separately. - tunnelReady.Status = metav1.ConditionTrue - tunnelReady.Reason = "Ready" - tunnelReady.Message = "tunnel deployment ready; waiting for playit endpoint assignment" - default: tunnelReady.Status = metav1.ConditionFalse tunnelReady.Reason = "UnknownProvider" @@ -1245,6 +1250,20 @@ func computeTunnelConditions( return conds } +func hasRelayReadinessProbe(dep *appsv1.Deployment) bool { + return hasTunnelReadinessProbe(dep, "/readyz") +} + +func hasTunnelReadinessProbe(dep *appsv1.Deployment, path string) bool { + for _, container := range dep.Spec.Template.Spec.Containers { + if container.Name == "tunnel" && container.ReadinessProbe != nil && container.ReadinessProbe.HTTPGet != nil { + probe := container.ReadinessProbe.HTTPGet + return probe.Path == path && probe.Port.IntValue() == 8091 + } + } + return false +} + // removeCondition removes a condition by type from the list, or returns the list unchanged. func removeCondition(conds []metav1.Condition, condType string) []metav1.Condition { out := make([]metav1.Condition, 0, len(conds)) diff --git a/operator/internal/controller/gameserver_tunnel.go b/operator/internal/controller/gameserver_tunnel.go index aa30681cf..b2a19ef2d 100644 --- a/operator/internal/controller/gameserver_tunnel.go +++ b/operator/internal/controller/gameserver_tunnel.go @@ -229,10 +229,10 @@ func (r *GameServerReconciler) reconcileTunnel( refusedSecret = "" replicas := int32(1) dep.Spec.Replicas = &replicas - dep.Spec.Strategy = appsv1.DeploymentStrategy{Type: appsv1.RollingUpdateDeploymentStrategyType} - if tunnel.Provider == "tailscale" { - // Avoid two tailnet devices racing for the same hostname on rollout. - dep.Spec.Strategy.Type = appsv1.RecreateDeploymentStrategyType + // A provider identity owns one relay session. Recreate unregisters the + // old child before its replacement can claim the same names/ports. + if tunnel.Provider == "frp" || tunnel.Provider == "playit" || tunnel.Provider == "tailscale" { + dep.Spec.Strategy = appsv1.DeploymentStrategy{Type: appsv1.RecreateDeploymentStrategyType} } dep.Spec.Selector = &metav1.LabelSelector{ MatchLabels: map[string]string{ @@ -307,6 +307,7 @@ func (r *GameServerReconciler) reconcileTunnel( envVars = []corev1.EnvVar{ {Name: "GAMESERVER_NAME", Value: gs.Name}, {Name: "GAMESERVER_NAMESPACE", Value: gs.Namespace}, + {Name: "GAMESERVER_UID", Value: string(gs.UID)}, {Name: "TUNNEL_TYPE", Value: "frp"}, {Name: "FRP_SERVER_ADDR", Value: tunnel.Frp.ServerAddr}, {Name: "FRP_SERVER_PORT", Value: fmt.Sprintf("%d", serverPort)}, @@ -445,6 +446,16 @@ func (r *GameServerReconciler) reconcileTunnel( Capabilities: &corev1.Capabilities{Drop: []corev1.Capability{"ALL"}}, }, }} + if tunnel.Provider == "frp" || tunnel.Provider == "playit" { + container := &dep.Spec.Template.Spec.Containers[0] + container.StartupProbe = &corev1.Probe{ProbeHandler: corev1.ProbeHandler{HTTPGet: &corev1.HTTPGetAction{Path: "/healthz", Port: intstr.FromInt32(8091)}}, PeriodSeconds: 2, TimeoutSeconds: 1, FailureThreshold: 30} + container.ReadinessProbe = &corev1.Probe{ProbeHandler: corev1.ProbeHandler{HTTPGet: &corev1.HTTPGetAction{Path: "/readyz", Port: intstr.FromInt32(8091)}}, PeriodSeconds: 2, TimeoutSeconds: 1, FailureThreshold: 1} + if tunnel.Provider == "playit" { + // The pinned Playit IPC has no live connection signal. Pod + // readiness uses process health; TunnelReady remains Unknown. + container.ReadinessProbe.HTTPGet.Path = "/healthz" + } + } return controllerutil.SetControllerReference(gs, dep, r.Scheme) }) diff --git a/operator/specs.md b/operator/specs.md index 3085dac8c..cbbb08c88 100644 --- a/operator/specs.md +++ b/operator/specs.md @@ -20,6 +20,8 @@ finalizer until those references are gone. These explicit waits use Existing restore fence-write requeues and writer-drain polling remain unchanged, and every resumed restore pass still verifies the workload fence and live writers. +- **Relay health:** frp identities use GameServer UID without changing advertised port names. frp/Playit Deployments use one replica, Recreate, a wrapper startup probe and a provider readiness probe on TCP 8091. `TunnelReady` requires the current observed generation and updated/ready replicas with the new probe; legacy unprobed Pods cannot report success. Playit v1.0.10's cached IPC assignment cannot confirm a relay connection, so it reports unknown/not ready rather than optimistic success. Preserve `TunnelResources` defaults and override removal. See `docs/relay-health.md` for health bounds, migration and the provider limitation. + - **GameServer lifecycle:** reconcile GameServer CRs into StatefulSets, Services, PVCs, and pod-injected agent sidecars; manage pod affinity, RCON/console integration, file I/O, and graceful stop sequences. - **Module installation:** pull OCI module bundles from remote or local sources, verify signatures (cosign), and materialize GameTemplate CRs; refresh source indexes periodically. Translation files (`locales/.json`) are carried by every module source (OCI layers, directory, git, HTTP archive, upload, `gp-module package` tarball) and validated against the ids derived from the module (`i18n/modlocale`); valid entries are written to the `