Skip to content

codeflare-operator-manager Keeps going into CrashLoopBackOff #698

Open
Open
codeflare-operator-manager Keeps going into CrashLoopBackOff#698
@donovat

Description

@donovat
donovat
opened on Sep 2, 2025

OpenShift v4.18
Installed via Operator: CodeFlare Operator V1.16.0

codeflare-operator-manager starts then crashes with CrashLoopBackOff - which then causes the Operator to re-install, which then re-starts the codeflare operator, and the loop starts again.

The logs from the Pod..

2025-09-02T11:09:54Z	INFO	setup	Build info	{"operatorVersion": "", "appwrapperVersion": "UNKNOWN", "date": "2025-06-16 10:15"}
2025-09-02T11:09:54Z	INFO	setup	Successfully configured operator	{"config": {"clientConnection":{"qps":50,"burst":100},"metrics":{"bindAddress":":8080"},"health":{"bindAddress":":8081","readinessEndpointName":"readyz","livenessEndpointName":"healthz"},"leaderElection":{"leaderElect":null,"leaseDuration":"0s","renewDeadline":"0s","retryPeriod":"0s","resourceLock":"","resourceName":"","resourceNamespace":""},"kuberay":{"rayDashboardOAuthEnabled":true,"ingressDomain":"","mTLSEnabled":true},"appwrapper":{"enabled":false,"Config":{"autopilot":{"injectAntiAffinities":true,"monitorNodes":true,"resourceTaints":{"nvidia.com/gpu":[{"key":"autopilot.ibm.com/gpuhealth","value":"WARN","effect":"PreferNoSchedule"},{"key":"autopilot.ibm.com/gpuhealth","value":"TESTING","effect":"NoSchedule"},{"key":"autopilot.ibm.com/gpuhealth","value":"EVICT","effect":"NoExecute"}]},"preferNoScheduleWeight":50},"userRBACAdmissionCheck":true,"faultTolerance":{"admissionGracePeriod":60000000000,"warmupGracePeriod":300000000000,"failureGracePeriod":60000000000,"resetPause":90000000000,"retryLimit":3,"deletionGracePeriod":600000000000,"gracePeriodCeiling":86400000000000,"successTTLCeiling":604800000000000}}}}}
2025-09-02T11:09:54Z	INFO	We detected being on OpenShift!
2025-09-02T11:09:54Z	INFO	setup	setting up health endpoints
2025-09-02T11:09:54Z	INFO	setup	setting up RayCluster controller
2025-09-02T11:09:54Z	INFO	We detected being on OpenShift!
2025-09-02T11:09:54Z	INFO	setup	setting up AppWrapper components
2025-09-02T11:09:54Z	INFO	setup	Embedded AppWrapper controller is disabled by config
2025-09-02T11:09:54Z	INFO	setup	starting manager
2025-09-02T11:09:54Z	INFO	controller-runtime.metrics	Starting metrics server
2025-09-02T11:09:54Z	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": ":8080", "secure": false}
2025-09-02T11:09:54Z	INFO	starting server	{"name": "health probe", "addr": "[::]:8081"}
2025-09-02T11:09:54Z	INFO	Starting EventSource	{"controller": "cert-rotator", "source": "kind source: *v1.Secret"}
2025-09-02T11:09:54Z	INFO	Starting EventSource	{"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2025-09-02T11:09:54Z	INFO	Starting EventSource	{"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2025-09-02T11:09:54Z	INFO	Starting Controller	{"controller": "cert-rotator"}
2025-09-02T11:09:54Z	INFO	cert-rotation	starting cert rotator controller
2025-09-02T11:09:54Z	INFO	cert-rotation	no cert refresh needed
2025-09-02T11:09:54Z	INFO	cert-rotation	certs are ready in /tmp/k8s-webhook-server/serving-certs
2025-09-02T11:09:54Z	INFO	Starting workers	{"controller": "cert-rotator", "worker count": 1}
2025-09-02T11:09:54Z	INFO	cert-rotation	no cert refresh needed
2025-09-02T11:09:54Z	ERROR	cert-rotation	Webhook not found. Unable to update certificate.	{"name": "codeflare-operator-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "error": "ValidatingWebhookConfiguration.admissionregistration.k8s.io \"codeflare-operator-validating-webhook-configuration\" not found"}
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
	/root/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:822
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
	/root/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:791
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224
2025-09-02T11:09:54Z	ERROR	cert-rotation	Webhook not found. Unable to update certificate.	{"name": "codeflare-operator-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration", "error": "MutatingWebhookConfiguration.admissionregistration.k8s.io \"codeflare-operator-mutating-webhook-configuration\" not found"}
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
	/root/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:822
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
	/root/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.12.0/pkg/rotator/rotator.go:791
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224
2025-09-02T11:09:55Z	INFO	setup	Waiting for certificate generation to complete
2025-09-02T11:09:55Z	INFO	cert-rotation	CA certs are injected to webhooks
2025-09-02T11:09:55Z	INFO	setup	Certs ready
2025-09-02T11:09:55Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "ray.io/v1, Kind=RayCluster", "path": "/mutate-ray-io-v1-raycluster"}
2025-09-02T11:09:55Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-ray-io-v1-raycluster"}
2025-09-02T11:09:55Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "ray.io/v1, Kind=RayCluster", "path": "/validate-ray-io-v1-raycluster"}
2025-09-02T11:09:55Z	INFO	controller-runtime.webhook	Starting webhook server
2025-09-02T11:09:55Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-ray-io-v1-raycluster"}
2025-09-02T11:09:55Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2025-09-02T11:09:55Z	INFO	controller-runtime.webhook	Serving webhook server	{"host": "", "port": 9443}
2025-09-02T11:09:55Z	INFO	Starting EventSource	{"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.RayCluster"}
2025-09-02T11:09:55Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
2025-09-02T11:09:55Z	INFO	Starting EventSource	{"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.ServiceAccount"}
2025-09-02T11:09:55Z	INFO	Starting EventSource	{"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.Service"}
2025-09-02T11:09:55Z	INFO	Starting EventSource	{"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.Secret"}
2025-09-02T11:09:55Z	INFO	Starting EventSource	{"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.Ingress"}
2025-09-02T11:09:55Z	INFO	Starting EventSource	{"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.NetworkPolicy"}
2025-09-02T11:09:55Z	INFO	Starting EventSource	{"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.Route"}
2025-09-02T11:09:55Z	INFO	Starting EventSource	{"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "source": "kind source: *v1.ClusterRoleBinding"}
2025-09-02T11:09:55Z	INFO	Starting Controller	{"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster"}
2025-09-02T11:09:55Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:53
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:54
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:05Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:15Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:25Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:35Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:45Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:10:55Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:05Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:15Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:25Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:35Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:45Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:55Z	ERROR	Could not wait for Cache to sync	{"controller": "codeflare-raycluster-controller", "controllerGroup": "ray.io", "controllerKind": "RayCluster", "error": "failed to wait for codeflare-raycluster-controller caches to sync: timed out waiting for cache to be synced for Kind *v1.RayCluster"}
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:200
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:231
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/manager/runnable_group.go:226
2025-09-02T11:11:55Z	INFO	Stopping and waiting for non leader election runnables
2025-09-02T11:11:55Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "cert-rotator"}
2025-09-02T11:11:55Z	INFO	cert-rotation	stopping cert rotator controller
2025-09-02T11:11:55Z	INFO	All workers finished	{"controller": "cert-rotator"}
2025-09-02T11:11:55Z	INFO	Stopping and waiting for leader election runnables
2025-09-02T11:11:55Z	INFO	Stopping and waiting for caches
2025-09-02T11:11:55Z	INFO	pkg/mod/k8s.io/client-go@v0.31.4/tools/cache/reflector.go:243: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2025-09-02T11:11:55Z	INFO	pkg/mod/k8s.io/client-go@v0.31.4/tools/cache/reflector.go:243: watch of *v1.Route ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2025-09-02T11:11:55Z	INFO	pkg/mod/k8s.io/client-go@v0.31.4/tools/cache/reflector.go:243: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2025-09-02T11:11:55Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "RayCluster.ray.io", "error": "no matches for kind \"RayCluster\" in version \"ray.io/v1\""}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:71
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/root/go/pkg/mod/k8s.io/apimachinery@v0.31.4/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/kind.go:64
2025-09-02T11:11:55Z	INFO	Stopping and waiting for webhooks
2025-09-02T11:11:55Z	INFO	controller-runtime.webhook	Shutting down webhook server with timeout of 1 minute
2025-09-02T11:11:55Z	INFO	Stopping and waiting for HTTP servers
2025-09-02T11:11:55Z	INFO	shutting down server	{"name": "health probe", "addr": "[::]:8081"}
2025-09-02T11:11:55Z	INFO	controller-runtime.metrics	Shutting down metrics server with timeout of 1 minute
2025-09-02T11:11:55Z	INFO	Wait completed, proceeding to shutdown the manager
2025-09-02T11:11:55Z	ERROR	setup	error running manager	{"error": "failed to wait for codeflare-raycluster-controller caches to sync: timed out waiting for cache to be synced for Kind *v1.RayCluster"}
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/log/deleg.go:142
github.com/go-logr/logr.Logger.Error
	/root/go/pkg/mod/github.com/go-logr/logr@v1.4.2/logr.go:301
main.exitOnError
	/workspace/main.go:398
main.main
	/workspace/main.go:237
runtime.main
	/usr/local/go/src/runtime/proc.go:272

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions