Skip to content

Raise default REST client QPS/burst when unset#2414

Open
AryanP123 wants to merge 1 commit intoskupperproject:mainfrom
AryanP123:controller-kube-client-qps
Open

Raise default REST client QPS/burst when unset#2414
AryanP123 wants to merge 1 commit intoskupperproject:mainfrom
AryanP123:controller-kube-client-qps

Conversation

@AryanP123
Copy link
Copy Markdown
Contributor

Fixes #2412

@AryanP123
Copy link
Copy Markdown
Contributor Author

Context:

With many Listeners/Connectors, readiness lagged because rest.Config had QPS/Burst = 0, so client-go used implicit ~5 QPS / 10 burst and the controller spent a long time client-throttled.

Set QPS=100, Burst=200 when both are still unset and RateLimiter is nil

Not dynamic. Values apply when clients are built at process start. change env --> restart controller.

Testing e2e:
service-scale with high service_count (e.g. 300). compare time / retries on Wait for all Connectors/Listeners Ready vs baseline.

@AryanP123 AryanP123 requested a review from ajssmith March 26, 2026 20:28
Comment thread internal/kube/client/client.go Outdated
if cfg.QPS > 0 && cfg.Burst <= 0 {
b := int(cfg.QPS * 2)
if b < 100 {
b = 100
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if 100 is a good number if, for example, QPS==1, maybe just use what you have on line 49?

Comment thread internal/kube/client/client.go Outdated
}
if cfg.Burst > 0 && cfg.QPS <= 0 {
cfg.QPS = float32(cfg.Burst) / 4
if cfg.QPS < 25 {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing here. If Burst is set to 4, is it appropriate to set this to 25?

Copy link
Copy Markdown
Member

@fgiorgetti fgiorgetti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall results are really good with the suggested defaults.
Total e2e test time dropped from 16m to 6m locally.

return
}
if cfg.QPS > 0 && cfg.Burst <= 0 {
b := int(cfg.QPS * 2)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the reasoning behind this operation and the other one in line 56?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For when only QPS is set, client-go still needs a burst for the token bucket, so we set it to twice the qps. And then when only burst is set, the same idea but in reverse.

@AryanP123 AryanP123 force-pushed the controller-kube-client-qps branch from 6fbc86e to 45757dd Compare April 13, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slow Listener / Connector readiness at scale (many CRs stay non‑Ready long after creation)

3 participants