Add opt-in cleanup policy for related-resource copies#181
Conversation
|
Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kcp-ci-bot
added
kind/feature
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kcp-ci-bot
added
needs-ok-to-test
|
Hi @wojciech12. Thanks for your PR. I'm waiting for a kcp-dev member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kcp-ci-bot
added
the
size/XXL
For `origin: service` related resources the agent never prunes a copy when its
origin object is deleted mid-life; the copy is orphaned permanently (no
per-object finalizer, no set-diff). Even on primary teardown, copies whose
origin already vanished are not reclaimed.
Add a `cleanupPolicy` field to RelatedResourceSpec with three values:
- Orphan (default): copies are never deleted (== legacy cleanup:false).
- OnPrimaryDeletion: copies deleted only on primary deletion (== legacy cleanup:true).
- MatchOrigin: a copy is pruned as soon as its origin object is gone, and all
copies are deleted on primary teardown (also reclaiming mid-life orphans).
The prune is List-and-diff driven by the primary reconcile, never a finalizer on
related objects (issues kcp-dev#172/kcp-dev#116). Copies are labelled with owning-primary and
identifier provenance so only agent-created copies are ever deleted, scoped to
the destination client so the origin object is never touched. The List is
cluster-wide (scoped by the provenance label selector) so copies mapped into a
different namespace via spec.object.namespace rewrites are pruned too. The
deprecated `cleanup` bool is still honoured via EffectiveCleanupPolicy(). CEL
rules require `watch` for MatchOrigin+origin:service and reject cleanup:true+Orphan.
Extends the opt-in cleanup work from kcp-dev#114 (PR kcp-dev#164).
Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
bb0462a to
62c8745
Compare
kcp-ci-bot
added
dco-signoff: yes
xmudrii
left a comment
xmudrii
left a comment
There was a problem hiding this comment.
Reviewed the cleanup-policy change end-to-end (API types, CRD, syncer_related.go, object_syncer.go, tests). The core design — List-and-diff pruning driven by the primary reconcile with provenance labels, instead of per-object finalizers — is a sound way to sidestep the finalizer problems from #172/#116, and the EffectiveCleanupPolicy() back-compat mapping plus the CEL guards read well.
One item I'd treat as blocking, two worth addressing, and a couple of minor notes — details inline:
- Blocker:
Identifieris now used verbatim as a label value but has no length/charset validation; identifiers the CRD accepts today produce labels the API server rejects, silently breaking sync. - Provenance labels can collide across PublishedResources whose primaries share name+namespace and reuse an identifier.
- Provenance labels aren't re-stamped on the client-side-apply update path (the default), so copies predating the feature are never pruned.
rememberRelatedObjectsnow runs on an empty resolved set → annotation churn on the primary.- The prune has no fake-client unit test, though the package tests deletion that way elsewhere.
| func relatedCopyLabels(primary ctrlruntimeclient.Object, identifier, agentName string) map[string]string { | ||
| set := map[string]string{ | ||
| relatedPrimaryNameHashLabel: crypto.Hash(primary.GetName()), | ||
| relatedIdentifierLabel: identifier, |
There was a problem hiding this comment.
Identifier is used verbatim as a label value here, and the comment above says it's "validated to be alphanumeric by the API" — but there's no such validation: the CRD schema for identifier is bare type: string (no maxLength/pattern) and the Go field carries no kubebuilder markers.
I ran this branch's relatedCopyLabels() output through the apiserver's own metav1validation.ValidateLabels:
- 63-char identifier → valid; 64-char → rejected (
must be no more than 63 bytes) connection/details,connection details,-creds→ rejected by the label-value regex
So an identifier the CRD accepts can make the copy's labels invalid, and the apply/create then fails — sync silently breaks for that related resource. (New failure surface specifically for origin: kcp; origin: service was already constrained via the related-resources.syncagent.kcp.io/<identifier>.N annotation key.) Suggest hashing the identifier like name/namespace, or adding +kubebuilder:validation:MaxLength=63 + an alphanumeric Pattern to Identifier — and fixing the comment.
| // namespaces are hashed because they can exceed the label value limit or contain invalid | ||
| // characters; the identifier is validated to be alphanumeric by the API, so it is used verbatim. | ||
| // The agent name (when set) is included so that each agent only ever prunes its own copies. | ||
| func relatedCopyLabels(primary ctrlruntimeclient.Object, identifier, agentName string) map[string]string { |
There was a problem hiding this comment.
The label set is {primary-name-hash, primary-namespace-hash, identifier, agent}, and the prune List is typed only by projected Kind — nothing encodes the primary's GVK or the owning PublishedResource. Two PublishedResources that project copies to the same Kind, reuse the same identifier, and have primaries sharing name+namespace produce byte-identical labels, so one primary's prune selector matches the other's copies (verified: identical hashes, selector matches). Consider adding the PublishedResource name and/or primary GVK to the provenance labels.
| // ensureDestMetadata stamps the syncer's additional destination labels/annotations onto the given | ||
| // object. It is additive (it never removes labels the object already carries) and is used to | ||
| // attach related-resource provenance to destination copies. | ||
| func (s *objectSyncer) ensureDestMetadata(obj *unstructured.Unstructured) { |
There was a problem hiding this comment.
ensureDestMetadata runs from applyServerSide, ensureDestinationObject (create), and adoptExistingDestinationObject, but not from syncObjectSpec — the client-side-merge update path, which is the default (--enable-server-side-apply defaults to false). New copies get stamped at birth, so this is fine going forward; but copies that already exist when a user enables MatchOrigin/OnPrimaryDeletion are updated in place and never receive the provenance labels, so they're never enumerated for pruning — exactly the mid-life-orphan reclaim this PR targets. Worth re-stamping in the update path or documenting the limitation.
| // Remember the related objects on the primary object for the end-user. This is rebuilt from the | ||
| // freshly-resolved set so stale entries for pruned objects do not linger. | ||
| if relRes.Origin == syncagentv1alpha1.RelatedResourceOriginService { | ||
| annRequeue, err := s.rememberRelatedObjects(ctx, log, remote, relRes.Identifier, resolvedObjects) |
There was a problem hiding this comment.
With the early len(resolvedObjects) == 0 return removed, rememberRelatedObjects now runs even when zero objects resolve (for origin: service). On an empty resolution it wipes all related-resources.syncagent.kcp.io/<identifier>.* annotations and patches the primary, re-adding them next cycle — extra writes/requeues. Minor, but it's a behavior change for all policies, not just the pruning ones.
| // (mid-life prune). It only ever operates on the destination client, so origin objects are never | ||
| // touched, and it only ever sees objects that carry our provenance labels, so hand-created objects | ||
| // are never in scope. | ||
| func (s *ResourceSyncer) pruneRelatedCopies(ctx context.Context, log *zap.SugaredLogger, dest syncSide, primary *unstructured.Unstructured, projectedGVK schema.GroupVersionKind, selector labels.Selector, keep sets.Set[string], deleteAll bool) (requeue bool, err error) { |
There was a problem hiding this comment.
The package already unit-tests deletion/cleanup by driving ResourceSyncer.Process() against fakectrlruntimeclient (e.g. the DeletionTimestamp cases in syncer_test.go), and the fake client supports label-selector List. The new prune (keep-set filtering, deleteAll teardown, already-deleting requeue, empty-set) is currently covered only by e2e plus the two pure-function unit tests. A fake-client unit test of pruneRelatedCopies would match the package convention and guard the destructive path directly.
3 participants
For
origin: servicerelated resources the agent never prunes a copy when its origin object is deleted mid-life; the copy is orphaned permanently (no per-object finalizer, no set-diff). Even on primary teardown, copies whose origin already vanished are not reclaimed.Add a
cleanupPolicyfield to RelatedResourceSpec with three values:The prune is List-and-diff driven by the primary reconcile, never a finalizer on related objects (issues #172/#116). Copies are labelled with owning-primary and identifier provenance so only agent-created copies are ever deleted, scoped to the destination client so the origin object is never touched. The List is cluster-wide (scoped by the provenance label selector) so copies mapped into a different namespace via spec.object.namespace rewrites are pruned too. The deprecated
cleanupbool is still honoured via EffectiveCleanupPolicy(). CEL rules requirewatchfor MatchOrigin+origin:service and reject cleanup:true+Orphan.Extends the opt-in cleanup work from #114 (PR #164).
Summary
What Type of PR Is This?
/kind feature
Related Issue(s)
Fixes #
Release Notes
Add a
cleanupPolicyfield to RelatedResourceSpec with three values: