Feature Request: Add `ExpectNoPrimary` flag to `PlannedReparentShard` and `EmergencyReparentShard`

[mhamza15]

Feature Description

PlannedReparentShard and EmergencyReparentShard both support an --expected-primary flag that acts as a CAS guard. The operation only proceeds if the current shard primary matches the specified alias. However, there's no way to express the inverse: "I expect this shard to have no primary."

This proposes adding a new expect_no_primary field to both PlannedReparentShardRequest and EmergencyReparentShardRequest in the proto. When set to true, the operation would fail with FAILED_PRECONDITION if the shard already has a primary.

Use Case(s)

When a shard has no primary, vtorc can race with an operator (Kubernetes or human) and promote a new primary. Without this flag, the operator's PRS will follow vtorc's ERS and do two quick reparents in succession. With the flag, the PRS can fail if it finds the shard already with a primary.

[Devanshusharma2005]

@mhamza15 can I work on this?

[mhamza15]

@Devanshusharma2005 Go ahead, thanks!

[Devanshusharma2005]

I’m reading through the PRS/ERS reparent wiring and had a quick question about the interaction between expected_primary and expect_no_primary.
From what I can tell, expected_primary acts as a CAS on “primary == X”, while expect_no_primary would act as a CAS on “no primary”. It seems like a caller could technically set both at the same time, which feels logically contradictory (“primary is X” and “no primary”).
Should these two be treated as mutually exclusive (e.g. fail fast with FAILED_PRECONDITION if both are set), or is the intent to let the existing checks run and rely on callers not combining them?

[mhamza15]

Fail fast seems preferable to me 👍

[timvaillancourt]

👋 @mhamza15 when you say "shard has no primary", do you mean the topo shard record has no PrimaryAlias, or the running tablets have no type PRIMARY?

The --expected-primary uses the topo PrimaryAlias field - that's easy to use, but will often be non-nil even when in reality, there is no PRIMARY: example unplanned PRIMARY failure that hasn't been actioned

If you go the route of checking the live tablet types, this can be expensive on the topo (finding other shard members) and RPC calls, and may return partial results from cells we can't reach. That could add a lot of overhead/dependencies to ERS that I would hope to avoid