fix(walrestore): serve pg_rewind without prefetching and flag machinery#1007
Merged
Conversation
dosubot
Bot
added
size:M
mnencia
force-pushed
the
dev/rewind-restore-mode
branch
from
July 17, 2026 08:43
549588f to
272051d
Compare
mnencia
approved these changes
Jul 17, 2026
mnencia
force-pushed
the
dev/rewind-restore-mode
branch
4 times, most recently
from
July 20, 2026 14:50
99394e6 to
5428a33
Compare
leonardoce
added a commit
to cloudnative-pg/cloudnative-pg
that referenced
this pull request
Jul 21, 2026
When a demoted primary rejoins the cluster after a failover, the instance manager runs `pg_rewind --restore-target-wal` so the WAL segments no longer in its `pg_wal` can be fetched from the archive. The `restore_command` handed to pg_rewind was the same `manager wal-restore` invocation used during normal recovery, WAL prefetching included. pg_rewind walks the timeline backwards, fetches every file it needs exactly once, and treats any restore failure as fatal: with `wal.maxParallel > 1` a prefetch miss is practically guaranteed, the miss records the end-of-wal-stream flag on disk, and the next invocation consumes the flag and fails without even contacting the object store, aborting the whole rewind. A flag left over from the pod's previous life aborts even the first invocation. To make things worse, a failed pg_rewind was only retried by restarting the whole join under the reconcile exponential backoff, which grows to roughly 1000 seconds, so a transient restore failure could stall the rejoin for hours. `manager wal-restore` now has a `--rewind` flag that disables prefetching and the end-of-wal-stream flag machinery. The instance manager writes a rewind-specific `restore_command` into `override.conf` right before running pg_rewind, and the demotion path rewrites the standard configuration immediately after, so the flag never leaks into normal recovery. pg_rewind itself is now retried in-process with a bounded backoff before the error surfaces, which is safe because pg_rewind collects every WAL segment it needs before writing anything into the target data directory: a failed run leaves the target unchanged. The same context is conveyed to WAL restore plugins through the new `mode` field of `WALRestoreRequest` (cloudnative-pg/cnpg-i#351), and plugin-barman-cloud honors it in cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field keep their current behavior, helped by the in-process retry advancing one segment per attempt. The cnpg-i dependency points to a pseudo-version of that pull request and will move to the next tagged release once it is available. Related to #10419, which shares the flag machinery but is a distinct failure mode and is not closed here. Closes #11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
cnpg-bot
pushed a commit
to cloudnative-pg/cloudnative-pg
that referenced
this pull request
Jul 21, 2026
When a demoted primary rejoins the cluster after a failover, the instance manager runs `pg_rewind --restore-target-wal` so the WAL segments no longer in its `pg_wal` can be fetched from the archive. The `restore_command` handed to pg_rewind was the same `manager wal-restore` invocation used during normal recovery, WAL prefetching included. pg_rewind walks the timeline backwards, fetches every file it needs exactly once, and treats any restore failure as fatal: with `wal.maxParallel > 1` a prefetch miss is practically guaranteed, the miss records the end-of-wal-stream flag on disk, and the next invocation consumes the flag and fails without even contacting the object store, aborting the whole rewind. A flag left over from the pod's previous life aborts even the first invocation. To make things worse, a failed pg_rewind was only retried by restarting the whole join under the reconcile exponential backoff, which grows to roughly 1000 seconds, so a transient restore failure could stall the rejoin for hours. `manager wal-restore` now has a `--rewind` flag that disables prefetching and the end-of-wal-stream flag machinery. The instance manager writes a rewind-specific `restore_command` into `override.conf` right before running pg_rewind, and the demotion path rewrites the standard configuration immediately after, so the flag never leaks into normal recovery. pg_rewind itself is now retried in-process with a bounded backoff before the error surfaces, which is safe because pg_rewind collects every WAL segment it needs before writing anything into the target data directory: a failed run leaves the target unchanged. The same context is conveyed to WAL restore plugins through the new `mode` field of `WALRestoreRequest` (cloudnative-pg/cnpg-i#351), and plugin-barman-cloud honors it in cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field keep their current behavior, helped by the in-process retry advancing one segment per attempt. The cnpg-i dependency points to a pseudo-version of that pull request and will move to the next tagged release once it is available. Related to #10419, which shares the flag machinery but is a distinct failure mode and is not closed here. Closes #11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit ad1d52c)
cnpg-bot
pushed a commit
to cloudnative-pg/cloudnative-pg
that referenced
this pull request
Jul 21, 2026
When a demoted primary rejoins the cluster after a failover, the instance manager runs `pg_rewind --restore-target-wal` so the WAL segments no longer in its `pg_wal` can be fetched from the archive. The `restore_command` handed to pg_rewind was the same `manager wal-restore` invocation used during normal recovery, WAL prefetching included. pg_rewind walks the timeline backwards, fetches every file it needs exactly once, and treats any restore failure as fatal: with `wal.maxParallel > 1` a prefetch miss is practically guaranteed, the miss records the end-of-wal-stream flag on disk, and the next invocation consumes the flag and fails without even contacting the object store, aborting the whole rewind. A flag left over from the pod's previous life aborts even the first invocation. To make things worse, a failed pg_rewind was only retried by restarting the whole join under the reconcile exponential backoff, which grows to roughly 1000 seconds, so a transient restore failure could stall the rejoin for hours. `manager wal-restore` now has a `--rewind` flag that disables prefetching and the end-of-wal-stream flag machinery. The instance manager writes a rewind-specific `restore_command` into `override.conf` right before running pg_rewind, and the demotion path rewrites the standard configuration immediately after, so the flag never leaks into normal recovery. pg_rewind itself is now retried in-process with a bounded backoff before the error surfaces, which is safe because pg_rewind collects every WAL segment it needs before writing anything into the target data directory: a failed run leaves the target unchanged. The same context is conveyed to WAL restore plugins through the new `mode` field of `WALRestoreRequest` (cloudnative-pg/cnpg-i#351), and plugin-barman-cloud honors it in cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field keep their current behavior, helped by the in-process retry advancing one segment per attempt. The cnpg-i dependency points to a pseudo-version of that pull request and will move to the next tagged release once it is available. Related to #10419, which shares the flag machinery but is a distinct failure mode and is not closed here. Closes #11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit ad1d52c)
leonardoce
added a commit
to cloudnative-pg/cloudnative-pg
that referenced
this pull request
Jul 21, 2026
When a demoted primary rejoins the cluster after a failover, the instance manager runs `pg_rewind --restore-target-wal` so the WAL segments no longer in its `pg_wal` can be fetched from the archive. The `restore_command` handed to pg_rewind was the same `manager wal-restore` invocation used during normal recovery, WAL prefetching included. pg_rewind walks the timeline backwards, fetches every file it needs exactly once, and treats any restore failure as fatal: with `wal.maxParallel > 1` a prefetch miss is practically guaranteed, the miss records the end-of-wal-stream flag on disk, and the next invocation consumes the flag and fails without even contacting the object store, aborting the whole rewind. A flag left over from the pod's previous life aborts even the first invocation. To make things worse, a failed pg_rewind was only retried by restarting the whole join under the reconcile exponential backoff, which grows to roughly 1000 seconds, so a transient restore failure could stall the rejoin for hours. `manager wal-restore` now has a `--rewind` flag that disables prefetching and the end-of-wal-stream flag machinery. The instance manager writes a rewind-specific `restore_command` into `override.conf` right before running pg_rewind, and the demotion path rewrites the standard configuration immediately after, so the flag never leaks into normal recovery. pg_rewind itself is now retried in-process with a bounded backoff before the error surfaces, which is safe because pg_rewind collects every WAL segment it needs before writing anything into the target data directory: a failed run leaves the target unchanged. The same context is conveyed to WAL restore plugins through the new `mode` field of `WALRestoreRequest` (cloudnative-pg/cnpg-i#351), and plugin-barman-cloud honors it in cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field keep their current behavior, helped by the in-process retry advancing one segment per attempt. The cnpg-i dependency points to a pseudo-version of that pull request and will move to the next tagged release once it is available. Related to #10419, which shares the flag machinery but is a distinct failure mode and is not closed here. Closes #11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit ad1d52c) (cherry picked from commit f5f1b8a)
The operator now tells WAL restore plugins when a restore request is made on behalf of pg_rewind (cloudnative-pg/cnpg-i#351). pg_rewind walks the timeline backwards, fetches every WAL file it needs exactly once, and treats any restore failure as fatal, so both optimizations meant for an instance in recovery must stay off: prefetching the following segments is wasted work that ends in an archive miss, and the end-of-wal-stream flag recorded by such a miss makes a later invocation fail on a segment that is available in the archive, aborting the whole rewind. The cnpg-i dependency points to a pseudo-version of that pull request and will be moved to the next tagged release once it is available. Ref: cloudnative-pg/cloudnative-pg#11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The previous commit stopped pg_rewind restores from checking the end-of-wal-stream flag, but left an existing flag on disk untouched. A flag set by a normal-recovery invocation before this pod was demoted would then resurface once the rewind finished and wrongly abort the following normal-recovery invocation. Clear the flag unconditionally in rewind mode, before the spool short-circuit, so a request for a WAL file already staged in the spool cannot skip the clear. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
leonardoce
force-pushed
the
dev/rewind-restore-mode
branch
from
July 21, 2026 12:27
5428a33 to
f0c47a7
Compare
leonardoce
requested review from
NiccoloFei,
fcanovai,
gbartolini and
leonardoce
as code owners
July 21, 2026 12:27
leonardoce
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a demoted primary rejoins the cluster after a failover, the instance manager runs
pg_rewind --restore-target-wal, and the resultingrestore_commandinvocations are served by this sidecar exactly as if an instance in recovery were asking. pg_rewind walks the timeline backwards, fetches every WAL file it needs exactly once, and treats any restore failure as fatal, so two optimizations meant for recovery break it: prefetching the following segments is wasted work that, past the end of the timeline, is guaranteed to end in an archive miss, and the end-of-wal-stream flag recorded by that miss makes a later invocation fail without contacting the object store, on a segment the archive actually has, aborting the whole rewind.The restore request now carries the context it is made in (cloudnative-pg/cnpg-i#351). When it says
MODE_REWIND, the sidecar restores exactly the requested file: no prefetching, no end-of-wal-stream flag check, no flag recording. Requests from operators predating the field keep the current behavior.This is the plugin-side counterpart of cloudnative-pg/cloudnative-pg#11204. The cnpg-i dependency points to a pseudo-version of the protocol pull request and will move to the next tagged release once it is available.