Commit 17e331e
committed
Add replay queue spill-to-disk for large transactions
When the replay queue exceeds spock.exception_replay_queue_size (in MB),
subsequent entries are written to a temporary BufFile on disk instead of
accumulating in memory. This prevents OOM on the subscriber when
replaying large transactions with exception handling enabled.
Key design decisions:
- In-memory queue entries (ApplyReplayContext) hold libpq-allocated data
buffers. apply_replay_queue_reset walks the list to PQfreemem them
before calling MemoryContextReset.
- Spilled entries are relocated to TopMemoryContext so they survive the
MemoryContextReset in handle_commit. The caller frees them explicitly
after replication_handler returns.
- When the replay queue is exhausted mid-transaction (exception before
COMMIT was received), the loop seamlessly transitions to stream reading
via continue instead of break, matching the pre-spill behavior.
- Added from_pq field to ApplyReplayEntry to distinguish libpq-allocated
buffers (PQfreemem) from palloc'd spill-read buffers (pfree).
Includes regression tests covering all three exception_behaviour modes
(transdiscard, discard, sub_disable) both without conflict and with a
primary key conflict on the last record.1 parent a116148 commit 17e331e
9 files changed
Lines changed: 951 additions & 47 deletions
File tree
- include
- src
- tests/regress
- expected
- sql
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
59 | | - | |
| 59 | + | |
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | | - | |
| 53 | + | |
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
145 | | - | |
| 145 | + | |
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
| |||
1153 | 1153 | | |
1154 | 1154 | | |
1155 | 1155 | | |
1156 | | - | |
1157 | | - | |
1158 | | - | |
| 1156 | + | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
1159 | 1160 | | |
1160 | 1161 | | |
1161 | | - | |
| 1162 | + | |
1162 | 1163 | | |
1163 | 1164 | | |
1164 | 1165 | | |
| |||
0 commit comments