Skip to content

Commit 17e331e

Browse files
committed
Add replay queue spill-to-disk for large transactions
When the replay queue exceeds spock.exception_replay_queue_size (in MB), subsequent entries are written to a temporary BufFile on disk instead of accumulating in memory. This prevents OOM on the subscriber when replaying large transactions with exception handling enabled. Key design decisions: - In-memory queue entries (ApplyReplayContext) hold libpq-allocated data buffers. apply_replay_queue_reset walks the list to PQfreemem them before calling MemoryContextReset. - Spilled entries are relocated to TopMemoryContext so they survive the MemoryContextReset in handle_commit. The caller frees them explicitly after replication_handler returns. - When the replay queue is exhausted mid-transaction (exception before COMMIT was received), the loop seamlessly transitions to stream reading via continue instead of break, matching the pre-spill behavior. - Added from_pq field to ApplyReplayEntry to distinguish libpq-allocated buffers (PQfreemem) from palloc'd spill-read buffers (pfree). Includes regression tests covering all three exception_behaviour modes (transdiscard, discard, sub_disable) both without conflict and with a primary key conflict on the last record.
1 parent a116148 commit 17e331e

9 files changed

Lines changed: 951 additions & 47 deletions

File tree

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ REGRESS = preseed infofuncs init_fail init preseed_check basic conflict_secondar
5656
interfaces foreign_key copy sequence triggers parallel functions row_filter \
5757
row_filter_sampling att_list column_filter apply_delay \
5858
extended node_origin_cascade multiple_upstreams tuple_origin autoddl \
59-
sync_event sync_table generated_columns drop
59+
sync_event sync_table generated_columns spill_transaction drop
6060

6161
# The following test cases are disabled while developing.
6262
#

include/spock.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ extern bool spock_include_ddl_repset;
5050
extern bool allow_ddl_from_functions;
5151
extern int restart_delay_default;
5252
extern int restart_delay_on_exception;
53-
extern int spock_replay_queue_size; /* Deprecated - no longer used */
53+
extern int spock_replay_queue_size;
5454
extern int spock_feedback_frequency;
5555
extern bool check_all_uc_indexes;
5656
extern bool spock_enable_quiet_mode;

src/spock.c

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,7 @@ bool spock_include_ddl_repset = false;
142142
bool allow_ddl_from_functions = false;
143143
int restart_delay_default;
144144
int restart_delay_on_exception;
145-
int spock_replay_queue_size; /* Deprecated - no longer used */
145+
int spock_replay_queue_size;
146146
int spock_feedback_frequency;
147147
bool check_all_uc_indexes = false;
148148
bool spock_enable_quiet_mode = false;
@@ -1153,12 +1153,13 @@ _PG_init(void)
11531153
NULL);
11541154

11551155
DefineCustomIntVariable("spock.exception_replay_queue_size",
1156-
"DEPRECATED: apply-worker replay queue size (no longer used)",
1157-
"This setting is deprecated and has no effect. "
1158-
"The replay queue now dynamically allocates memory as needed.",
1156+
"Maximum in-memory size for the apply replay queue",
1157+
"When the replay queue exceeds this size, subsequent "
1158+
"entries are spilled to a temporary file on disk. "
1159+
"Set to 0 to disable spilling (unlimited memory).",
11591160
&spock_replay_queue_size,
11601161
4,
1161-
0,
1162+
-1,
11621163
MAX_KILOBYTES / 1024,
11631164
PGC_SIGHUP,
11641165
GUC_UNIT_MB,

0 commit comments

Comments
 (0)