pg_dump: --create-empty-files-for-excluded-data (directory format)#47
Open
NikolayS wants to merge 2 commits into
Open
pg_dump: --create-empty-files-for-excluded-data (directory format)#47NikolayS wants to merge 2 commits into
NikolayS wants to merge 2 commits into
Conversation
When used with --exclude-table-data in -Fd output, still emit TABLE DATA TOC entries and placeholder .dat files containing a COPY end marker for excluded tables. Restrict the option to directory format and COPY output; document it and add a TAP test covering validation and dump contents. Co-authored-by: Cursor <cursoragent@cursor.com>
Stock pg_restore handles placeholder COPY end-marker files correctly; the earlier restore failure was from mixed build artifacts after branch switching without make clean. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review/test of Kirk Wolak's patch (kirkw/postgres @
pg-dump-empty-excluded-data), cherry-picked onto our fork's master for a clean 2-commit diff. Authorship preserved.What it does
Adds
--create-empty-files-for-excluded-datatopg_dump. For directory-format dumps, tables whose data is excluded via--exclude-table-data[-and-children]still get aNNNN.datfile written containing only the COPY end marker (\.\n\n\n) instead of being omitted entirely.Motivation: enables a parallel-dump workflow for huge tables — dump the schema with placeholder data files, then have an external tool fill those
.datfiles (e.g. split COPY streams across workers) without any merge step.pg_restorethen loads them like any normal directory-format data file.Commits
Add --create-empty-files-for-excluded-data for directory-format pg_dump.Restore pg_restore coverage in pg_dump-only TAP test.Guard rails (all verified)
The option is rejected with a clear error unless the dump is:
--exclude-table-data/--exclude-table-data-and-children--inserts/--column-inserts/--rows-per-insert)Testing done (this review)
Built on macOS (meson, PG 19beta1). Manual end-to-end verification of every assertion in the new TAP test
t/012_pg_dump_empty_excluded_data.pl:TABLE DATA public skip_data; two.datfiles produced..datis exactly5c2e 0a0a 0a(\.\n\n\n) — byte-identical to howdumpTableData_copy()ends a genuinely empty table..datnon-empty; restore round-trips (keep=2 rows, skip=0 rows)..datwith real COPY data and re-runningpg_restoreloads all rows correctly.Note: the formal
meson testTAP harness could not run on this Mac because SIP stripsDYLD_LIBRARY_PATHwhen initdb is launched via/usr/bin/python3(a known PG-on-macOS quirk, unrelated to this patch). Assertions were instead reproduced manually against the freshly built binaries.Review notes / nits (non-blocking)
26;23/24are unused —23would have been the natural next number.dumpTableData_empty()is also wired into the INSERT branch ofdumpTableData(), but that path is unreachable since the option rejects all INSERT modes up front. Harmless/defensive.This is the pg_dump-only half of Kirk's work (the pg_restore split-file-loading side is a separate future patch).