Query Tool result export enhancements (JSON/XML, encoding, BOM, copy-with-headers)#10062
Open
dpage wants to merge 2 commits into
Open
Query Tool result export enhancements (JSON/XML, encoding, BOM, copy-with-headers)#10062dpage wants to merge 2 commits into
dpage wants to merge 2 commits into
Conversation
Several long-standing requests around exporting/copying Query Tool results, all in the results download/copy path: - Save results as JSON or XML in addition to CSV, selectable from a drop-down on the "Save results to file" toolbar button. The download generator is now format-aware and streams JSON/XML as well as CSV. - New "Output file encoding" preference (CSV/TXT output) controlling the character encoding of saved results; defaults to utf-8. - New "Add byte order mark (BOM)?" preference that prepends a UTF BOM to saved CSV/TXT files for better interoperability with applications such as Microsoft Excel. - New "Copy with headers?" preference seeding the default state of the results grid "Copy with headers" toggle. Adds integration tests for the JSON/XML/BOM/encoding download paths and updates the preferences and Query Tool toolbar documentation. Closes pgadmin-org#3205 Closes pgadmin-org#4128 Closes pgadmin-org#4129 Closes pgadmin-org#6695
WalkthroughQuery Tool result export now supports CSV, JSON, and XML; CSV/TXT export gains selectable output encodings and an optional UTF BOM; results-grid copy can default to include headers; frontend adds a download options menu; backend streams JSON/XML or CSV with encoding/BOM handling; tests and docs updated. ChangesQuery Tool Export Enhancements
Sequence DiagramsequenceDiagram
participant User
participant ResultSetToolbar
participant ResultSet
participant DownloadEndpoint
participant DatabaseDriver
User->>ResultSetToolbar: Click "Save as CSV/JSON/XML"
ResultSetToolbar->>ResultSet: TRIGGER_SAVE_RESULTS(dataFormat)
ResultSet->>ResultSet: saveResultsToFile(fileName, progress, dataFormat)
ResultSet->>DownloadEndpoint: POST download request (format)
DownloadEndpoint->>DatabaseDriver: execute_on_server_as_csv(data_format)
DatabaseDriver->>DatabaseDriver: _generate_json/_generate_xml or CSV writer
DatabaseDriver-->>DownloadEndpoint: Streamed chunks (text)
DownloadEndpoint->>DownloadEndpoint: Apply encoding/BOM rules, set mimetype & filename
DownloadEndpoint-->>User: Download response (with correct encoding/BOM and content-type)
Estimated Code Review Effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested Reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
@dpage You appear to be, as they say, on a roll. |
There was a problem hiding this comment.
Pull request overview
This PR enhances the pgAdmin Query Tool “save/copy results” path by adding JSON/XML exports, configurable output encoding + optional BOM for CSV/TXT exports, and a preference-seeded “copy with headers” default.
Changes:
- Add streaming JSON and XML export formats for Query Tool results (alongside existing CSV/TXT).
- Add Query Tool preferences for output file encoding, optional BOM, and default “copy with headers”.
- Extend integration tests and update documentation/release notes for the new export/copy behavior.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| web/pgadmin/utils/driver/psycopg3/connection.py | Add JSON/XML streaming generators and route export generation by format. |
| web/pgadmin/tools/sqleditor/utils/query_tool_preferences.py | Register new Query Tool preferences for encoding/BOM and copy-with-headers default. |
| web/pgadmin/tools/sqleditor/tests/test_download_csv_query_tool.py | Add integration scenarios covering JSON/XML export + encoding/BOM paths. |
| web/pgadmin/tools/sqleditor/static/js/components/sections/ResultSetToolbar.jsx | Add “Save results” split-button drop-down and seed copy-with-headers from preference. |
| web/pgadmin/tools/sqleditor/static/js/components/sections/ResultSet.jsx | Send requested export format to backend; map format to MIME type and file extension. |
| web/pgadmin/tools/sqleditor/init.py | Make download endpoint format-aware; apply encoding/BOM for CSV and UTF-8 for JSON/XML. |
| docs/en_US/release_notes_9_16.rst | Add release note entries for the new export/copy features. |
| docs/en_US/query_tool_toolbar.rst | Document the new export format drop-down and encoding/BOM settings. |
| docs/en_US/preferences.rst | Document new CSV/TXT Output and Results Grid preferences. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+2182
to
+2200
| is_utf = output_encoding.lower().replace('-', '').replace( | ||
| '_', '').startswith('utf') | ||
|
|
||
| str_gen = gen(conn_obj, | ||
| trans_obj, | ||
| quote=blueprint.csv_quoting.get(), | ||
| quote_char=blueprint.csv_quote_char.get(), | ||
| field_separator=blueprint.csv_field_separator.get(), | ||
| replace_nulls_with=blueprint.replace_nulls_with.get(), | ||
| data_format=data_format) | ||
|
|
||
| def encoded_gen(text_gen): | ||
| is_first_chunk = True | ||
| for chunk in text_gen: | ||
| if is_first_chunk: | ||
| is_first_chunk = False | ||
| if add_bom and is_utf: | ||
| chunk = '\ufeff' + chunk | ||
| yield chunk.encode(output_encoding, errors='replace') |
Two fixes from code review of the Query Tool result export feature: - Avoid emitting a double byte-order mark for the 'utf-16' and 'utf-32' output encodings. Those codecs (without an explicit endianness suffix) self-emit a BOM, so hand-prepending another produced two BOMs and a corrupt file. We now only hand-write the BOM for codecs that do not emit one themselves (utf-8 and the explicit-endian utf-16/32-le/-be forms), guaranteeing exactly one BOM for every utf-* encoding. - Validate the user-configurable output encoding up front with codecs.lookup() before building the streaming Response, returning a clean 400 instead of raising LookupError mid-stream (which produced a truncated 200 with a raw traceback). Adds test scenarios asserting utf-16 output carries exactly one BOM and that an invalid encoding returns a 400.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
web/pgadmin/tools/sqleditor/tests/test_download_csv_query_tool.py (1)
372-376: ⚡ Quick winHarden BOM dictionary for explicit-endian UTF variants.
The BOM dictionary only includes
utf8,utf16,utf32. If a future test scenario uses an explicit-endian encoding like'utf-16-le'withadd_bom=True, the normalized key'utf16le'will raiseKeyErrorat line 376. Current scenarios don't trigger this (only'utf-16','utf-8','latin-1'tested), but adding coverage for explicit-endian UTF encodings would fail.Consider using
.get()with a fallback or expanding the dictionary:🛡️ Recommended defensive refactor
- bom = { - 'utf8': codecs.BOM_UTF8, - 'utf16': codecs.BOM_UTF16, - 'utf32': codecs.BOM_UTF32, - }[normalized] + bom = { + 'utf8': codecs.BOM_UTF8, + 'utf16': codecs.BOM_UTF16, + 'utf16le': codecs.BOM_UTF16_LE, + 'utf16be': codecs.BOM_UTF16_BE, + 'utf32': codecs.BOM_UTF32, + 'utf32le': codecs.BOM_UTF32_LE, + 'utf32be': codecs.BOM_UTF32_BE, + }.get(normalized, codecs.BOM_UTF8)Alternatively, fail explicitly for unsupported encodings:
- }[normalized] + }.get(normalized) + if bom is None: + self.fail(f"BOM constant not defined for encoding '{self.encoding}'")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/pgadmin/tools/sqleditor/tests/test_download_csv_query_tool.py` around lines 372 - 376, The BOM lookup using the dict keyed by normalized (the variable normalized) can raise KeyError for explicit-endian encodings (e.g., 'utf16le'); update the logic around the bom assignment in test_download_csv_query_tool.py so it uses a defensive lookup: either extend the mapping to include keys like 'utf16le','utf16be','utf32le','utf32be' mapping to the appropriate codecs.BOM_* or use dict.get(normalized) with a clear fallback/explicit error message; ensure the symbol names involved are the local variables normalized and bom so tests will either receive the correct BOM for explicit-endian encodings or fail with a descriptive error instead of a KeyError.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@web/pgadmin/tools/sqleditor/tests/test_download_csv_query_tool.py`:
- Around line 372-376: The BOM lookup using the dict keyed by normalized (the
variable normalized) can raise KeyError for explicit-endian encodings (e.g.,
'utf16le'); update the logic around the bom assignment in
test_download_csv_query_tool.py so it uses a defensive lookup: either extend the
mapping to include keys like 'utf16le','utf16be','utf32le','utf32be' mapping to
the appropriate codecs.BOM_* or use dict.get(normalized) with a clear
fallback/explicit error message; ensure the symbol names involved are the local
variables normalized and bom so tests will either receive the correct BOM for
explicit-endian encodings or fail with a descriptive error instead of a
KeyError.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 04e4f8a9-a59d-433e-8cb8-d86ed5c523d3
📒 Files selected for processing (2)
web/pgadmin/tools/sqleditor/__init__.pyweb/pgadmin/tools/sqleditor/tests/test_download_csv_query_tool.py
🚧 Files skipped from review as they are similar to previous changes (1)
- web/pgadmin/tools/sqleditor/init.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A batch of long-standing Query Tool result export/copy enhancements, all in
the results download/copy path.
drop-down on the Save results to file toolbar button. The download
generator is now format-aware and streams JSON/XML as well as CSV. JSON/XML
are always emitted as UTF-8; XML emits column names as escaped
nameattributes so column names that are not valid XML element names are handled
safely.
the character encoding used when saving results; defaults to utf-8, with a
free-text option for encodings that are not listed.
CSV/TXT files for better interoperability with applications such as Microsoft
Excel. (Applies to CSV/TXT output only.)
grid "Copy with headers" toggle (still toggleable per-copy).
Testing
download paths through the real
/query_tool/download/endpoint; theexisting CSV scenarios continue to pass, confirming the generator refactor
is non-regressive.
pycodestyleandeslintclean.entries.
Closes #3205
Closes #4128
Closes #4129
Closes #6695
Summary by CodeRabbit
New Features
Documentation
Tests