Query Tool result export enhancements (JSON/XML, encoding, BOM, copy-with-headers)

docs/en_US/preferences.rst

@@ -589,6 +589,13 @@ Use the fields on the *CSV/TXT Output* panel to control the CSV/TXT output.
   quoted in the CSV/TXT output; select *Strings*, *All*, or *None*.
 * Use the *Replace null values with* option to replace null values with
   specified string in the output file. Default is set to 'NULL'.
+* Use the *Output file encoding* drop-down listbox to specify the character
+  encoding used when saving query results to a file. The default is utf-8; an
+  encoding that is not listed can also be typed in.
+* Use the *Add byte order mark (BOM)?* switch to add a byte order mark at the
+  start of the saved file when a UTF encoding is used. This helps applications
+  such as Microsoft Excel detect the encoding correctly. This applies to the
+  CSV/TXT output only.
 
 .. image:: images/preferences_sql_display.png
     :alt: Preferences sqleditor display options
@@ -721,6 +728,9 @@ preferences for copied data.
   character for copied data.
 * Use the *Result copy quoting* drop-down listbox to select which type of fields
   require quoting; select *All*, *None*, or *Strings*.
+* When the *Copy with headers?* switch is set to true, the column headers are
+  included by default when copying data from the results grid. This can still
+  be toggled per-copy from the results grid copy options menu.
 * When the *Striped rows?* switch is set to true, the result grid will display
   rows with alternating background colors.
 

docs/en_US/query_tool_toolbar.rst

@@ -212,10 +212,12 @@ Data Editing Options
    | *Save Data Changes*  | Click the *Save Data Changes* icon to save data changes (insert, update, or delete) in the Data   | F6             |
    |                      | Output Panel to the server.                                                                       |                |
    +----------------------+---------------------------------------------------------------------------------------------------+----------------+
-   | *Save results to*    | Click the Save results to file icon to save the result set of the current query as a delimited    | F8             |
-   | *file*               | text file (CSV, if the field separator is set to a comma). This button will only be enabled when  |                |
-   |                      | a query has been executed and there are results in the data grid. You can specify the CSV/TXT     |                |
-   |                      | settings in the Preference Dialogue under SQL Editor -> CSV/TXT output.                           |                |
+   | *Save results to*    | Click the Save results to file icon to save the result set of the current query. By               | F8             |
+   | *file*               | default it is saved as a delimited text file (CSV, if the field separator is set to a             |                |
+   |                      | comma). Use the adjacent drop-down list to instead save the results as JSON or XML.               |                |
+   |                      | This button is only enabled when a query has been executed and there are results in               |                |
+   |                      | the data grid. You can specify the CSV/TXT settings (including the output file encoding           |                |
+   |                      | and byte order mark) in the Preferences dialog under Query Tool -> CSV/TXT Output.                |                |
    +----------------------+---------------------------------------------------------------------------------------------------+----------------+
    | Graph Visualiser     | Use the Graph Visualiser button to generate graphs of the query results.                          |                |
    +----------------------+---------------------------------------------------------------------------------------------------+----------------+

docs/en_US/release_notes_9_16.rst

@@ -26,6 +26,10 @@ Bundled PostgreSQL Utilities
 New features
 ************
 
+  | `Issue #3205 <https://github.com/pgadmin-org/pgadmin4/issues/3205>`_ -  Allow saving Query Tool results as JSON and XML in addition to CSV, via a drop-down on the Save results to file button.
+  | `Issue #4128 <https://github.com/pgadmin-org/pgadmin4/issues/4128>`_ -  Add a preference to choose the character encoding used when saving Query Tool results to a file.
+  | `Issue #4129 <https://github.com/pgadmin-org/pgadmin4/issues/4129>`_ -  Add a "Copy with headers?" preference to control whether column headers are included by default when copying results grid data.
+  | `Issue #6695 <https://github.com/pgadmin-org/pgadmin4/issues/6695>`_ -  Add a preference to write a UTF byte order mark (BOM) when saving Query Tool results to a file, for better interoperability with applications such as Microsoft Excel.
   | `Issue #9626 <https://github.com/pgadmin-org/pgadmin4/issues/9626>`_ -  Add support for the TOAST tuple target storage parameter in the Materialized View dialog.
   | `Issue #9646 <https://github.com/pgadmin-org/pgadmin4/issues/9646>`_ -  Make the init container security context in the Helm chart configurable via containerSecurityContext, consistent with the main container.
   | `Issue #9699 <https://github.com/pgadmin-org/pgadmin4/issues/9699>`_ -  Add support for closing a tab with a middle-click on its title.

web/pgadmin/tools/sqleditor/__init__.py

@@ -8,6 +8,7 @@
 ##########################################################################
 
 """A blueprint module implementing the sqleditor frame."""
+import codecs
 import os
 import pickle
 import re
@@ -2166,20 +2167,82 @@ def start_query_download_tool(trans_id):
                 }
             )
 
+        # Output format: csv (default), json or xml.
+        data_format = (data.get('format') or 'csv').lower()
+        if data_format not in ('csv', 'json', 'xml'):
+            data_format = 'csv'
+
+        # Encoding and BOM apply to the CSV/text output only; the structured
+        # formats are always emitted as UTF-8.
+        if data_format == 'csv':
+            output_encoding = blueprint.csv_output_encoding.get() or 'utf-8'
+            add_bom = blueprint.csv_add_bom.get()
+        else:
+            output_encoding = 'utf-8'
+            add_bom = False
+        # Validate the (free-text, user-configurable) encoding up front so
+        # an invalid codec returns a clean 400 here, rather than raising a
+        # LookupError mid-stream after the 200 Response has been returned.
+        try:
+            codecs.lookup(output_encoding)
+        except LookupError:
+            return make_json_response(
+                status=400,
+                success=0,
+                errormsg=gettext(
+                    "Unknown output encoding '{0}'."
+                ).format(output_encoding)
+            )
+
+        normalized_encoding = output_encoding.lower().replace(
+            '-', '').replace('_', '')
+        is_utf = normalized_encoding.startswith('utf')
+        # The 'utf-16' and 'utf-32' codecs (without an explicit endianness
+        # suffix) emit their own BOM, so we must not hand-prepend one too;
+        # doing so would produce two BOMs and corrupt the output. The
+        # explicit-endian forms (utf-16-le/-be, utf-32-le/-be) and utf-8 do
+        # not self-emit a BOM, so for those we keep writing it ourselves.
+        codec_self_emits_bom = normalized_encoding in ('utf16', 'utf32')
+
+        str_gen = gen(conn_obj,
+                      trans_obj,
+                      quote=blueprint.csv_quoting.get(),
+                      quote_char=blueprint.csv_quote_char.get(),
+                      field_separator=blueprint.csv_field_separator.get(),
+                      replace_nulls_with=blueprint.replace_nulls_with.get(),
+                      data_format=data_format)
+
+        def encoded_gen(text_gen):
+            is_first_chunk = True
+            for chunk in text_gen:
+                if is_first_chunk:
+                    is_first_chunk = False
+                    # Only hand-prepend a BOM when the codec does not emit
+                    # one itself, otherwise we'd end up with two BOMs.
+                    if add_bom and is_utf and not codec_self_emits_bom:
+                        chunk = '\ufeff' + chunk
+                yield chunk.encode(output_encoding, errors='replace')
+
+        if data_format == 'json':
+            base_mimetype = 'application/json'
+        elif data_format == 'xml':
+            base_mimetype = 'application/xml'
+        elif blueprint.csv_field_separator.get() == ',':
+            base_mimetype = 'text/csv'
+        else:
+            base_mimetype = 'text/plain'
+
         r = Response(
-            gen(conn_obj,
-                trans_obj,
-                quote=blueprint.csv_quoting.get(),
-                quote_char=blueprint.csv_quote_char.get(),
-                field_separator=blueprint.csv_field_separator.get(),
-                replace_nulls_with=blueprint.replace_nulls_with.get()),
-            mimetype='text/csv' if
-            blueprint.csv_field_separator.get() == ','
-            else 'text/plain'
+            encoded_gen(str_gen),
+            mimetype='{0}; charset={1}'.format(base_mimetype, output_encoding)
         )
 
         import time
-        extn = 'csv' if blueprint.csv_field_separator.get() == ',' else 'txt'
+        if data_format == 'csv':
+            extn = 'csv' if blueprint.csv_field_separator.get() == ',' \
+                else 'txt'
+        else:
+            extn = data_format
         filename = data['filename'] if data.get('filename', '') != "" else \
             '{0}.{1}'.format(int(time.time()), extn)
 

web/pgadmin/tools/sqleditor/static/js/components/sections/ResultSet.jsx

@@ -476,16 +476,17 @@ export class ResultSetUtils {
       });
   }
 
-  async saveResultsToFile(fileName, onProgress) {
+  async saveResultsToFile(fileName, onProgress, dataFormat='csv') {
+    const mimeTypes = {csv: 'text/csv', json: 'application/json', xml: 'application/xml'};
     try {
       await DownloadUtils.downloadFileStream({
         url: url_for('sqleditor.query_tool_download', {
           'trans_id': this.transId,
         }),
         options: {
           method: 'POST',
-          body: JSON.stringify({filename: fileName, query_commited: this.hasQueryCommitted})
-        }}, fileName, 'text/csv', onProgress);
+          body: JSON.stringify({filename: fileName, query_commited: this.hasQueryCommitted, format: dataFormat})
+        }}, fileName, mimeTypes[dataFormat] ?? 'text/csv', onProgress);
       this.eventBus.fireEvent(QUERY_TOOL_EVENTS.TRIGGER_SAVE_RESULTS_END);
     } catch (error) {
       this.eventBus.fireEvent(QUERY_TOOL_EVENTS.TRIGGER_SAVE_RESULTS_END);
@@ -1052,16 +1053,17 @@ export function ResultSet() {
       setLoaderText(null);
     });
 
-    eventBus.registerListener(QUERY_TOOL_EVENTS.TRIGGER_SAVE_RESULTS, async ()=>{
-      let extension = queryToolCtx.preferences?.sqleditor?.csv_field_separator === ',' ? '.csv': '.txt';
+    eventBus.registerListener(QUERY_TOOL_EVENTS.TRIGGER_SAVE_RESULTS, async (dataFormat='csv')=>{
+      const csvExtension = queryToolCtx.preferences?.sqleditor?.csv_field_separator === ',' ? '.csv': '.txt';
+      let extension = {csv: csvExtension, json: '.json', xml: '.xml'}[dataFormat] ?? csvExtension;
       let fileName = 'data-' + new Date().getTime() + extension;
       if(!queryToolCtx.params.is_query_tool) {
         fileName = queryToolCtx.params.node_name + extension;
       }
       setLoaderText(gettext('Downloading results...'));
       await rsu.current.saveResultsToFile(fileName, (p)=>{
         setLoaderText(gettext('Downloading results(%s)...', p));
-      });
+      }, dataFormat);
       setLoaderText('');
     });
 

web/pgadmin/tools/sqleditor/static/js/components/sections/ResultSetToolbar.jsx

@@ -282,6 +282,7 @@ export function ResultSetToolbar({query, canEdit, totalRowCount, pagination, all
   /* Menu button refs */
   const copyMenuRef = React.useRef(null);
   const pasetMenuRef = React.useRef(null);
+  const downloadMenuRef = React.useRef(null);
 
   const queryToolPref = queryToolCtx.preferences.sqleditor;
 
@@ -309,8 +310,8 @@ export function ResultSetToolbar({query, canEdit, totalRowCount, pagination, all
   const addRow = useCallback(()=>{
     eventBus.fireEvent(QUERY_TOOL_EVENTS.TRIGGER_ADD_ROWS, [[]], {isNewRow: true});
   }, []);
-  const downloadResult = useCallback(()=>{
-    eventBus.fireEvent(QUERY_TOOL_EVENTS.TRIGGER_SAVE_RESULTS);
+  const downloadResult = useCallback((fmt='csv')=>{
+    eventBus.fireEvent(QUERY_TOOL_EVENTS.TRIGGER_SAVE_RESULTS, fmt);
   }, []);
   const showGraphVisualiser = useCallback(()=>{
     eventBus.fireEvent(QUERY_TOOL_EVENTS.TRIGGER_GRAPH_VISUALISER);
@@ -348,6 +349,14 @@ export function ResultSetToolbar({query, canEdit, totalRowCount, pagination, all
     setDisableButton('save-result', (totalRowCount||0) < 1);
   }, [totalRowCount]);
 
+  useEffect(()=>{
+    // Seed the "Copy with headers" toggle default from the user preference.
+    setCheckedMenuItems((prev)=>({
+      ...prev,
+      copy_with_headers: queryToolPref.copy_column_headers,
+    }));
+  }, [queryToolPref.copy_column_headers]);
+
   useEffect(()=>{
     eventBus.registerListener(QUERY_TOOL_EVENTS.TRIGGER_COPY_DATA, copyData);
     return ()=>eventBus.deregisterListener(QUERY_TOOL_EVENTS.TRIGGER_COPY_DATA, copyData);
@@ -432,7 +441,10 @@ export function ResultSetToolbar({query, canEdit, totalRowCount, pagination, all
           </PgButtonGroup>
           <PgButtonGroup size="small">
             <PgIconButton title={gettext('Save results to file')} icon={<GetAppRoundedIcon />}
-              onClick={downloadResult} shortcut={queryToolPref.download_results}
+              onClick={()=>downloadResult('csv')} shortcut={queryToolPref.download_results}
+              disabled={buttonsDisabled['save-result']} />
+            <PgIconButton title={gettext('Save results options')} icon={<KeyboardArrowDownIcon />} splitButton
+              name="menu-downloadoptions" ref={downloadMenuRef} onClick={openMenu}
               disabled={buttonsDisabled['save-result']} />
           </PgButtonGroup>
           <PgButtonGroup size="small">
@@ -490,6 +502,16 @@ export function ResultSetToolbar({query, canEdit, totalRowCount, pagination, all
       >
         <PgMenuItem hasCheck value="paste_with_serials" checked={checkedMenuItems['paste_with_serials']} onClick={checkMenuClick}>{gettext('Paste with SERIAL/IDENTITY values?')}</PgMenuItem>
       </PgMenu>
+      <PgMenu
+        anchorRef={downloadMenuRef}
+        open={menuOpenId=='menu-downloadoptions'}
+        onClose={handleMenuClose}
+        label={gettext('Save Results Options Menu')}
+      >
+        <PgMenuItem onClick={()=>downloadResult('csv')}>{gettext('Save as CSV/Text')}</PgMenuItem>
+        <PgMenuItem onClick={()=>downloadResult('json')}>{gettext('Save as JSON')}</PgMenuItem>
+        <PgMenuItem onClick={()=>downloadResult('xml')}>{gettext('Save as XML')}</PgMenuItem>
+      </PgMenu>
     </>
   );
 }