Skip to content

2.1: node deparse, walker control, tree rewriting and comparison, fragment parsing - #24

Merged
mysticmind merged 3 commits into
mainfrom
core-2.1-prototype
Oct 9, 2026
Merged

mysticmind merged 3 commits into
mainfrom
core-2.1-prototype

Conversation

@mysticmind

@mysticmind mysticmind commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

The core additions for 2.1. All additive; nothing from 2.0 changes behaviour. These are also the pieces a schema diff library on top of this one needs.

1. Node-level deparse

Parser.DeparseNode(node), also available as node.Deparse(), turns one node back into SQL.

select.WhereClause.Deparse()      // c.a = $1 AND c.b IN (SELECT id FROM vip)
select.FromClause[0].Deparse()    // customers c JOIN orders o ON o.cid = c.id
table.Deparse()                   // customers c
  • Supported: statements, expressions, FROM items (RangeVar, JoinExpr, subselects, functions), ResTarget, SortBy, WithClause, CommonTableExpr, TypeName, plus Node wrappers, RawStmt and a whole ParseResult.
  • Not supported: nodes that are not SQL on their own (Alias, String, Integer, List). They return an error such as a Alias cannot be deparsed on its own.
  • How: libpg_query can only deparse whole statements, so the node is placed in a minimal SELECT (in the WHERE, FROM, target list, ORDER BY or WITH position), deparsed, and the surrounding SELECT is cut off. The original tree is not modified.
  • This removes the earlier limit where only statements could return their text.

2. Walker control

  • Walk(Func<NodeVisit, WalkAction>): the visitor returns Continue, SkipChildren or Stop. The existing enumerable Walk() could not skip a subtree.

  • NodeVisit.FieldName and NodeVisit.Index: which property of the parent a node came from (for example nameof(SelectStmt.WhereClause)), and its position when that property is a list. Added as init-only properties, so the positional NodeVisit(Node, Parent, Depth) shape is unchanged.

  • NodeVisit.Ancestors and FindAncestor<T>(): the containing nodes, nearest first, and the nearest one of a given type. This answers questions such as "is this table reference inside a CTE?" or "inside a subquery?".

  • NodeVisit.Statement and StatementIndex: the top-level statement a node belongs to, when the walk starts from a ParseResult.

  • NodeVisit equality is now defined explicitly: same node (by reference), same place.

3. Parameter references

Parser.ParameterRefs(query) returns each $n with its number and its start and end in the query string (UTF-16 offsets). It is built on Scan, so references inside string literals and comments are left out. Async variant included.

When a parameter is cast ($1::int, CAST($2 AS numeric(10,2))), ParameterRef.TypeName holds the type. That part needs the parse tree, so a query that scans but does not parse still returns its parameters, only without types.

4. Changing and comparing trees

  • Rewrite(visitor): the visitor returns NodeEdit.Keep, NodeEdit.ReplaceWith(node) or NodeEdit.Remove for each node. The tree is changed in place, children before parents, and visits carry the same context as Walk (ancestors, field, index, statement). A replacement that does not fit the property throws with a message naming the property and both types.
  • EqualsIgnoringLocations: structural comparison of two trees or nodes that skips every location field. SQL that differs only in whitespace, comments, keyword case or redundant parentheses compares equal. It is iterative, so it is safe on deep trees.
  • AsNode(): wraps a node for a property or list that takes any node.

Together these make "normalize, then compare" possible, for example dropping the casts PostgreSQL adds when it stores an index expression and then comparing with what was written.

5. Parsing fragments and quoting names

  • Parser.ParseExpression: parses exactly one expression, such as a column default, a CHECK condition or pg_get_expr output. Lists, aliases, extra clauses and extra statements are rejected.
  • Parser.ParseTypeName: parses one type name, such as numeric(10,2) or text[].
  • PgIdentifier.Quote / NeedsQuoting: PostgreSQL's quote_ident rules, using the scanner's keyword categories, so no keyword list is maintained here.

Compared with other bindings

I read the sources of Ruby pg_query, pglast and @pgsql/traverse for these three features.

  • Node deparse: Ruby has deparse_stmt and deparse_expr (the same wrap-in-a-SELECT approach), covering statements and expressions only. This covers more node kinds.
  • Walker: now has skip, stop, parent, field, index, the ancestor chain and statement context. Replacing and removing nodes are now covered by Rewrite. Still missing compared with them: handlers per node type, and inserting sibling nodes.
  • Parameters: Ruby reports byte offsets and the cast type; this reports string offsets, the number and the cast type.

Verification

  • 85 new test cases across NodeToolsTests and TreeEditingTests; 245 pass on .NET 8, 9 and 10 locally.
  • The package smoke test exercises every addition, and a local Native AOT publish ran them with no trim or AOT warnings.
  • README sections added, including "What's new in 2.1". Example output was run, not written by hand.

Limits

  • Rewrite is recursive. On a very deep tree it throws InsufficientExecutionStackException instead of overflowing. Walk and EqualsIgnoringLocations are iterative and have no such limit.
  • Node-level deparse takes no DeparseOptions (no pretty printing), because cutting off the wrapper relies on the compact output.
  • ParseExpression and ParseTypeName locations are relative to an internal wrapper, not to the text passed in.
  • ParameterRefs types need a parse; a query that scans but does not parse returns its parameters without types.

Version

Bumped to 2.1.0.

- Parser.DeparseNode and node.Deparse() turn a single node back into
  SQL: statements, expressions, FROM items, select list and ORDER BY
  items, WITH clauses and type names
- Walk(visitor) lets the visitor skip a node's children or stop the
  walk; NodeVisit now reports the parent's property and list index the
  node came from
- Parser.ParameterRefs finds $n parameter references with their
  offsets in the query string
…otype

Closes the gaps against Ruby pg_query, pglast and @pgsql/traverse:

- NodeVisit.Ancestors and FindAncestor<T>() give the path from a node
  up to the root of the walk
- NodeVisit.Statement and StatementIndex give the top-level statement a
  node belongs to
- ParameterRef.TypeName holds the type a parameter is cast to
…ng; bump to 2.1.0

- Rewrite keeps, replaces or removes nodes in place, children first
- EqualsIgnoringLocations compares trees and nodes by structure
- AsNode wraps a node for a property or list that takes any node
- Parser.ParseExpression and Parser.ParseTypeName parse a single
  expression or type name on its own
- PgIdentifier.Quote and NeedsQuoting follow PostgreSQL's quote_ident,
  using the scanner's keyword categories
- Version 2.1.0, with a What's new in 2.1 section in the README
@mysticmind mysticmind changed the title Prototype for 2.1: node-level deparse, walker control, parameter references 2.1: node deparse, walker control, tree rewriting and comparison, fragment parsing Oct 9, 2026
@mysticmind
mysticmind marked this pull request as ready for review October 9, 2026 13:23
@mysticmind
mysticmind merged commit 92b7b12 into main Oct 9, 2026
25 checks passed
@mysticmind
mysticmind deleted the core-2.1-prototype branch October 9, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant