FEATURE: Add an AST normalization pass for target-format nesting rules

UPGRADING.md

@@ -1,5 +1,57 @@
 # Upgrading Markbridge
 
+## Unreleased — AST normalization runs by default
+
+A new `Markbridge::Normalizer` pass runs between the parse-time `yield` hook
+and rendering. It rewrites the AST so the renderer only gets markup the target
+format can express. It is **on by default** for every `*_to_markdown` call,
+`convert`, and `render`.
+
+What changes in the output, with no code change on your side:
+
+- A link inside a link (`[url][url]…[/url][/url]`) collapses to a single link.
+  CommonMark does not allow nested links.
+- A block element inside an inline container — a quote, list, table, or a
+  `Poll`/`Event` node inside a link, bold, or a heading — is moved out, so the
+  inline element does not break. This is not link-specific.
+- A fenced or multi-line code block inside an inline container is moved out; a
+  one-line code span stays.
+- A formatting wrapper left empty by the above is removed (no empty `**` `**`).
+  An empty link is kept, because it renders as a plain URL.
+
+Each change is reported under `conversion.diagnostics[:normalization]`, next
+to `unknown_tags`.
+
+The default rules are legality only. Discourse policy is not built in. A
+linked image (`[![alt](src)](url)`) is valid CommonMark, so the default leaves
+it alone. To move image-likes out of links, add the rules yourself. This is
+the pattern that replaces hoisting logic a consumer used to have in a custom
+`Url` tag:
+
+```ruby
+NORMALIZER =
+  Markbridge::Normalizer
+    .default
+    .rule(parent: Markbridge::AST::Url, child: Markbridge::AST::Image, strategy: :hoist_after)
+    .rule(parent: Markbridge::AST::Url, child: Markbridge::AST::Upload, strategy: :hoist_after)
+    .rule(parent: Markbridge::AST::Url, child: Markbridge::AST::Attachment, strategy: :hoist_after)
+    .freeze
+
+Markbridge.convert(input, format: :bbcode, normalize: NORMALIZER)
+```
+
+Build the normalizer once (a constant) and reuse it. `#normalize` keeps no
+state on the instance, so one frozen instance is safe for every conversion,
+also across threads, and passing it is as fast as the default path.
+
+To skip normalization:
+
+```ruby
+Markbridge.convert(input, format: :bbcode, normalize: false)
+```
+
+See [docs/normalization.md](docs/normalization.md).
+
 ## 0.3.0 — quote attribution fields and URL rendering
 
 ### `AST::Quote` attribution fields renamed and typed

bench/corpus_bench.rb

@@ -43,9 +43,22 @@ def report(name, corpus, &work)
        )
 end
 
+# A violation-heavy variant of a corpus: prepends a few degenerate
+# constructs (linked image, nested links, block-in-link) to each post so
+# the normalizer's worst case — including the destination-stack rewalk on
+# moved subtrees — is exercised on realistic surrounding text.
+def build_heavy(corpus)
+  violating =
+    "[url=https://ex.com/a][img]https://ex.com/i.png[/img][/url] " \
+      "[url=https://a.com][url=https://b.com]x[/url][/url] " \
+      "[url=https://ex.com][quote]q[/quote][/url] "
+  corpus.map { |post| violating + post }
+end
+
 # ASCII/multibyte corpus pair per source format.
 CORPORA = {
   bbcode: -> { [Corpus.ascii, Corpus.multibyte] },
+  bbcode_heavy: -> { [build_heavy(Corpus.ascii), build_heavy(Corpus.multibyte)] },
   mediawiki: -> { [Corpus.mediawiki, Corpus.mediawiki_multibyte] },
   html: -> { [Corpus.html, Corpus.html_multibyte] },
   text_formatter: -> { [Corpus.text_formatter, Corpus.text_formatter_multibyte] },
@@ -98,6 +111,44 @@ def report(name, corpus, &work)
       end
     end,
   ],
+  # The default-on gate: the normalization walk over violation-free trees
+  # in isolation (pre-parsed ASTs; the clean corpus never mutates, so every
+  # round measures the same zero-violation traversal). Compare against zero.
+  "norm_only" => [
+    :bbcode,
+    lambda do |corpus, tag|
+      parser = Markbridge::Parsers::BBCode::Parser.new
+      asts = corpus.map { |post| parser.parse(post) }
+      normalizer = Markbridge::Normalizer.shared_for(:discourse)
+      report("norm_only/#{tag}", asts) { |ast| normalizer.normalize(ast) }
+    end,
+  ],
+  # End-to-end baseline with normalization off; `fresh` minus this is the
+  # zero-violation overhead of default-on normalization.
+  "fresh_no_norm" => [
+    :bbcode,
+    lambda do |corpus, tag|
+      report("fresh_no_norm/#{tag}", corpus) do |post|
+        Markbridge.bbcode_to_markdown(post, normalize: false)
+      end
+    end,
+  ],
+  # Worst-case bound: end-to-end over a violation-heavy corpus, with and
+  # without normalization (the difference is the mutation + rewalk cost).
+  "norm_heavy" => [
+    :bbcode_heavy,
+    lambda do |corpus, tag|
+      report("norm_heavy/#{tag}", corpus) { |post| Markbridge.bbcode_to_markdown(post) }
+    end,
+  ],
+  "norm_heavy_off" => [
+    :bbcode_heavy,
+    lambda do |corpus, tag|
+      report("norm_heavy_off/#{tag}", corpus) do |post|
+        Markbridge.bbcode_to_markdown(post, normalize: false)
+      end
+    end,
+  ],
   "mw_fresh" => [
     :mediawiki,
     lambda do |corpus, tag|

docs/architecture.md

@@ -26,6 +26,12 @@ Markbridge follows a **Parse → AST → Render** architecture:
    Registry                                   Library
 ```
 
+Between the AST and Render, an optional `Markbridge::Normalizer` pass
+rewrites the tree so the renderer is only handed markup the target format
+can express (no link inside a link, no block inside an inline container,
+etc.). It runs by default at the conversion level; the three components
+above are unchanged by it. See [AST Normalization](normalization.md).
+
 ### Phase 1: Parsing (BBCode → AST)
 
 **Purpose:** Convert BBCode text into a structured tree

docs/normalization.md

@@ -0,0 +1,123 @@
+# AST Normalization
+
+Markup can nest elements in ways Markdown cannot express: a link inside a
+link, a block element inside an inline container (a link label, but also bold
+or a heading), a fenced code block inside emphasis. If the renderer prints
+these as-is, the Markdown breaks — the inner link wins and the outer one turns
+into text, a block's blank lines break out of the emphasis around it.
+
+`Markbridge::Normalizer` walks the AST once, between the parse-time `yield`
+hook and rendering, and rewrites it so the renderer only gets markup the
+target format can express. It runs **by default**. The renderer's tags stay
+simple string emitters; the rules about what may nest in what live here
+instead.
+
+## Where it runs
+
+```
+parse → yield(ast) → normalize → render
+```
+
+Because it runs after the `yield` hook, changes you make to the AST in that
+block are normalized too. It runs for every source format and for
+`Markbridge.render`, because normalization is about the *target* format, not
+the source.
+
+## The default rules
+
+The default rules are CommonMark legality. Break one and the Markdown does
+not parse back as the tree meant:
+
+- No link inside a link, at any depth (§6.3). The inner link is unwrapped.
+- An inline container holds inline content only, so a block element inside one
+  is moved out. This is not link-specific: emphasis (`Bold`, `Italic`, …) and
+  headings are inline containers too, so a poll inside bold or a list inside a
+  heading is handled the same way as a block inside a link. The lists are
+  `Normalizer::INLINE_CONTAINERS` and `Normalizer::BLOCK_NODES` (which covers
+  `List`, `Table`, `Quote`, `Details`, `HorizontalRule`, `Align`, and the
+  Discourse `Poll`/`Event` nodes).
+- A code span inside an inline container is fine while it stays on one line. A
+  fenced or multi-line block is moved out.
+
+Discourse-specific policy is **not** built in. Moving an image out of a link,
+for example, is a rule you add yourself (see below). A linked image
+(`[![alt](src)](url)`) is valid CommonMark, so the default leaves it alone.
+
+## Strategies
+
+Each match resolves to one strategy:
+
+| Strategy | Effect |
+|----------|--------|
+| `:keep` | Allow it. This records a decision and keeps it out of the report. |
+| `:hoist_after` | Move the node out and put it right after the outermost matching ancestor, keeping the document order. An image in a bold that sits in a link is moved after the whole link (out of both), because the bold is inside the link. The walker only moves a node out to a sibling; it never puts one into a wrapper it was not already in. |
+| `:unwrap` | Remove the element and put its children in its place. The built-in case is a link inside a link: `[[text](inner)](outer)` becomes `[text](outer)`. The inner link and its href are dropped; its text stays under the outer link. |
+| `:textify` | Replace the subtree with its plain text (`@name` for a mention, the joined text otherwise). |
+| `:drop` | Remove it. |
+| callable | `->(boundary, node) { … }` that returns a strategy symbol, an `Array<AST::Node>` to put in its place, or `nil` to drop it. Use this for anything the built-in strategies do not cover. |
+
+A formatting wrapper (bold, italic, color, …) that ends up empty after a
+hoist or drop is removed, so no empty `**` `**` markers are left. A link is
+the exception: an empty link is kept, because it renders as a plain URL.
+
+## Diagnostics
+
+Every change is reported through the same channel as `unknown_tags`:
+
+```ruby
+conversion = Markbridge.convert(input, format: :bbcode)
+conversion.diagnostics[:normalization]
+# => [{ parent: "Url", child: "Url", strategy: :unwrap, count: 1 }]
+```
+
+For a migration this feeds per-post warnings and shows which sources produce
+broken trees. The key is absent when nothing changed.
+
+## Opting out and customizing
+
+`normalize:` takes `true` (default, the shared normalizer), `false` (skip), or
+a `Normalizer` instance:
+
+```ruby
+# Skip normalization
+Markbridge.convert(input, format: :bbcode, normalize: false)
+
+# Add your own rules on top of the defaults
+normalizer = Markbridge::Normalizer.default
+normalizer.rule(parent: Markbridge::AST::Url, child: Markbridge::AST::Image, strategy: :hoist_after)
+Markbridge.convert(input, format: :bbcode, normalize: normalizer)
+```
+
+Build a customized normalizer once and reuse it. `#normalize` and
+`#violations` keep no state on the instance, so one instance (freeze it if you
+like) is safe to use for every conversion, also across threads. Passing your
+own instance is as fast as the default path — there is no per-call rule
+build.
+
+A rule for a `(parent, child)` pair that already exists is replaced, so your
+`#rule` calls override the defaults. Matching is by exact class
+(`instance_of?`), so a rule for `AST::Url` does not catch a subclass.
+
+`Markbridge::Normalizer.shared_default` is the default normalizer, built once
+and frozen; the `normalize: true` path uses it. Do not change it — call
+`.default` for a fresh, customizable one.
+
+## Validation
+
+The same rules, without changing the tree:
+
+```ruby
+Markbridge::Normalizer.default.violations(ast)
+# => [{ parent: "Url", child: "Url", strategy: :unwrap }]
+```
+
+Two uses: check in your own test suite that the trees your parsers and tag
+fixtures build have no violations, or run it as a lint over a corpus without
+changing any output. After a `normalize`, `violations` returns nothing —
+normalization is done in a single pass.
+
+## Adding a target format
+
+`Normalizer.default` builds the rule table; the engine (`RuleSet`, `Walker`)
+does not care about the format. A second target would add another builder and
+a matching class method next to `default`; nothing else changes.

docs/renderers/discourse.md

@@ -958,8 +958,17 @@ doc = AST::Document.new([
 renderer.render(doc) # => "content" (Unknown wrapper ignored)
 ```
 
+## AST normalization
+
+The renderer's tags are simple string emitters. They assume the tree is
+already legal for the target format (no link inside a link, no block element
+inside an inline container like bold or a heading). Making it legal is the job
+of a separate pass, `Markbridge::Normalizer`, which runs by default between
+parse and render. See **[AST Normalization](../normalization.md)**.
+
 ## Next Steps
 
+- **[AST Normalization](../normalization.md)** - Target-format nesting rules applied before rendering
 - **[BBCode Parser Guide](../parsers/bbcode.md)** - Learn how to build AST from BBCode
 - **[Extending Markbridge](../extending.md)** - Add custom tags and renderers
 - **[Architecture Overview](../architecture.md)** - Understand the full pipeline