Address html parser report

Author: ItsMeSamey

CHANGELOG.md

@@ -6,6 +6,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+### Compatibility
+
+- Impact: Non-breaking
+- Migration: Not required
+- Downstream scope: Small
+
 ### Added
 
 - Initial OSS release documentation set.

README.md

@@ -27,6 +27,8 @@ Typical HTML parsers optimize for browser-like behavior, strict correctness, or
 - **Navigation:** `parentNode`, `firstChild`, `lastChild`, `nextSibling`, `prevSibling`, `children` (element-only).
 - **In-place attributes:** attribute values are materialized/decoded lazily and cached in-place.
 - **Configurable parse work:** eager/lazy child views and optional whitespace-text dropping.
+- **Opt-in diagnostics:** `queryOneDebug` / `queryOneRuntimeDebug` expose near-miss reasons without changing hot-path APIs.
+- **Opt-in instrumentation:** compile-time hook wrappers for parse/query timing and node-count stats.
 
 Target Zig version: `0.15.2`.
 
@@ -96,6 +98,16 @@ const sel = try html.Selector.compileRuntime(arena.allocator(), "a[href^=https]"
 const node = doc.queryOneCompiled(&sel);
 ```
 
+### Debug query diagnostics
+
+```zig
+var report: html.QueryDebugReport = .{};
+const node = try doc.queryOneRuntimeDebug("a[href^=https]", &report);
+if (node == null) {
+    // Inspect report.visited_elements and report.near_misses
+}
+```
+
 ## Parse Option Recipes
 
 Two bundles are used by the benchmark harness and conformance runner:
@@ -112,6 +124,8 @@ Two bundles are used by the benchmark harness and conformance runner:
 
 `children()` returns a borrowed `[]const u32` index slice into the document's node array.
 
+See `docs/malformed-html-guidance.md` for a mode matrix and fallback workflow on malformed pages.
+
 ## Selector Support (v1)
 
 Supported (intentionally limited scope):
@@ -163,6 +177,13 @@ zig build conformance
 
 See `docs/conformance.md` for what’s covered and what’s intentionally out of scope.
 
+## Migration Notes
+
+- `CHANGELOG.md` now includes compatibility labels in `Unreleased`:
+  - `Impact: Breaking|Non-breaking`
+  - `Migration: Required|Not required`
+  - `Downstream scope: Small|Medium|Large`
+
 ## Documentation
 
 See `docs/README.md`.

bench/README.md

@@ -64,5 +64,5 @@ The benchmark output also includes a hard gate table:
 
 - `PASS/FAIL: ours-fastest > lol-html` per fixture
 - strict-mode regression checks against baseline:
-  - parse throughput not worse than -3% per fixture
-  - query parse/match/compiled not worse than -2%
+  - parse/query throughput not worse than -1%
+  - each failing case is automatically rerun 3 times and judged by rerun median

docs/README.md

@@ -4,6 +4,7 @@
 - [API Reference](api-reference.md)
 - [Selector Reference](selectors.md)
 - [Performance Guide](performance.md)
+- [Malformed HTML Guidance](malformed-html-guidance.md)
 - [Conformance Notes](conformance.md)
 - [Architecture](architecture.md)
 - [Troubleshooting](troubleshooting.md)

docs/api-reference.md

@@ -22,6 +22,8 @@ Query entrypoints:
 - `try doc.queryAllRuntime(selector)`
 - `doc.queryOneCompiled(&selector)`
 - `doc.queryAllCompiled(&selector)`
+- `doc.queryOneDebug(comptime selector, report)`
+- `try doc.queryOneRuntimeDebug(selector, report)`
 
 Helpers:
 
@@ -56,6 +58,8 @@ Scoped query entrypoints:
 - `try queryAllRuntime(selector)`
 - `queryOneCompiled(&selector)`
 - `queryAllCompiled(&selector)`
+- `queryOneDebug(comptime selector, report)`
+- `try queryOneRuntimeDebug(selector, report)`
 
 ## `Selector`
 
@@ -72,6 +76,16 @@ Scoped query entrypoints:
 
 - `normalize_whitespace: bool = true`
 
+## Debug/Instrumentation Types
+
+- `QueryDebugReport` and related debug enums (`DebugFailureKind`, `NearMiss`)
+- Wrapper helpers:
+  - `parseWithHooks(doc, input, opts, hooks)`
+  - `queryOneRuntimeWithHooks(doc, selector, hooks)`
+  - `queryOneCompiledWithHooks(doc, selector, hooks)`
+  - `queryAllRuntimeWithHooks(doc, selector, hooks)`
+  - `queryAllCompiledWithHooks(doc, selector, hooks)`
+
 ## Lifetime and Safety Notes
 
 - Nodes borrow from their owning `Document`.

docs/malformed-html-guidance.md

@@ -0,0 +1,34 @@
+# Malformed HTML Guidance
+
+This parser is permissive by design. For unreliable web HTML, choose parse options based on what your pipeline needs to optimize.
+
+## Mode Matrix
+
+| Mode | Parse Options | Best For | Tradeoffs |
+|---|---|---|---|
+| `strictest` | `.eager_child_views = true`, `.drop_whitespace_text_nodes = false` | Maximum traversal predictability and content fidelity | Higher parse-time work and memory traffic |
+| `fastest` | `.eager_child_views = false`, `.drop_whitespace_text_nodes = true` | Throughput-first extraction and selector-heavy scraping | Whitespace-only text nodes are dropped; child views materialize lazily on first `children()` |
+
+## Practical Fallback Playbook
+
+1. Start in `fastest` for bulk scraping.
+2. If a target site shows unstable text extraction or navigation assumptions, switch that site to `strictest`.
+3. Keep selectors robust:
+4. Prefer anchored selectors (`#id`, stable attrs) over deep sibling chains.
+5. Avoid dependence on whitespace-only text nodes.
+6. Use `queryOneRuntimeDebug` to inspect non-match reasons before changing selectors.
+
+## Example
+
+```zig
+try doc.parse(&input, .{
+    .eager_child_views = false,
+    .drop_whitespace_text_nodes = true,
+});
+
+var report: html.QueryDebugReport = .{};
+const node = try doc.queryOneRuntimeDebug("article .title > a", &report);
+if (node == null) {
+    // Inspect report.near_misses and report.visited_elements
+}
+```

docs/performance.md

@@ -32,6 +32,7 @@ Output artifacts:
 - Parse throughput is benchmarked against `strlen`, `lexbor`, and parse-only `lol-html`.
 - Query parse/match sections are measured on `htmlparser` only.
 - For repeated runtime selector workloads, prefer `Selector.compileRuntime` + `query*Compiled` APIs.
+- For malformed pages and fallback strategy guidance, see `docs/malformed-html-guidance.md`.
 
 ## Methodology
 

docs/troubleshooting.md

@@ -7,6 +7,7 @@ Checklist:
 - Verify selector syntax (`queryOneRuntime` can surface `error.InvalidSelector`).
 - Selectors are matched case-insensitively against tag/attribute names by default.
 - Confirm you are querying the expected scope (`Document` vs `Node` scoped queries).
+- For selector diagnosis details (visited count + near misses), use `queryOneRuntimeDebug(...)` and inspect `QueryDebugReport`.
 
 ## Missing or Unexpected `innerText`
 
@@ -25,6 +26,10 @@ Checklist:
 
 This is expected. Parsing and lazy attr/entity decode mutate the source buffer in place.
 
+## Malformed Page Behavior
+
+Use `docs/malformed-html-guidance.md` for mode selection (`strictest` vs `fastest`) and fallback workflow.
+
 ## Example Drift Policy
 
 All user-facing snippets must be backed by code under `examples/` and verified via:

src/debug/instrumentation.zig

@@ -0,0 +1,156 @@
+const std = @import("std");
+const ast = @import("../selector/ast.zig");
+const ParseOptions = @import("../html/document.zig").ParseOptions;
+
+pub const QueryInstrumentationKind = enum(u8) {
+    one_runtime,
+    one_compiled,
+    all_runtime,
+    all_compiled,
+};
+
+pub const ParseInstrumentationStats = struct {
+    elapsed_ns: u64,
+    input_len: usize,
+    node_count: usize,
+};
+
+pub const QueryInstrumentationStats = struct {
+    elapsed_ns: u64,
+    selector_len: usize,
+    kind: QueryInstrumentationKind,
+    matched: ?bool = null,
+};
+
+fn elapsedNs(start: i128, finish: i128) u64 {
+    if (finish <= start) return 0;
+    return @intCast(finish - start);
+}
+
+fn HookDeclType(comptime H: type) type {
+    return switch (@typeInfo(H)) {
+        .pointer => |p| p.child,
+        else => H,
+    };
+}
+
+fn matchedFromValue(value: anytype) ?bool {
+    return switch (@typeInfo(@TypeOf(value))) {
+        .optional => value != null,
+        else => true,
+    };
+}
+
+pub fn parseWithHooks(doc: anytype, input: []u8, comptime opts: ParseOptions, hooks: anytype) !void {
+    if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onParseStart")) {
+        hooks.onParseStart(input.len);
+    }
+
+    const start = std.time.nanoTimestamp();
+    try doc.parse(input, opts);
+    const stats: ParseInstrumentationStats = .{
+        .elapsed_ns = elapsedNs(start, std.time.nanoTimestamp()),
+        .input_len = input.len,
+        .node_count = doc.nodes.items.len,
+    };
+
+    if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onParseEnd")) {
+        hooks.onParseEnd(stats);
+    }
+}
+
+pub fn queryOneRuntimeWithHooks(doc: anytype, selector: []const u8, hooks: anytype) @TypeOf(doc.queryOneRuntime(selector)) {
+    if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onQueryStart")) {
+        hooks.onQueryStart(.one_runtime, selector.len);
+    }
+
+    const start = std.time.nanoTimestamp();
+    const out = doc.queryOneRuntime(selector);
+    if (out) |value| {
+        if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onQueryEnd")) {
+            hooks.onQueryEnd(QueryInstrumentationStats{
+                .elapsed_ns = elapsedNs(start, std.time.nanoTimestamp()),
+                .selector_len = selector.len,
+                .kind = .one_runtime,
+                .matched = matchedFromValue(value),
+            });
+        }
+        return value;
+    } else |err| {
+        if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onQueryEnd")) {
+            hooks.onQueryEnd(QueryInstrumentationStats{
+                .elapsed_ns = elapsedNs(start, std.time.nanoTimestamp()),
+                .selector_len = selector.len,
+                .kind = .one_runtime,
+                .matched = null,
+            });
+        }
+        return err;
+    }
+}
+
+pub fn queryOneCompiledWithHooks(doc: anytype, sel: *const ast.Selector, hooks: anytype) @TypeOf(doc.queryOneCompiled(sel)) {
+    if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onQueryStart")) {
+        hooks.onQueryStart(.one_compiled, sel.source.len);
+    }
+
+    const start = std.time.nanoTimestamp();
+    const value = doc.queryOneCompiled(sel);
+    if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onQueryEnd")) {
+        hooks.onQueryEnd(QueryInstrumentationStats{
+            .elapsed_ns = elapsedNs(start, std.time.nanoTimestamp()),
+            .selector_len = sel.source.len,
+            .kind = .one_compiled,
+            .matched = matchedFromValue(value),
+        });
+    }
+    return value;
+}
+
+pub fn queryAllRuntimeWithHooks(doc: anytype, selector: []const u8, hooks: anytype) @TypeOf(doc.queryAllRuntime(selector)) {
+    if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onQueryStart")) {
+        hooks.onQueryStart(.all_runtime, selector.len);
+    }
+
+    const start = std.time.nanoTimestamp();
+    const out = doc.queryAllRuntime(selector);
+    if (out) |iter| {
+        if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onQueryEnd")) {
+            hooks.onQueryEnd(QueryInstrumentationStats{
+                .elapsed_ns = elapsedNs(start, std.time.nanoTimestamp()),
+                .selector_len = selector.len,
+                .kind = .all_runtime,
+                .matched = null,
+            });
+        }
+        return iter;
+    } else |err| {
+        if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onQueryEnd")) {
+            hooks.onQueryEnd(QueryInstrumentationStats{
+                .elapsed_ns = elapsedNs(start, std.time.nanoTimestamp()),
+                .selector_len = selector.len,
+                .kind = .all_runtime,
+                .matched = null,
+            });
+        }
+        return err;
+    }
+}
+
+pub fn queryAllCompiledWithHooks(doc: anytype, sel: *const ast.Selector, hooks: anytype) @TypeOf(doc.queryAllCompiled(sel)) {
+    if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onQueryStart")) {
+        hooks.onQueryStart(.all_compiled, sel.source.len);
+    }
+
+    const start = std.time.nanoTimestamp();
+    const iter = doc.queryAllCompiled(sel);
+    if (comptime @hasDecl(HookDeclType(@TypeOf(hooks)), "onQueryEnd")) {
+        hooks.onQueryEnd(QueryInstrumentationStats{
+            .elapsed_ns = elapsedNs(start, std.time.nanoTimestamp()),
+            .selector_len = sel.source.len,
+            .kind = .all_compiled,
+            .matched = null,
+        });
+    }
+    return iter;
+}