Skip to content

Make DocGen audit token-safe and harden model artifact recovery#2

Open
fajarnugraha37 wants to merge 31 commits into
mainfrom
agent/phase2-validated-output
Open

Make DocGen audit token-safe and harden model artifact recovery#2
fajarnugraha37 wants to merge 31 commits into
mainfrom
agent/phase2-validated-output

Conversation

@fajarnugraha37

@fajarnugraha37 fajarnugraha37 commented Jul 16, 2026

Copy link
Copy Markdown
Owner

Summary

Makes DocGen 2.0 token-safe and resilient against malformed provider artifacts, repeated missing-model failures, and late audit rejection.

The implementation remains language-, framework-, runtime-, protocol-, storage-, and architecture-neutral.

Token-safe audit execution

  • deterministic validation runs before any audit-provider call
  • deterministic failure stops immediately with audit-provider calls 0
  • audit.llmMode supports off, advisory, and blocking
  • default and legacy migration use off
  • required-section wording mismatches migrate to warnings instead of technology-specific hard failures
  • invalid or out-of-context evidence and model references are sanitized
  • unsupported FACT claims are downgraded to INFERENCE
  • stale indexed sources remain a hard failure
  • domain arrays or objects named classification are preserved instead of being overwritten as semantic metadata

Model artifact recovery

  • accepts wrapper, key, and payload variations; JSON strings; arrays; descriptor objects; filename and camel-case aliases; and direct singleton repairs
  • salvages valid objects from partial bundles
  • performs bounded batch and per-object recovery
  • emits explicit zero-confidence UNKNOWN placeholders when configured
  • keeps strict failure policy available
  • atomically publishes only the reconciled model set
  • removes repeated fatal missing model object behavior

Cross-platform reliability

  • provider-fixture execution now uses a Windows command shim
  • model-recovery regressions execute on Windows as well as Linux
  • CI validates Node.js 22 and 24 on Ubuntu and Node.js 24 on Windows

Verification

Hosted GitHub Actions run 29555709192 passed completely:

  • Ubuntu / Node.js 22: syntax, full unit and integration suite, installer dry-run
  • Ubuntu / Node.js 24: syntax, full unit and integration suite, installer dry-run
  • Windows / Node.js 24: syntax, full unit and integration suite, installer dry-run

Current head: b11f6af8a40c8353f4d7296e11f7389583d0f889.

The PR remains open and unmerged.

@fajarnugraha37 fajarnugraha37 changed the title Agent/phase2 validated output Implement Phase 2A runtime stabilization and Phase 2B stack-neutral quality gates Jul 16, 2026
@fajarnugraha37 fajarnugraha37 changed the title Implement Phase 2A runtime stabilization and Phase 2B stack-neutral quality gates Stabilize provider normalization, quality audit, and incomplete model bundle repair Jul 16, 2026

Copy link
Copy Markdown
Owner Author

Fixed the two real-repository failures reported after Phase 2 validation:

  1. Quality failed: deterministicFailures=426

    • Added deterministic provider-output canonicalization for confidence, classification aliases, evidence aliases, and source-model references.
    • Provider FACT values without direct/inherited evidence are downgraded to INFERENCE; invalid source paths, stale hashes, invalid line ranges, and truly unresolved refs still fail.
    • Existing reusable traces are re-finalized during docgen resume, so no .docgen deletion is required.
    • Required sections now use normalized semantic heading overlap rather than exact substring matching.
  2. Model bundle is missing object: decisions

    • Valid bundle objects are preserved.
    • Only omitted model objects are requested through a targeted bounded repair call.
    • Alternate wrappers/key styles are accepted, and the stage still fails explicitly if repair remains incomplete.

Regression coverage increased to 22 tests. Local syntax checks, all 22 tests, and installer dry run passed. Final branch head after removing temporary staging files: 1bbfb5865f347299fab04bacbcc89196bb9b639d.

The PR remains open and unmerged.

@fajarnugraha37
fajarnugraha37 force-pushed the agent/phase2-validated-output branch from 1bbfb58 to 376ea13 Compare July 16, 2026 23:09
@fajarnugraha37
fajarnugraha37 force-pushed the agent/phase2-validated-output branch from 376ea13 to 5d46ac3 Compare July 16, 2026 23:11

Copy link
Copy Markdown
Owner Author

History cleanup completed. The earlier transport/staging SHA mentioned above has been superseded.

Current clean head: 5d46ac39a96e81d9ebb1d130b4840b06cc22e4a5

This is one atomic commit on top of the previous PR head and contains only the seven intended source/test changes. The file blobs match the locally validated 22-test source exactly; no bootstrap workflow or patch fragments remain.

@fajarnugraha37
fajarnugraha37 force-pushed the agent/phase2-validated-output branch from 77d25cc to 179edfd Compare July 17, 2026 01:14
@fajarnugraha37 fajarnugraha37 changed the title Stabilize provider normalization, quality audit, and incomplete model bundle repair Harden provider artifacts, quality audit, and model-bundle recovery Jul 17, 2026

Copy link
Copy Markdown
Owner Author

Re-audit completed for the repeated missing object failure class.

This is no longer a decisions-specific patch. The active model pipeline now:

  • accepts normalized names, *.json keys, nested known/unknown wrappers, descriptor arrays, JSON-string payloads, array payloads, and direct singleton repair objects;
  • salvages valid partial output;
  • performs one batch repair and at most one independent repair per unresolved model;
  • atomically publishes the reconciled model set;
  • defaults exhausted recovery to an explicit UNKNOWN/confidence-0/no-evidence placeholder;
  • exposes degraded models through docgen status;
  • retains opt-in strict failure through execution.missingModelPolicy: "fail".

The old fatal literal Model bundle is missing object is absent from the active source. Regression coverage is now 35 tests, including repeated omission and never-return scenarios. Local syntax, all 35 tests, installer dry run, and diff validation passed.

Clean atomic head: 179edfd10a5ce4fc10d4046f675c8f1dfc852560. PR remains open and unmerged.

@fajarnugraha37
fajarnugraha37 force-pushed the agent/phase2-validated-output branch from 179edfd to d764e7f Compare July 17, 2026 01:22

Copy link
Copy Markdown
Owner Author

Final re-audit pass completed.

Three additional recurrence paths were found and closed after the first hardening pass:

  1. an initial bundle containing zero recognizable requested objects now enters bounded recovery instead of failing before repair;
  2. metadata-only responses such as { "message": "cannot comply", "status": "error" } are rejected rather than misclassified as direct singleton models;
  3. artifact descriptors prioritize explicit model identity (modelName, filename, path, name) over generic id fields.

The full local gate now passes 37/37 tests, plus syntax checks, installer dry run, and git diff --check. The five final delta blobs on GitHub match the locally tested hashes exactly.

Current clean atomic head: d764e7fcefd55c02978be1a6b744ce4febc71248.

The old active fatal literal Model bundle is missing object has zero search results. Default mode uses bounded recovery and an explicit UNKNOWN/no-evidence degraded placeholder; strict failure remains opt-in through execution.missingModelPolicy: "fail".

PR remains open and unmerged.

@fajarnugraha37
fajarnugraha37 changed the base branch from main to ci/pr2-diagnostic-base July 17, 2026 04:29
@fajarnugraha37
fajarnugraha37 changed the base branch from ci/pr2-diagnostic-base to main July 17, 2026 04:46
@fajarnugraha37 fajarnugraha37 changed the title Harden provider artifacts, quality audit, and model-bundle recovery Make DocGen audit token-safe and harden model artifact recovery Jul 17, 2026

Copy link
Copy Markdown
Owner Author

Fixed the failing checks.

Root causes:

  • the audit sanitizer overwrote domain classification arrays/objects as semantic metadata;
  • regression assertions still expected behavior from before input sanitization and bounded recovery;
  • the model-bundle provider fixture was launched directly as .mjs on Windows instead of through a .cmd shim.

Final GitHub Actions run 29555709192 is green on Ubuntu Node.js 22, Ubuntu Node.js 24, and Windows Node.js 24. Syntax checks, the full unit/integration suite, and installer dry-run all passed.

Current head: b11f6af8a40c8353f4d7296e11f7389583d0f889.

The PR remains open and unmerged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant