feat(website): serve the documentation at webjs.dev/docs#1101
Conversation
|
Found while migrating: a root The plan for Silently is the important part. There is no error and no warning, and an app with a middleware that never loads is indistinguishable from an app that has none. That is why it survived: Every other routing convention matches on the file STEM (a per-segment Fixed in Verified on both runtimes ( |
|
Decision: one llms.txt for the site, not two The two apps each published their own
I kept the enumeration derived from the live pages rather than a curated list. Adding a doc page puts it in the index, the sitemap, and the corpus with nothing else to remember, which is the same property the old docs-app index had and the main reason it was worth preserving. Two things the merge exposed that were already broken: The search endpoint rooted its index at The progressive-enhancement page's counter sample rendered as |
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Went through the whole migration: the moved routes, the shared-chrome layout, the redirect host, the middleware fix, and every surface that names the docs URL.
The migration itself holds up. The redirect host has no open-redirect or header-injection hole (WHATWG URL strips CR/LF before parsing and pathname always starts with /, so a //evil.com path resolves onto webjs.dev), invariants 8 and 9 hold in the new sub-layout, and every asset the moved pages need is present and reachable through the auth gate.
Two real bugs, both from the docs no longer owning their own document.
The mobile drawer paints under the site header, because the root layout wraps children in a stacking context and the drawer cannot escape it. The old docs app could not hit this: its header and its drawer were siblings in the same context. Worth remembering as a general hazard for anything the sub-layout wants to float.
code-highlight.js leaks a document.body MutationObserver per navigation into /docs, for the same structural reason: the script moved from a root layout (never swapped) into a sub-layout (inside the router's swap range), and the router re-executes scripts it swaps in. The script had no idempotence guard because it never needed one.
The rest is stale text, and more of it than I expected: five CTAs that kept their new-tab screen-reader cue after becoming same-origin links, plus DOCS_URL and docs.webjs.dev surviving in the agent skill, two hooks, README, and website/AGENTS.md. The .claude/skills/* files got updated but the two hooks that inject the same guidance did not, which is the kind of pair that is easy to half-do.
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Second pass, this time on the tests and the client-side behaviour rather than the file moves, plus the design feedback that the docs still read as pasted in.
The design half came down to background. The sidebar had an opaque fill and a divider rule, so the page's warm glow stopped dead at the content column and the two halves of every docs page had different backgrounds. It was also full-bleed while every other page is centred in the header's 1240px container, and it kept the old app's type voice (orange h1, a rule above every h2) where the site's other long-form surface uses serif in --fg with space instead of rules. All three now match, and only the mobile drawer keeps a fill because it floats over what it covers.
The behavioural half was more interesting, and both bugs share a root cause: the docs stopped owning their own document. Things that were safe in a root layout are not safe in a sub-layout inside the router's swap range, and things that were safe on their own origin are not safe on a shared body element. A top-level script in a swapped range was never reactivated, so entering the docs by soft navigation left every code sample unhighlighted and never installed the observer that would have fixed later navigations. And the drawer's open state, an attribute on body, rode along to whatever page you left for and locked its scrolling.
On the tests, two of mine were weaker than they read. The cwd-independence check could not fail: prod mode does not bust the module cache, so the chdir happened after the modules had already resolved their roots. It runs in a fresh process now. The header/footer equality is real but proves less than its comment claimed, since it always slices the root layout's header; the count assertion beside it is what actually catches a second one. I corrected the comment rather than dressing it up.
Also worth noting: the bun boot check was probing the redirect host's empty 301s, which pass the status check and read zero preloads, so it looked like docs coverage while providing none.
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Third pass, zoomed out to whole-change coherence and the things a per-file read misses. This one was the most useful of the three, because most of what it found is invisible in the diff.
Two guards had gone quiet rather than red, which is worse than breaking. docs-pages-well-formed kept globbing the old app, so it matched one redirect stub with no template in it and ran its balanced-tag check on an empty string. Its tripwire asked for "more than zero" pages, which one satisfies, so nothing complained while all 45 real pages went uncovered. And they matter more than before: the bug that test was written against was an unclosed tag swallowing a client-router marker, and the docs are inside the marketing app's router for the first time. Separately, the doc-sync hook still accepted ^docs/ as a documentation surface, so staging a tsconfig from a three-file redirect host would have satisfied the gate.
The lockfile was the one thing that would have failed the merge outright rather than just rotting: the docs workspace was renamed and npm ci validates that.
The rest was drift I had propagated instead of finishing: both asset generators still wrote into the deleted docs/public, CI step names still advertised docs boot coverage this PR deliberately removed, and README plus the AGENTS.md app-layout block still described the docs as a separate app.
The client-router fix is out of this PR, filed as #1102. Nothing here depends on it any more, since moving code-highlight.js to the root layout means no swapped script needs reactivating. It is a real bug and the fix is small, but it changes swap semantics, and AGENTS.md rule 1 asks for a browser or e2e assertion on a client-router change rather than the unit test it shipped with. That is a PR of its own, not a rider on a docs migration. packages/core is now back to one changed line on this branch, the docs URL in an error message.
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Fourth pass, the convergence check: re-verify the previous rounds' fixes, then look at the moved pages themselves and the SEO surface, which is the whole point of the change.
The worst finding is one the migration itself introduced. Deleting the docs' own root layout took its title and description with it, so every one of the 45 pages was serving the landing page's blurb as its search snippet and social card, differing only in <title>. On a change whose entire justification is search, that is the wrong regression to ship, and neither the chrome test nor the repo-health SEO test looked at description or og. The sub-layout carries docs-scoped metadata now.
Three mobile problems, all consequences of the new shell rather than anything inherited. A wide table on /docs/runtime pushed the document past the viewport and dragged the fixed header with it, because pre had an overflow container and table did not. The menu button only gained aria-expanded after its first click, so it announced as a plain button until then and never gained state at all with JS off. And the closed drawer was only translated off-screen, so 45 links sat in the tab order ahead of the content on every docs page.
Three internal docs links have been 404ing since before the move (/docs/route-handlers, /docs/caching, /docs/advanced, none of which are real slugs). Fixing them is a footnote; the guard is the point, so the next one fails in CI rather than shipping.
Last, the per-page markdown routes are full-text copies of pages that also exist as HTML, and text/plain cannot carry a canonical. Indexing both would put 45 near-duplicates of the docs on the exact domain this change exists to consolidate, which is the same reasoning the sitemap already uses to exclude the /docs redirect. They now carry X-Robots-Tag: noindex and stay fetchable, which is what an agent asking for llms.txt actually wants. /llms.txt itself is left indexable: it is a link list, not a copy of anything. (That one is in website/lib/docs-llms.server.ts, which GitHub renders as a rename with no diff, so it has no inline anchor.)
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Fifth pass, aimed at the most recent commits, because each round's fixes have been the source of the next round's findings and this one is the clearest example.
The metadata fix I shipped last round made the thing it was fixing worse. Metadata merges as a shallow spread per layer, so naming openGraph in the sub-layout REPLACES the root's object instead of merging into it. Every docs URL lost og:image, its dimensions, og:url, and the summary_large_image card, so they shared as bare text cards. The description regression I was fixing was real, but I traded it for a bigger one, and my own comment claimed "the og:image stays the site's, deliberately" while the rendered HTML said otherwise. Worse, the test I wrote alongside it asserted only the fields I had just set, so it passed throughout. That is the lesson worth writing down: a test written from the fix rather than from the risk will confirm whatever you just did.
The table fix had the same shape. It solved the mobile overflow and introduced a desktop one, because display: block also makes a table shrink to fit, so row rules stopped short of the column edge on 15 pages. It only ever needed to apply at the narrow breakpoint.
Two things four rounds had not looked at. At exactly 860 CSS pixels the docs had no navigation at all: my hand-written media query includes 860 and Tailwind's max-[860px]: variant excludes it, so the grid had already collapsed and the button was already hidden while the drawer rules had not taken over. And the link guard I added last round could not see the sidebar, which is the one surface linking all 45 pages, because those hrefs are single-quoted literals rendered through a template hole. It also dropped fragment links entirely instead of checking their path, which is how a dead /docs/components#state walked straight past it.
Smaller but real: the highlighter was the only script in the root layout without a CSP nonce, so turning on the opt-in nonce CSP would have blocked exactly it; the llms 404 path reused the 200's five-minute public cache, so a shared cache would pin a not-found whose body echoes the requested topic; and the corpus builder used raw NUL bytes as its sentinel, which makes git treat a 16KB shared server module as binary and therefore unreviewable.
The docs lived on docs.webjs.dev, which cost us twice. In search, a subdomain accrues authority to itself rather than to webjs.dev, and the docs are the largest body of indexable content the project has. In design, they carried their own layout, nav, and footer, so a reader crossing over from the marketing site left one design system and entered another. Moving them onto the marketing app as app/docs/** fixes both at once. The pages now render inside the site's own shell, so the header, footer, theme toggle, fonts, and design tokens are literally the same code the rest of the site uses. The only docs-specific chrome left is the page-tree sidebar. docs.webjs.dev becomes a redirect-only service. It has to keep resolving indefinitely, because framework error messages in already-published npm packages point at it and a published version can never be corrected after the fact.
The root middleware lookup was the single literal `middleware.js`, so an app whose root middleware was written in TypeScript never had it loaded. There was no error and no warning: a missing middleware looks exactly like an app that has none, which is why this went unnoticed even though examples/blog ships a root middleware.ts that has never run. TypeScript is the documented default for an app, the scaffold writes middleware.ts for the api template, the dev supervisor watches both extensions, and every other routing convention matches on the file stem. This was the one lookup that did not. Resolves .ts, .js, .mts, and .mjs, .ts first to match the supervisor's order. Found while making docs.webjs.dev a redirect-only host, whose whole behaviour is a root middleware.
The documentation URL changes with the move, so every place that names it has to move with it: the error messages the framework throws, the scaffold's generated footer and agent rule files, AGENTS.md, README, the skill references, and the tests that anchor on those strings. The error-message URLs in core and server can only be corrected going forward. Anyone running an already-published version keeps getting the old host, which is exactly why docs.webjs.dev has to keep redirecting rather than being retired. Two smaller things fixed on the way. The UI website derived its docs link from its own DOCS_URL env var, which is now one service variable that can drift from WEBSITE_URL for no reason, so it reads WEBSITE_URL + /docs instead. And the progressive-enhancement page's counter sample rendered as @click=${'() => this.inc()'}, quoted, which is the exact thing invariant 4 forbids, on the page teaching the model.
The move is easy to half-undo without anything looking wrong: someone reintroduces a docs-only header, or adds a doc page that never reaches the sitemap, and the pages still render fine. These are the assertions that make those fail. Shared chrome is asserted by comparing the rendered header and footer of a doc page against a marketing page byte for byte, through the real request pipeline rather than the layout function, since 'the docs end up wrapped in the root layout' is the property under test. The search endpoint gains its first tests. Its index was rewritten to reuse the llms.txt extraction, which removed a process.cwd() root and an import that reached four levels out of the app into packages/. The cwd one is pinned by moving the working directory out from under a fresh handler, because the suite otherwise runs from the one directory that cannot catch it. Also repoints the doc tests, the api-coverage corpus, and the knowledge and preload sweeps at the app that serves the docs now.
The docs host renders no HTML now, only redirects, so it has no public/input.css to compile and the image build step for it would fail. The website carries the docs, so its DOCS_URL cross-link is gone, and the docs service gains SITE_URL, which is where it sends every request.
Review turned up nine things. The two that matter to a reader: The mobile docs drawer painted UNDER the site header. The root layout wraps children in a relative z-1 element, which is a stacking context, so nothing the docs layout renders can rise above the z-20 header no matter what z-index it claims. The old docs app did not hit this because its header lived in the same context as its drawer. The drawer now opens below the header instead of trying to cover it, which sidesteps the stacking context and leaves the header usable while it is open. code-highlight.js attached a document.body MutationObserver every time it ran, and it now sits inside the client router's swap range, so each marketing-to-docs navigation leaked another one. Guarded. The rest were stale text: five docs CTAs kept a screen-reader 'opens in a new tab' cue after becoming same-origin links, and DOCS_URL and docs.webjs.dev survived in the agent skill, two hooks, README, and website/AGENTS.md. Also widens the dev supervisor's middleware watch list to the four extensions the server now loads, with a test that reads the server's list rather than restating it, since a middleware that loads but never triggers a dev restart is the same quiet failure in a new place.
The docs still read as a section pasted into the site rather than part of it, and the reason was mostly background. The sidebar carried an opaque fill and a divider rule, so the page's warm glow stopped at the content column and the two halves had different backgrounds. It was also full-bleed while every other page is centred in a 1240px container, and it kept the old docs app's type voice: an orange h1 and a rule above every h2, where the site's other long-form surface uses serif headings in --fg with space instead of rules. The sidebar is transparent now, the shell shares the header's container so the nav lines up with the wordmark above it, and the prose tracks the blog renderer's scale. Only the mobile drawer keeps a fill, because it floats over the content it covers. Also from review: code-highlight.js never ran when a reader entered the docs by soft navigation. It sat in the docs sub-layout, inside the router's swap range, and a top-level script in a swapped range was never reactivated. Fixed in the router, and the script moved to the root layout (never swapped) scoped to .prose-docs, so it no longer depends on swap mechanics at all. Marketing pages keep the spans lib/highlight.ts already emitted at SSR. The drawer's open state lived on <body>, outside every swap range, so it rode along to the next page and locked its scrolling. Cleared from the root layout, which survives navigation. Smaller: doc-search probed a window.navigate global the router has never defined, so every search result did a full reload; the redirect host answered /__webjs/core/* with a canned body; the docs main had no skip-link target and then no tabindex; and the bun boot check probed the redirect host's empty 301s while believing it covered the docs.
The lockfile still recorded the docs workspace under its old name, which npm ci validates, so CI would have failed on it. Two guards had gone quiet rather than red. docs-pages-well-formed still globbed docs/app, which now matches one redirect stub with no template in it, so the balanced-tag check ran on an empty string and passed while covering none of the 45 real pages; its tripwire said 'more than zero', which that satisfies, so it now asserts a real floor. And the doc gate hook still accepted ^docs/ as a documentation surface, where staging a tsconfig from a three-file redirect host would satisfy it. Both favicon and OG generators still wrote into the deleted docs/public, and CI step names, README, and the AGENTS.md app-layout block still described the docs as a separate app. The client-router fix is reverted out of this PR and filed as #1102. Nothing here depends on it any more: code-highlight.js moved to the root layout, so it no longer needs a swapped script to be reactivated. It is a real bug, but it changes swap semantics and AGENTS.md rule 1 asks for a browser or e2e assertion on a client-router change, which is more than belongs in a docs migration.
Dropping the docs' own root layout took its title and description with it, so all 45 pages were advertising the landing page's blurb as their search snippet and social card, with only the title differing. That is a regression the migration introduced, and on an SEO-motivated change it is the wrong one to ship. The sub-layout now carries docs-scoped metadata; pages still own their titles. Three mobile problems, all from the new shell. A wide table pushed the whole document sideways on /docs/runtime, taking the fixed header with it, because pre had an overflow container and table did not. The menu button gained aria-expanded only after its first click, so it announced as a plain button until then and never gained state with JS off. And the closed drawer was merely translated off-screen, so 45 links stayed in the tab order in front of the content on every docs page. Also: three internal docs links pointed at slugs that do not exist (route-handlers, caching, advanced), shipping 404s since before the move. Fixed, with a test that walks every internal /docs href through the real handler so the next one fails instead of shipping. And the per-page markdown routes now carry X-Robots-Tag: noindex. They are full-text copies of pages that also exist as HTML, and text/plain cannot carry a canonical, so indexing both would put 45 near-duplicates on the exact domain this whole change exists to consolidate. They stay fetchable, which is what an agent asking for llms.txt wants. The site index at /llms.txt is deliberately left indexable.
The metadata fix in the last commit made things worse before better. Metadata merges as a shallow spread per layer, so the sub-layout's openGraph object REPLACED the root's rather than merging into it, and every docs URL lost og:image, its dimensions, og:url, and the summary_large_image card. Restating every field is the fix; the test now asserts the image and card type survive, which is what it should have checked in the first place rather than only the fields I had just set. The table overflow fix was applied at every width, so a table also shrank to fit its content on desktop and its row rules stopped short of the column edge on 15 pages. Scoped to the narrow breakpoint, where it is the only thing that stops a wide table dragging the fixed header off screen. At exactly 860px the docs had no navigation at all: the hand-written media query includes 860 while Tailwind's max-[860px] variant excludes it, so the grid had collapsed and the button was hidden while the drawer rules had not taken over. The link guard could not see the sidebar, which is the only surface that links all 45 pages, because its hrefs are single-quoted literals in a template hole. It also dropped fragment links whole rather than checking their path, which is how a dead /docs/components#state survived it. Smaller: the highlighter had no CSP nonce, so enabling the opt-in nonce CSP would have blocked exactly it; the llms 404 path reused the 5-minute public cache, so a shared cache would pin a not-found whose body echoes the requested topic; and the corpus builder used raw NUL bytes as a sentinel, which made git treat a 16KB shared server module as binary and therefore unreviewable.
The previous commit said the test covers og:image and the card type. It did not; I added the assertion after the fact rather than before. It asserts the fields a shallow metadata merge would silently drop, which is the failure that actually happened, and the counterfactual confirms it fails when the merge is broken again.
The whole sidebar scrolled, so the search field scrolled out of reach on a long page and the scrollbar ran the column's full height, pressing against the field. The aside is a column now: the search box is pinned and only the link list below it scrolls, with the scrollbar inset off the links. The links also pulled themselves back out with a negative margin, so their text sat flush against the container edge and the hover pill bled past it. Dropping it indents the text by the link's own padding and lines the pill up with the sidebar edge.
6f4d996 to
bc6f528
Compare
The order was a hand-copied list and it had already drifted: it carried a slug that no longer exists and was missing six real pages, so the AI-facing index put Runtime and Security in an alphabetical tail rather than in Getting Started and Infrastructure. Deriving it from the sidebar removes the failure mode instead of correcting one instance of it. It parses the layout source rather than importing it, since the layout pulls in browser components and this module is server-only. Also adds the assertion that actually rots in the other direction: the sitemap and llms.txt both enumerate from disk, and /docs is a redirect rather than a listing, so the sidebar is the only human path into a doc page. A new page would land in both indexes while being unreachable by a reader, with every test green. The Bun proof gets its own CI step, matching the convention the other listener proofs use, so its own comment is no longer describing a step that did not exist. That matters because denylist entries in the Bun runner justify skipping a file by pointing at exactly such a step. The /docs/auth and /docs/authentication pages ship identical titles and contradict each other about whether the framework ships auth. Pre-existing, but this PR is what starts submitting the pair to crawlers, so it is filed as #1103 rather than fixed here.
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Sixth pass, late-stage: whole-diff coherence, whether the test SET covers the real risk surface, a functional sweep against the running server, and merge readiness.
The blocker was that the branch conflicted with main. 8ce50392 modified docs/app/docs/deployment/page.ts while this PR deletes it, and GitHub reported CONFLICTING. The content risk was the real part: resolving it the obvious way (delete wins, since the file moved) would have dropped the X-Forwarded-Proto parity line and the whole "embedding moves the proxy-header responsibility to you" caveat from the live docs, which is a security-relevant deploy warning. Rebased; git's rename detection carried the change onto the moved file, and I verified website/app/docs/deployment/page.ts is now byte-identical to main's version rather than assuming it.
The test-set gap was a direction I had not covered. The sitemap and llms.txt both enumerate topics from disk, and /docs is a redirect rather than a listing page, so the sidebar is the only surface a human can navigate from. A new doc page would appear in the sitemap and the AI index automatically while being unreachable by a reader, with every test green. That is now asserted, and the counterfactual (drop a page in without a nav entry) fails it.
The CURATED_ORDER list in the corpus builder had already drifted before this PR: one slug that no longer exists, six real pages missing, so /llms.txt listed Runtime and Security in an alphabetical tail. I derived it from the sidebar rather than correcting the list, since correcting it just resets the clock on the same drift. (That is in website/lib/docs-llms.server.ts, which GitHub renders as a rename with no diff, so it takes no inline anchor.)
Also fixed a comment that described a CI step which did not exist, and added the step, because the Bun runner's denylist entries justify skipping files by pointing at exactly that kind of dedicated step.
One thing I did not fix here: /docs/auth and /docs/authentication ship identical titles and open by contradicting each other about whether the framework ships auth. It is pre-existing content, but this PR is what starts submitting the pair to crawlers from a sitemap, so it is filed as #1103 rather than folded in.
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Seventh pass, and this one is clean.
Checked the rebase is honest rather than assuming it: the pre-rebase and post-rebase diffs are byte-identical apart from blob hashes and hunk offsets, 8ce5039 is an ancestor of HEAD, and both of its deployment-page edits survived the move into the website app. Its listener and forwarded-proto work is untouched by this branch.
Checked the newest commit's edge cases: an empty layout parse degrades to alphabetical and is caught by the ordering test, an unlinked topic ranks to the tail and is caught by the orphan test, and the two caches are independent. The live /llms.txt order matches the sidebar exactly.
Everything green: website 102 node plus 24 browser, test/docs 29, preload-subset, repo-health, the root-middleware resolution tests, app-boot on both runtimes, and webjs check in both apps.
On the deploy: the Dockerfile drops only the deleted docs stylesheet, and a real Tailwind build of the website picks up the docs-only classes, so the merged docs are styled in the image. Compose, the Railway start commands and the healthcheck paths are unchanged, and the docs service has no variables set so SITE_URL correctly falls back to the production origin.
One note for whoever picks this up next, not a defect: booting docs/ from this worktree through the symlinked CLI resolves @webjsdev/server from the primary checkout, which lacks the middleware fix, so deep paths 404. That is a worktree artifact; the behaviour is correct through createRequestHandler and in the deploy.
The section label had no horizontal padding while the links had 12px, so the label sat further left than the items under it. Both carry the same small inset now, which is what makes the label read as a heading for the group rather than a separate column.
Every list in the docs rendered as indented plain text with no bullets or numbers: Tailwind's preflight sets list-style: none on ul/ol, and the prose rules re-added only the padding. Without a visible marker the 24px inset read as an arbitrary, inconsistent indent (the report that exposed it was 'why is this section indented', on a numbered how-it-works list whose numbers were invisible). With markers restored the indentation rule across the docs is simple and uniform: text sits flush unless it carries a marker, and the inset exists only to house that marker. Also fixes the description extractor stopping dead at an escaped quote, which published the garbage fragment 'WebJs pages and components are SSR\' as the progressive-enhancement entry in /llms.txt and the search index, plus a nav comment describing padding values that were never the ones on the element.
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Eighth pass, a fresh full read of the whole diff after the padding commit.
Three things, and the interesting one only looked cosmetic. The report was "why is this section indented", on the suspense page's how-it-works list. The actual cause: Tailwind's preflight sets list-style: none on every ul/ol, and the prose rules re-added only the padding, so every list in the docs has been rendering as indented plain text with NO bullets or numbers. The numbered steps on that page were a numbered list whose numbers were invisible, at which point the indent genuinely is arbitrary. Markers are restored now, which makes the indentation rule across the docs uniform: text sits flush unless it carries a marker, and the inset exists only to house the marker. A guard asserts the restatement stays, since deleting it renders fine, just wrong, and nothing else would notice.
The second is in the corpus extractor: the description regex stopped dead at an escaped quote, so the progressive-enhancement page, whose description contains SSR\'d, published the garbage fragment "WebJs pages and components are SSR" as its entry in /llms.txt and fed the same fragment to the search index. Fixed to cross escapes and unescape the capture, with a test asserting the known page is whole and that no index line anywhere ends in a backslash, so the next escaped quote fails loudly. (The extractor lives in website/lib/docs-llms.server.ts, a rename GitHub shows with no diff, so it takes no inline anchor.)
Third was a comment on the sidebar nav describing padding values that were never on the element, wrong at birth and made more wrong by the padding commit next to it. Corrected to describe what is actually there.
The padding commit itself is sound: the label and links share the inset, the utilities exist in the compiled stylesheet, and the live render matches.
The list was a ul with numbers hand-written into the item text, which the invisible-marker era let pass as a plain numbered list. With markers restored it rendered a bullet stacked on each number. A real ol is also what a screen reader or crawler should have been given all along.
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Ninth pass, converging. Verified the escape-crossing regexes (unescape order relative to entity decoding is safe, no metadata uses control escapes that would mis-fold), the marker CSS in both themes and on the one nested list, and the suites.
One finding: the routing page's route-resolution order was a ul with the numbers hand-written into the item text. The invisible-marker era let it pass as a plain numbered list; restoring markers rendered a bullet stacked on each number. It is now a real ol with the hand numbers dropped, which is also what a screen reader or a crawler should have been handed all along. Swept the rest of the site for the same pattern; this was the only instance.
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Tenth pass, clean.
Confirmed the ol conversion on the live page: one ordered list, four decimal-numbered items, no hand-written numbers left, and a repo-wide sweep found no other page with the old pattern. Website suite green (103 server, 24 browser), the routing page renders with zero console errors, and the key routes (/docs redirect, /llms.txt, /sitemap.xml) all answer correctly.
Nothing left to flag. The two follow-ups on the board are #1102 (swapped-range script reactivation) and #1103 (the duplicate auth pages).
1 participant
Closes #1098
The documentation moves from
docs.webjs.devonto the marketing app atwebjs.dev/docs, rendering inside the site's own shell so the header, footer, theme toggle, fonts, and design tokens are literally the same code every other page uses. The only docs-specific chrome left is the page-tree sidebar.Two reasons. In search, a subdomain accrues authority to itself rather than to
webjs.dev, and the docs are the largest body of indexable content the project has (45 routes). In design, the docs carried a parallel layout that had already drifted from the marketing site, so a reader crossing over left one design system and entered another.docs.webjs.devbecomes a redirect-only service with path-preserving 301s, and it has to keep resolving indefinitely: framework error messages in already-published npm packages point at it, and a published version can never be corrected after the fact.What changed
docs/app/docs/**moved towebsite/app/docs/**(45 pages plus the sidebar layout, the search endpoint, the llms.txt routes)./docs308s to/docs/getting-startedrather than serving a second landing page at a near-duplicate URL..prose-docstypography, and docs-scoped metadata, and reads the root layout's design tokens; the docs@themeblock and its duplicate font and token declarations are deleted, not merged.--fg, no rules between sections, code in a--bg-subtlecard) instead of the old app's orange h1 and per-h2 rule. Only the nav scrolls, so the search box stays reachable on a long page.webjs.dev/llms.txtnow enumerates every doc page inline, in sidebar order derived from the sidebar itself, instead of pointing at a second index on another host./llms-full.txtand the per-page markdown routes came along, and both carryX-Robots-Tag: noindexso 45 full-text copies of the docs do not compete with the pages themselves on the domain this change exists to consolidate.docs.webjs.devserved no sitemap at all, which is why this migration was cheap.docs.webjs.devredirects path-preservingly with a 301, leaving/__webjs/*to the framework so the deploy healthcheck still passes.packages/core,packages/server,packages/cli(bin, generator, templates), the VS Code extension,AGENTS.md,README.md, the skill references, and the scaffold's per-agent rule files.Bugs found on the way
A root
middleware.tshas never been loaded.loadMiddlewarelooked up the single literalmiddleware.js, so a root middleware written in TypeScript was silently skipped: no error, no warning, indistinguishable from an app that has none.examples/blog/middleware.tsships in this repo, setsx-webjs: demoon every response, and that header has never been there. Fixed to resolve.ts/.js/.mts/.mjs, with a counterfactual and a cross-runtime proof. I did not work around it: shipping a workaround for a broken documented convention, in the PR about doc drift, is how drift gets normalized.The search endpoint rooted its index at
process.cwd()and imported its walker via../../../../packages/server/src/fs-walk.js, four levels out of the app into the framework. It now indexes offgetDocPages(), the same extraction the llms.txt routes use, so search and the corpus cannot disagree about a page title.doc-searchprobed awindow.navigateglobal the client router has never defined, so every search result did a full page reload.Also fixed: three internal docs links that had been 404ing (
/docs/route-handlers,/docs/caching,/docs/advanced), a dead#statefragment, and a counter sample on the progressive-enhancement page that rendered as@click=${'() => this.inc()'}, quoted, which is exactly what invariant 4 forbids, on the page teaching the model.Deliberately not in this PR
A top-level
<script>in a client-router swapped range is never reactivated, so a script emitted as a sibling of swapped content silently never runs. The docs hit this, and the first fix was on this branch. It is reverted and filed as #1102: nothing here depends on it any more (the highlighter moved to the root layout, which is never swapped), and it changes swap semantics, which AGENTS.md rule 1 says needs a browser or e2e assertion rather than the unit test it shipped with.packages/coreis down to one changed line on this branch, a docs URL in an error message./docs/authand/docs/authenticationship identical titles and contradict each other about whether the framework ships auth. Pre-existing content, but this PR is what starts submitting the pair to crawlers from a sitemap, so it is filed as #1103.Test plan
npm test3323 pass, 0 fail.webjs testinwebsite/102 pass, 0 fail. New:docs-chrome(header and footer compared byte for byte against a marketing page through the real request pipeline, the skip-link target, docs-scoped metadata, and the social-card fields a shallow merge would drop),docs-sitemap,docs-search,docs-links(every internal/docshref including the sidebar's, every doc page reachable from the nav, the llms ordering, the noindex set).web-test-runner24 pass, including a newdoc-searchsuite (debounce, endpoint URL, result rendering, empty state, click interception, supersede).WEBJS_E2E=178 pass, 0 fail.node scripts/run-bun-tests.js251 pass, 0 genuine failures. Newtest/bun/root-middleware.mjspasses under node 26 and bun 1.3, with its own CI step.app-boot.mjsprobes real docs routes on both runtimes (it was probing the redirect host's empty 301s, which pass vacuously).doc-searchat a different endpoint reds the browser test; restoring aprocess.cwd()root reds the search cwd test; a typo in a sidebar href and an unlinked doc page each red the link test; narrowing the dev-supervisor watch list reds the drift test.webjs createthen migrate, boot (/,/features/auth,/examples/todoall 200), andwebjs checkclean.webjs checkclean onwebsite/anddocs/;tsc --noEmitclean on both.Doc surfaces
Updated:
AGENTS.md,README.md,framework-dev.md,docs/AGENTS.md(rewritten for the redirect host),website/AGENTS.md,packages/server/AGENTS.md,packages/ui/packages/website/AGENTS.md, the skill (SKILL.mdplus therouting-and-pagesanddata-and-actionsreferences), the docs site's own middleware and architecture pages, the scaffold templates and generator,Dockerfile,compose.yaml,.github/workflows/ci.yml,scripts/dev-all.js, both asset generators,package-lock.json, and the.claudeskills and hooks that reference doc-surface paths.N/A: the MCP server (no introspection projection changed, and its knowledge layer reads
agent-docsandAGENTS.md, which are updated). The editor plugins (no intellisense, grammar, snippet, or command surface changed; the one VS Code docs URL is updated but the bundle is otherwise untouched).Follow-up, not in this PR
docs.webjs.devneeds its Railway service redeployed from this branch after merge, and the Cloudflare managed robots.txt block still needs turning off in the dashboard (tracked in #1088). The client-router script bug is #1102; the duplicate auth pages are #1103.