fix(server): noindex the app shell and serve a real robots.txt

[nachocossio]

Problem

app.mcpjam.com's SPA catch-all returns index.html with a 200 for every path — including /robots.txt. Crawlers requesting it get unparseable HTML, which robots parsers treat as allow-all, so the authenticated app shell is fully crawlable and indexable (robots.txt is per-host; the marketing site's file doesn't cover this origin).

Verified on prod before the fix:

$ curl -s -o /dev/null -w "%{http_code} %{content_type}" https://app.mcpjam.com/robots.txt
200 text/html; charset=UTF-8        ← HTML, not robots.txt

No X-Robots-Tag or <meta robots> anywhere.

Fix — policy: allow crawling + noindex

• client/public/robots.txt (new): valid, parseable robots.txt that allows crawling. It rides the same Vite publicDir pipeline as the favicon into dist/client/, where serveStatic (server/index.ts:563) wins over the SPA catch-all (:566).
• X-Robots-Tag: noindex on every response, added to the existing securityHeadersMiddleware (applies globally in both server/index.ts and the server/app.ts mirror).

Why not Disallow: /: a Disallow blinds robots to the noindex directive — Google can still index the URLs from external links (ugly, empty results). Allow + noindex is the reliable way to keep the shell out of search indexes. AI agents are unaffected (noindex only governs search indexing, not fetching).

Tests

Red→green: new server/middleware/__tests__/security-headers.test.ts (mirrors the existing middleware test style) — baseline case for the 4 existing headers passed against the old code, the 2 X-Robots-Tag cases failed (Received: null), all 3 pass with the change.

Refs ARM-16 (Agent Readiness board).

---

Summary by cubic

Serve a real robots.txt and add a global X-Robots-Tag: noindex to keep the authenticated app shell off search results while still allowing crawlers to fetch pages. Addresses ARM-16.

• Bug Fixes
  • Serve client/public/robots.txt so /robots.txt returns a valid file with Allow: /.
  • Add X-Robots-Tag: noindex in securityHeadersMiddleware for all routes (HTML and API).
  • Add tests covering the security headers, including X-Robots-Tag.

^{Written for commit cd4fb05. Summary will update on new commits.}

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[chelojimenez]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3dab7861-c34d-42c3-8beb-fe00d38cc7e8

📥 Commits

Reviewing files that changed from the base of the PR and between 659a60c and cd4fb05.

📒 Files selected for processing (3)

• mcpjam-inspector/client/public/robots.txt
• mcpjam-inspector/server/middleware/__tests__/security-headers.test.ts
• mcpjam-inspector/server/middleware/security-headers.ts

---

Walkthrough

Added crawler rules for the app shell and configured the security middleware to emit X-Robots-Tag: noindex alongside existing security headers. Added Vitest coverage using representative HTML and API routes to verify the headers are applied consistently.

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint install timed out. The project may have too many dependencies for the sandbox.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Internal preview

Preview URL: https://mcp-inspector-pr-3336.up.railway.app
Deployed commit: f645f5d
PR head commit: cd4fb05
Backend target: staging fallback.
Health: ✅ Convex reachable
Access is employee-only in non-production environments.

[cubic-dev-ai]

No issues found across 3 files

_{Re-trigger cubic}