Harden package pages against scraper-driven heap exhaustion

CHANGELOG.md

@@ -5,13 +5,58 @@ the most up-to-date version of this file.
 
 ## Unreleased
 
+- Challenge browser-like scraper traffic to package pages with Anubis
+  (@thomashoneyman)
+
+  Distributed scraper fleets (thousands of IPs presenting spoofed browser
+  user agents, so robots.txt does not apply) crawl the module documentation
+  pages of large generated packages hard enough to take the server down.
+  Deploys now install [Anubis](https://github.com/TecharoHQ/anubis), a
+  challenge proxy, and nginx routes cache misses under `/packages/` through
+  it: clients with browser-like user agents must pass a JavaScript
+  proof-of-work challenge once, while non-browser clients (curl, package
+  uploads, editor tooling, badge fetchers) and known-good search engine
+  crawlers pass through untouched. Cached package pages, `/search`, and all
+  other routes bypass Anubis entirely.
+
+- Serve cached pages to browsers, and treat HEAD like GET (@thomashoneyman)
+
+  The nginx Accept-header map had no default, so the composite Accept
+  strings real browsers send ("text/html,application/xhtml+xml,...") never
+  matched the page cache, and every browser page view was re-rendered by the
+  backend. Cache lookups now default to the HTML representation, making
+  cache hits real for browsers - and confining the Anubis detour above to
+  genuine cache misses. HEAD requests now take the same cache/challenge path
+  as GET instead of going straight to the backend, where Yesod runs the full
+  GET handler for them - previously a full-cost decode that looked free.
+
+- Give large package decodes an aggregate memory budget (@thomashoneyman)
+
+  v0.9.11 serialised decodes of package docs JSON files of 5MB or more, but
+  concurrent decodes of files just below that cutoff could still exhaust the
+  heap: scrapers crawling the ~1,900 module pages of next-purs-rsc (a 4.4MB
+  file) repeatedly took the server down by stacking four or five decodes of
+  it at once. Decodes of files of 1MB or more now share a 16MB in-flight
+  budget instead: a decode starts once the total size of large files being
+  decoded fits within the budget (or immediately if it is the only large
+  decode running, so files bigger than the budget itself are still served).
+  The queue remains bounded, failing fast with a 503 when full, and the
+  search index regeneration remains exempt from the bound.
+
+  Budget admission is first-come-first-served: without an ordering, a file
+  too large to share the budget (which must wait until nothing else is in
+  flight) can be overtaken indefinitely by a steady stream of smaller
+  decodes - load testing showed a crawler burst starving the search index
+  build behind a react-icons decode for over an hour.
+
 - Type search queries containing type variables now also match more concrete
   types: `a -> HTMLElement` finds `HTMLAnchorElement -> HTMLElement` the same
   way `_ -> HTMLElement` does (#395). Unlike a wildcard, instantiating a query
   variable charges a small penalty, so results that unify with the query
   directly rank first, and repeated variables must be instantiated
   consistently (`a -> a` ranks `Int -> Int` above `Int -> String`). Based on
   #396 by @klntsky. (@thomashoneyman)
+
 - The package and module badges on search results are now links to the
   package page and module docs page (#424, @joprice). Builtin modules such
   as Prim have no package page, so their package badge remains plain text.

deploy/SERVER.md

@@ -2,7 +2,7 @@
 
 This file documents the state that lives only on the server or in the
 DigitalOcean account, which no deploy touches and which would otherwise be
-tribal knowledge. Last verified: 2026-07-05.
+tribal knowledge. Last verified: 2026-07-06.
 
 ## The droplet
 
@@ -30,6 +30,38 @@ tribal knowledge. Last verified: 2026-07-05.
 - **Cache eviction**: `/etc/cron.weekly/pursuit-cache-eviction` (installed by
   `remote.sh` from `deploy/cache-eviction.sh`) deletes cache files not
   accessed in 90 days so the cache cannot fill the disk.
+- **Anubis signing key**: `/etc/anubis/pursuit.env` is generated by
+  `remote.sh` on first deploy and then left alone; it contains the ed25519
+  key that signs challenge-pass cookies. Deleting or regenerating it just
+  makes every browser re-solve a challenge once.
+
+## Anubis (bot challenge proxy)
+
+Scraper fleets - thousands of IPs, spoofed browser user agents, immune to
+robots.txt - crawl the module docs pages of large generated packages hard
+enough to exhaust the backend's heap (2026-07-06: ~4,800 IPs crawling
+next-purs-rsc's ~1,900 module pages caused repeated heap-overflow restarts).
+[Anubis](https://github.com/TecharoHQ/anubis) counters exactly this: requests
+with browser-like user agents must pass a JavaScript proof-of-work challenge
+once (then carry a cookie for a week); non-browser clients (curl, `pursuit`
+uploads, editor tooling, GitHub's badge fetcher) and known-good search
+crawlers pass through untouched.
+
+- Deployed by `remote.sh`: pinned `.deb` from upstream releases, config in
+  `/etc/anubis/pursuit.env`, policy installed on every deploy from
+  `deploy/anubis.botPolicies.yaml` (a verbatim copy of the upstream default).
+- Runs as `anubis@pursuit.service` on `127.0.0.1:8923` (metrics on
+  `127.0.0.1:9823`), forwarding admitted traffic to the backend on `:3000`.
+- nginx routes only *cache misses under `/packages/`* (plus Anubis's own
+  `/.within.website/` endpoints) through it - the expensive decode-triggering
+  routes. Cache hits, `/search` (including the uptime check, which must see
+  the real backend), `POST /packages` uploads, and everything else go
+  straight to the backend. If Anubis is down, those cache misses return 502
+  (cached pages keep serving) - see the emergency bypass below.
+- **Emergency bypass**: if Anubis itself misbehaves, edit
+  `/etc/nginx/sites-enabled/pursuit.conf` to point the `@anubis` location's
+  `proxy_pass` at `http://127.0.0.1:3000`, then `systemctl reload nginx`.
+  (The next deploy restores the routing.)
 
 ## DigitalOcean account state
 
@@ -49,9 +81,11 @@ tribal knowledge. Last verified: 2026-07-05.
 The whole-package docs JSON is the unit of storage; rendering any single
 documentation page decodes the entire file for that package version, which
 transiently needs tens of times the file size in heap. A few generated
-packages (react-icons, elmish-html, deku, google-apps, ...) have 10-25MB
-files, so concurrent uncached requests for their pages can exhaust the heap.
-`Handler.Database.lookupPackage` therefore serialises decodes of files over
-5MB, and the search index is built one package at a time
+packages (react-icons, elmish-html, deku, next-purs-rsc, google-apps, ...)
+have 4-25MB files, so concurrent uncached requests for their pages can
+exhaust the heap. `Handler.Database.lookupPackage` therefore admits decodes
+of files of 1MB or more against a shared 16MB in-flight budget, and the
+search index is built one package at a time
 (`createSearchIndexFromDatabase`). If heap-exhaustion restarts reappear in
-the journal (`journalctl -u pursuit | grep "Heap exhausted"`), start there.
+the journal (`journalctl -u pursuit | grep -E "Heap exhausted|status=251"`),
+start there.

deploy/anubis.botPolicies.yaml

@@ -0,0 +1,247 @@
+## Bot policy for the Anubis instance protecting pursuit.purescript.org.
+##
+## This is a verbatim copy of the default policy shipped with Anubis v1.25.0
+## (https://github.com/TecharoHQ/anubis/blob/v1.25.0/data/botPolicies.yaml),
+## kept in the repo so deploys are reproducible and changes are reviewable.
+## When upgrading the pinned Anubis version in remote.sh, diff this file
+## against the new release's default and fold in upstream changes.
+##
+## remote.sh installs this to /etc/anubis/pursuit.botPolicies.yaml on every
+## deploy; local edits on the server will be overwritten.
+
+## Anubis has the ability to let you import snippets of configuration into the main
+## configuration file. This allows you to break up your config into smaller parts
+## that get logically assembled into one big file.
+##
+## Of note, a bot rule can either have inline bot configuration or import a
+## bot config snippet. You cannot do both in a single bot rule.
+##
+## Import paths can either be prefixed with (data) to import from the common/shared
+## rules in the data folder in the Anubis source tree or will point to absolute/relative
+## paths in your filesystem. If you don't have access to the Anubis source tree, check
+## /usr/share/docs/anubis/data or in the tarball you extracted Anubis from.
+
+bots:
+  # You can import the entire default config with this macro:
+  # - import: (data)/meta/default-config.yaml
+
+  # Pathological bots to deny
+  - # This correlates to data/bots/_deny-pathological.yaml in the source tree
+    # https://github.com/TecharoHQ/anubis/blob/main/data/bots/_deny-pathological.yaml
+    import: (data)/bots/_deny-pathological.yaml
+  - import: (data)/bots/aggressive-brazilian-scrapers.yaml
+
+  # Aggressively block AI/LLM related bots/agents by default
+  - import: (data)/meta/ai-block-aggressive.yaml
+
+  # Consider replacing the aggressive AI policy with more selective policies:
+  # - import: (data)/meta/ai-block-moderate.yaml
+  # - import: (data)/meta/ai-block-permissive.yaml
+
+  # Search engine crawlers to allow, defaults to:
+  #   - Google (so they don't try to bypass Anubis)
+  #   - Apple
+  #   - Bing
+  #   - DuckDuckGo
+  #   - Qwant
+  #   - The Internet Archive
+  #   - Kagi
+  #   - Marginalia
+  #   - Mojeek
+  - import: (data)/crawlers/_allow-good.yaml
+  # Challenge Firefox AI previews
+  - import: (data)/clients/x-firefox-ai.yaml
+
+  # Allow common "keeping the internet working" routes (well-known, favicon, robots.txt)
+  - import: (data)/common/keep-internet-working.yaml
+
+  # # Punish any bot with "bot" in the user-agent string
+  # # This is known to have a high false-positive rate, use at your own risk
+  # - name: generic-bot-catchall
+  #   user_agent_regex: (?i:bot|crawler)
+  #   action: CHALLENGE
+  #   challenge:
+  #     difficulty: 16 # impossible
+  #     algorithm: slow # intentionally waste CPU cycles and time
+
+  # Requires a subscription to Thoth to use, see
+  # https://anubis.techaro.lol/docs/admin/thoth#geoip-based-filtering
+  - name: countries-with-aggressive-scrapers
+    action: WEIGH
+    geoip:
+      countries:
+        - BR
+        - CN
+    weight:
+      adjust: 10
+
+  # Requires a subscription to Thoth to use, see
+  # https://anubis.techaro.lol/docs/admin/thoth#asn-based-filtering
+  - name: aggressive-asns-without-functional-abuse-contact
+    action: WEIGH
+    asns:
+      match:
+        - 13335 # Cloudflare
+        - 136907 # Huawei Cloud
+        - 45102 # Alibaba Cloud
+    weight:
+      adjust: 10
+
+  # ## System load based checks.
+  # # If the system is under high load, add weight.
+  # - name: high-load-average
+  #   action: WEIGH
+  #   expression: load_1m >= 10.0 # make sure to end the load comparison in a .0
+  #   weight:
+  #     adjust: 20
+
+  ## If your backend service is running on the same operating system as Anubis,
+  ## you can uncomment this rule to make the challenge easier when the system is
+  ## under low load.
+  ##
+  ## If it is not, remove weight.
+  # - name: low-load-average
+  #   action: WEIGH
+  #   expression: load_15m <= 4.0 # make sure to end the load comparison in a .0
+  #   weight:
+  #     adjust: -10
+
+  # Generic catchall rule
+  - name: generic-browser
+    user_agent_regex: >-
+      Mozilla|Opera
+    action: WEIGH
+    weight:
+      adjust: 10
+
+dnsbl: false
+
+# #
+# impressum:
+#   # Displayed at the bottom of every page rendered by Anubis.
+#   footer: >-
+#     This website is hosted by Zombocom. If you have any complaints or notes
+#     about the service, please contact
+#     <a href="mailto:contact@domainhere.example">contact@domainhere.example</a>
+#     and we will assist you as soon as possible.
+
+#   # The imprint page that will be linked to at the footer of every Anubis page.
+#   page:
+#     # The HTML <title> of the page
+#     title: Imprint and Privacy Policy
+#     # The HTML contents of the page. The exact contents of this page can
+#     # and will vary by locale. Please consult with a lawyer if you are not
+#     # sure what to put here
+#     body: >-
+#       <p>Last updated: June 2025</p>
+
+#       <h2>Information that is gathered from visitors</h2>
+
+#       <p>In common with other websites, log files are stored on the web server saving details such as the visitor's IP address, browser type, referring page and time of visit.</p>
+
+#       <p>Cookies may be used to remember visitor preferences when interacting with the website.</p>
+
+#       <p>Where registration is required, the visitor's email and a username will be stored on the server.</p>
+
+#       <!-- ... -->
+
+# Open Graph passthrough configuration, see here for more information:
+# https://anubis.techaro.lol/docs/admin/configuration/open-graph/
+openGraph:
+  # Enables Open Graph passthrough
+  enabled: false
+  # Enables the use of the HTTP host in the cache key, this enables
+  # caching metadata for multiple http hosts at once.
+  considerHost: false
+  # How long cached OpenGraph metadata should last in memory
+  ttl: 24h
+  # # If set, return these opengraph values instead of looking them up with
+  # # the target service.
+  # #
+  # # Correlates to properties in https://ogp.me/
+  # override:
+  #   # og:title is required, it is the title of the website
+  #   "og:title": "Techaro Anubis"
+  #   "og:description": >-
+  #     Anubis is a Web AI Firewall Utility that helps you fight the bots
+  #     away so that you can maintain uptime at work!
+  #   "description": >-
+  #     Anubis is a Web AI Firewall Utility that helps you fight the bots
+  #     away so that you can maintain uptime at work!
+
+# By default, send HTTP 200 back to clients that either get issued a challenge
+# or a denial. This seems weird, but this is load-bearing due to the fact that
+# the most aggressive scraper bots seem to really, really, want an HTTP 200 and
+# will stop sending requests once they get it.
+status_codes:
+  CHALLENGE: 200
+  DENY: 200
+
+# Anubis can store temporary data in one of a few backends. See the storage
+# backends section of the docs for more information:
+#
+# https://anubis.techaro.lol/docs/admin/policies#storage-backends
+store:
+  backend: memory
+  parameters: {}
+
+# The weight thresholds for when to trigger individual challenges. Any
+# CHALLENGE will take precedence over this.
+#
+# A threshold has four configuration options:
+#
+#   - name: the name that is reported down the stack and used for metrics
+#   - expression: A CEL expression with the request weight in the variable
+#     weight
+#   - action: the Anubis action to apply, similar to in a bot policy
+#   - challenge: which challenge to send to the user, similar to in a bot policy
+#
+# See https://anubis.techaro.lol/docs/admin/configuration/thresholds for more
+# information.
+thresholds:
+  # By default Anubis ships with the following thresholds:
+  - name: minimal-suspicion # This client is likely fine, its soul is lighter than a feather
+    expression: weight <= 0 # a feather weighs zero units
+    action: ALLOW # Allow the traffic through
+  # For clients that had some weight reduced through custom rules, give them a
+  # lightweight challenge.
+  - name: mild-suspicion
+    expression:
+      all:
+        - weight > 0
+        - weight < 10
+    action: CHALLENGE
+    challenge:
+      # https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh
+      algorithm: metarefresh
+      difficulty: 1
+  # For clients that are browser-like but have either gained points from custom rules or
+  # report as a standard browser.
+  - name: moderate-suspicion
+    expression:
+      all:
+        - weight >= 10
+        - weight < 20
+    action: CHALLENGE
+    challenge:
+      # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
+      algorithm: fast
+      difficulty: 2 # two leading zeros, very fast for most clients
+  - name: mild-proof-of-work
+    expression:
+      all:
+        - weight >= 20
+        - weight < 30
+    action: CHALLENGE
+    challenge:
+      # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
+      algorithm: fast
+      difficulty: 4
+  # For clients that are browser like and have gained many points from custom rules
+  - name: extreme-suspicion
+    expression: weight >= 30
+    action: CHALLENGE
+    challenge:
+      # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
+      algorithm: fast
+      difficulty: 6