fix(compose): force api build to avoid stale published image

[colinwilliams91]

What & why

Fixes #48.

docker compose up -d on a fresh clone pulled the stale published trieve/firecrawl:latest image (built 2024-09-06) for the api service, which crashed on startup with Cannot find matching keyid because the bundled corepack predates npm's registry signing-key rotation. The worker service — which has no image: key — always built locally, so the two services ended up running different images, and api never came up. docker compose build && docker compose up -d worked because that path forces a local build.

Root cause: docker-compose.yaml declared both image: trieve/firecrawl and (via the x-common-service anchor) build: apps/api on the api service. With Compose's default pull_policy, docker compose up pulls the named image instead of building it.

Changes

• docker-compose.yaml — add pull_policy: build to the api service. This keeps the image: key (which is still useful as the tag a future publish workflow would produce) but forces Compose to always build from apps/api/Dockerfile and never pull the registry artifact. The fix mirrors what the worker service already does implicitly (it has no image: key, so it only ever builds locally). Scope is intentionally limited to api; puppeteer-service has the same image: + build: pattern but is left untouched per the scoped fix — flagging it here as a known follow-up.
• README.md — the self-host snippet recommended pulling the pinned published images (trieve/firecrawl:v0.0.46, trieve/puppeteer-service-ts:v0.0.6), which are also stale and fail with the same corepack error. Added a prominent warning above the snippet explaining the images are stale, that there is no release-triggered publish workflow in this repo, and that the supported path today is to clone and run docker compose up -d from the repo root (which now uses pull_policy: build).
• SELF_HOST.md — updated step 4 to reflect that docker compose up -d is now sufficient (no separate build step needed) because the api service sets pull_policy: build, and linked to issue [Bug] docker compose up pulls a 2024 published image that crashes on startup (corepack signature failure) #48.

Verification

docker compose config parses cleanly and renders pull_policy: build on the api service:

  api:
    build:
      context: .../apps/api
      dockerfile: Dockerfile
    image: trieve/firecrawl
    pull_policy: build      # <-- now set
    ...

(Any variable is not set warnings are just because no .env file exists in this scratch checkout; they are pre-existing and unrelated.)

Out of scope, but worth flagging

• puppeteer-service (docker-compose.yaml:24) has the same image: + build: trap. Left alone here per the scoped fix; happy to follow up.
• apps/puppeteer-service-ts/Dockerfile is malformed — two concatenated FROM node:18-slim stages (lines 1-21 and 22-70), so only the second stage takes effect. Probably worth a follow-up issue.
• Two scheduled cron workflows are failing on every run. .github/workflows/check-queues.yml (every 5 min) and .github/workflows/clean-before-24h-complete-jobs.yml (every 30 min) have been failing continuously — at least the last 24h, every single run. They curl https://api.firecrawl.dev/admin/${BULL_AUTH_KEY}/..., which suggests BULL_AUTH_KEY is not configured in the devflowinc/firecrawl-simple repo (or the endpoint has moved). This is unrelated to [Bug] docker compose up pulls a 2024 published image that crashes on startup (corepack signature failure) #48 but produces constant CI noise; the maintainers may want to either set the secret / update the endpoint, or archive these workflows (matching the existing .github/archive/ pattern used for the SDK workflows).
• No publish workflow exists. .github/workflows/build-docker-images.yml only runs on PRs with push: false and pr-<number> tags — there is no release-triggered job to publish trieve/firecrawl to Docker Hub, which is why :latest and the pinned tags drifted in the first place. The pull_policy: build fix sidesteps this for self-hosters, but a publish workflow would be a cleaner long-term answer.

Closes #48.

[colinwilliams91]

Follow-up: the failing cron workflows flagged in this PR's "Out of scope" section now have their own PR -- see #52 (tracking issue #50, plus a separate follow-up issue #51 for the stale SELF_HOST.md:86 "Bull Queue Manager UI" reference).

The fix moves both crons into .github/archive/ with on: [] (matching the existing archive convention) and also removes their two matching controllers (checkQueuesController, cleanBefore24hCompleteJobsController) plus the now-dead apps/api/src/services/alerts/ directory, mirroring upstream firecrawl/firecrawl which has fully removed all of these. BULL_AUTH_KEY and the redis-health / queues admin endpoints stay for self-hosters. Verified via pnpm run build (tsc passes clean) and pnpm test (no test references any of the removed symbols).