Bedetheque scraper

Python scraper for Bedetheque series pages with CSV/JSON exports, checkpoints, resume support, Docker packaging, and GHCR publishing.

Local usage

python scraper_bedetheque.py selftest
python scraper_bedetheque.py quick-test --max-series 5
python scraper_bedetheque.py scrape --letters A --max-series-per-letter 20
python scraper_bedetheque.py scrape

Outputs are written to output/ by default.

Docker

docker build -t bedetheque-scraper .
docker run --rm -v "$(pwd)/output:/data/output" bedetheque-scraper scrape

GHCR

Create and push the GitHub repository after authenticating gh:

gh auth login -h github.com
gh repo create Hitman47/bedetheque --public --source . --remote origin --push

The GitHub Actions workflow publishes the image on every push to main:

ghcr.io/<owner>/<repo>:latest
ghcr.io/<owner>/<repo>:<commit-sha>

For the default repository name planned here:

ghcr.io/hitman47/bedetheque:latest

If the package is private, log in on the NAS before pulling:

docker login ghcr.io

Portainer

Use portainer-stack.yml and set these variables if needed:

BEDETHEQUE_IMAGE=ghcr.io/hitman47/bedetheque:latest
GLUETUN_CONTAINER=gluetun
BEDETHEQUE_HOST_OUTPUT=/volume1/docker/bedetheque/output
BEDETHEQUE_DELAY_MIN=2.0
BEDETHEQUE_DELAY_MAX=5.0
BEDETHEQUE_CPUS=1.0
BEDETHEQUE_MEM_LIMIT=256m
BEDETHEQUE_MIN_DAYS_BETWEEN_FULL_EXPORTS=30
BEDETHEQUE_FORCE_SCRAPE=false

For a cautious first run:

BEDETHEQUE_LETTERS=A
BEDETHEQUE_MAX_SERIES_PER_LETTER=20