Skip to content

RELEASE_CHECKLIST.md

Latest commit

 

History

History
216 lines (190 loc) · 13.3 KB

RELEASE_CHECKLIST.md

File metadata and controls

216 lines (190 loc) · 13.3 KB
Raw

HyperTools 1.0 release checklist

The docs, notebooks, and README deliberately ship in dev form on the dev-1.0 branch and must flip to release form at publish. Several of those flips cannot be done earlier (PyPI does not yet have 1.0; the v1.0.0 tag does not exist yet), so they are done here, on master, in order — and the release-gate CI job (runs only on master / tags) hard-fails until every flip is done, so nothing can be forgotten.

Run everything from a clean master checkout on the master branch (not a detached tag checkout — the notebook migrator detects the branch via git rev-parse --abbrev-ref HEAD, which returns HEAD when detached).

0. Pre-flight (on dev-1.0)

  • dev-1.0 CI fully green (push + PR workflows).
  • Full suite green locally: pytest (2470+ passed, 0 failed).
  • Decide the release date and the version (1.0.0).
  • conda-forge prerequisite (GH #282). pydata-wrangler is merged and available on conda-forge: conda search -c conda-forge 'pydata-wrangler>=0.5.1' returns a hit. The hypertools feedstock (step 7) lists it as a runtime dependency, and conda-forge forbids pip-only run deps — so hold the release until it lands. (Submitted to conda-forge/staged-recipes; waiting on merge.)

1. Merge to master

  • Merge dev-1.0 → master (PR #281). Do NOT delete dev-1.0 yet (the pre-release notebooks still reference it until step 2 runs).

2. Flip everything to release form (on master)

  • Notebooks → PyPI spec + clean note (automated). python scripts/add_colab_install_cell.py Retargets all 15 committed tutorial install cells ... @ git+…@dev-1.0 → hypertools[<extras>] (extras preserved) and strips the (<branch> preview) / "On release this becomes …" note. The gallery (docs/auto_examples/*.ipynb) is gitignored and REGENERATED by docs/conf.py on each build, which emits the identical PyPI line on a master/tag build (the migrator only retargets any on-disk copy, and the docs-clean CI job release-gates the generated gallery).
  • README images: commit SHA → v1.0.0 tag (8 URLs). sed -E -i '' 's#(/ContextLab/hypertools/)[^/]+(/images/)#\1v1.0.0\2#g' readme.md (drop the '' after -i on GNU sed). This retargets WHATEVER ref is pinned (robust to the SHA having drifted), not just fc2429cb. Verify the POSITIVE: grep -c '/ContextLab/hypertools/v1.0.0/images/' readme.md → 8 and grep -Ec '/ContextLab/hypertools/[0-9a-f]{7,40}/images/' readme.md → 0.
  • CHANGELOG date. Edit CHANGELOG.md: ## 1.0.0 (unreleased) → ## 1.0.0 (YYYY-MM-DD) with the real release date.
  • (Optional prose) docs/tutorials/stock_forecasting.ipynb has a free-text "hypertools 1.0 preview" comment the migrator does not touch — reword if desired (not gate-enforced).
  • Verify the file-content gates locally BEFORE committing. Exclude the gallery-resolve gate — it checks the remote docs-notebooks branch, published in the next step, so it cannot pass yet: HYPERTOOLS_REQUIRE_RELEASE=1 pytest -v tests/test_notebook_install_gate.py tests/test_release_readiness_gate.py -k 'not gallery_colab_notebooks_are_published' → all green (no branch installs, no preview note, images on the tag, CHANGELOG dated).
  • Commit all of the above on master in one release commit.
  • Publish the gallery notebooks NOW — before any release gate needs them (this is what breaks the publish-order deadlock). The gallery "Open in Colab" badges point at blob/docs-notebooks/v1.0.0/auto_examples/<stem>.ipynb, and the release-gate job runs on BOTH the master push (step 4) and the v1.0.0 tag (step 5) — it fails until those notebooks exist, so they must be published before you push, not after. Build the gallery and publish: cd docs && make html then, from the repo root, python scripts/publish_gallery_notebooks.py --ref v1.0.0 --notebooks-dir docs/auto_examples --push (creates the docs-notebooks orphan branch on first run — one-time bootstrap — and also writes v1.0.0/manifest.json). Publishing static notebooks before PyPI is harmless: their %pip install hypertools[...] cells resolve 1.0 the moment PyPI is updated (step 6). Both the master "latest" docs and the v1.0.0 "stable" docs resolve to this ONE v1.0.0 namespace (docs/post_build.py _publish_ref), so this single publish covers both.
  • Verify the gallery gate now resolves: HYPERTOOLS_REQUIRE_RELEASE=1 pytest tests/test_release_readiness_gate.py::test_release_gate_gallery_colab_notebooks_are_published → green. This gate ties the published notebooks to THIS release: the manifest.json's source_commit must equal git rev-parse HEAD, its inventory must EXACTLY match the gallery you just built, and the docs-notebooks branch's actual .ipynb set (read once via the GitHub tree API) must equal the manifest. Publish from the release commit and do NOT commit afterward — any later commit changes HEAD, so you must rebuild the gallery and re-publish (the gate fails until you do). The same gate re-runs on the master push (step 4) and the tag (step 5).

3. Build + verify artifacts locally (do NOT upload yet)

The PyPI upload is irreversible, so it happens LAST — only after the exact release commit AND the tag have passed hosted CI (step 5/6). Build now so the same artifacts you verify are the ones you publish.

  • Record the build commit: git rev-parse HEAD (must equal the tagged commit before you upload in step 6).
  • Build from a CLEAN dist: rm -rf dist && python -m build → dist/hypertools-1.0.0.tar.gz + …-py3-none-any.whl.
  • twine check dist/* → PASSED.
  • Artifacts bundle the fonts + all license materials (font OFL, Apache-2.0 license + third-party notices for the vendored brainiak/ppca, CHANGELOG): tar tzf dist/*.tar.gz | grep -E 'NotoSans|OFL|LICENSE-APACHE|THIRD_PARTY|CHANGELOG' (5+ hits) and the same on the wheel via unzip -l dist/*.whl.
  • Fresh-venv smoke: install the wheel in a throwaway venv, import hypertools, hypertools.__version__ == '1.0.0'.
  • Record artifact digests: shasum -a 256 dist/* (keep with the build commit; verify these exact files are the ones uploaded in step 6).
  • Keep dist/ — you upload these exact files in step 6.

4. Push master + wait for its CI

  • git push origin master.
  • Wait for ALL master CI to go GREEN: the test matrix, wheel-smoke, docs-clean (incl. the generated-gallery release scan), dataset-gate, and release-gate (which now actually runs — the workflow triggers on master). This is the release-form CI on the exact commit you will tag.

5. Tag the green commit + wait for tag CI

  • git tag -a v1.0.0 -m "HyperTools 1.0.0" at the exact commit that just went green on master.
  • git push origin v1.0.0 (the workflow's tags: ['v*'] trigger runs CI on the tag).
  • Wait for the v1.0.0 tag CI to go GREEN (same jobs; release-gate + docs-clean gallery scan run on the tag too).

6. Publish to PyPI (the already-verified artifacts) + smoke

  • Confirm the tag points at the recorded build commit: git rev-parse v1.0.0^{commit} == the step-3 build commit, and shasum -a 256 dist/* == the step-3 digests. Publish ONLY if both match.
  • Prefer PyPI Trusted Publishing (OIDC) via a protected GitHub environment over a long-lived token. If uploading manually: use a project-scoped API token, an account with 2FA, and reviewed owner/recovery access.
  • Upload the EXACT files by name (not a dist/* glob, which could sweep a stale artifact): twine upload dist/hypertools-1.0.0.tar.gz dist/hypertools-1.0.0-py3-none-any.whl. (The static notebooks briefly resolving the previous PyPI release before this upload is harmless.)
  • Create a GitHub Release for the v1.0.0 tag with the 1.0 release notes (from CHANGELOG.md).
  • pip install hypertools in a clean env → installs 1.0.0; run the README quick-start snippet.
  • Gallery notebooks: confirm still resolved. They were already published to docs-notebooks/v1.0.0/ in step 2 (before the gates), so the Colab badges resolve; only re-run this if you rebuilt the gallery since: HYPERTOOLS_REQUIRE_RELEASE=1 pytest tests/test_release_readiness_gate.py::test_release_gate_gallery_colab_notebooks_are_published. (Publication is a MANUAL step today — there is no CI job for it; a contents: write publish-gallery-notebooks job on master/tags could automate it once token/environment handling is decided.)
  • Read the Docs: trigger/confirm a build of the v1.0.0 tag (and point the "stable"/default version at it). The released docs' Colab install cells must show %pip install "hypertools[interactive]" (no git+) — docs/conf.py emits this automatically on a tag build.
  • PyPI project page renders the README with all 8 images resolving (they now point at the v1.0.0 tag).

7. Publish to conda-forge (GH #282)

conda-forge builds from the PyPI sdist, so this runs AFTER the PyPI upload (step 6). hypertools is pure Python with no console-scripts, so it builds as a single noarch: python package. Its pydata-wrangler runtime dep must already be on conda-forge (the step-0 gate) — conda-forge forbids pip-only run deps.

  • Generate the recipe from the published sdist: grayskull pypi hypertools==1.0.0 (grayskull fetches the sdist, computes its SHA-256, and maps most deps automatically).
  • Hand-fix the generated recipe/meta.yaml: - matplotlib → matplotlib-base (conda-forge convention: avoids the Qt pull-in). - build: → noarch: python; host: = python >=3.10, pip, setuptools >=77. - carry the pyproject floors into run: (python >=3.10, numpy >=2.0, scikit-learn >=1.4.2, pandas >=2.2.2, matplotlib-base >=3.9.0, numba >=0.61.0, pydata-wrangler >=0.5.1, …). - license: MIT and list EVERY bundled license in license_file: LICENSE, hypertools/external/LICENSE-APACHE-2.0.txt, hypertools/external/fonts/OFL.txt (conda-forge review checks these). - test: → imports: hypertools + commands: pip check. - do NOT add the [predict] / [predict-hf] / [lsl] extras: skaters, chronos-forecasting, and pylsl are not on conda-forge, so they stay pip-only (note this in the recipe PR; the base package is unaffected). The rest of the extras map fine (kaleido → python-kaleido, torch → pytorch; plotly / scikit-image / kagglehub / gensim are all on conda-forge).
  • Open a PR to conda-forge/staged-recipes adding recipes/hypertools/meta.yaml; list the lab maintainers under recipe-maintainers. CI lints + test-builds; a conda-forge core member reviews and merges.
  • After merge, the bot auto-creates conda-forge/hypertools-feedstock, builds, and uploads. Verify in a clean env: conda install -c conda-forge hypertools installs 1.0.0 and import hypertools works.
  • (Ongoing) the conda-forge bot opens version-bump PRs automatically on each future PyPI release; maintainers just review/merge.

8. Cleanup

  • After the release is confirmed good, delete the dev-1.0 / dev-1.0-refactor branches if desired (the released artifacts no longer reference them; the release-gate guarantees this).

What the release-gate enforces (so you can't forget)

tests/test_notebook_install_gate.py + tests/test_release_readiness_gate.py, run with HYPERTOOLS_REQUIRE_RELEASE=1 by the release-gate CI job on master/tags, fail if ANY of these survive:

Check Release form required
published notebook install cells hypertools[…] PyPI spec (no git+/@<branch>)
notebook install-cell note no (… preview) / "On release this becomes …"
README image URLs …/ContextLab/hypertools/v<version>/images/… — the tag EXACTLY equal to v + pyproject version, not any semver or a commit SHA
README branch refs no dev-1.0-refactor / hypertools.git@dev…
CHANGELOG heading ## <version> (YYYY-MM-DD) — version == pyproject, and a REAL calendar date (not (unreleased), not 2026-99-99)
generated gallery (docs-clean job) every built docs/auto_examples/*.ipynb carries the PyPI spec (covers all 68 published notebooks, at the build layer)
gallery Colab notebooks published docs-notebooks/v<version>/manifest.json present, its source_commit == the release HEAD, its inventory == the built gallery, and the branch's actual .ipynb set (one GitHub tree request) == the manifest — so stale (old-RC), partial, or mismatched publishes all fail. Requires step 2's publish to have run FROM the release commit, BEFORE the master/tag push — see the deadlock note there.

Always-on (every branch): no notebook installs the defunct dev-1.0-refactor; all tutorial branch-installs share one branch; every README image is a single consistent ref and exists in the tree; CHANGELOG top version == pyproject.