ci: unified release_pypi.yml — stable to PyPI on tag, nightly .devN to anaconda.org on master push#866
Open
IvanaGyro wants to merge 5 commits into
Open
ci: unified release_pypi.yml — stable to PyPI on tag, nightly .devN to anaconda.org on master push#866IvanaGyro wants to merge 5 commits into
IvanaGyro wants to merge 5 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new script, tools/prepare_nightly_release.py, which automates updating pyproject.toml for nightly releases by parsing version information from version.cmake and appending a UTC timestamp. The code review feedback focuses on making the script more robust against formatting changes. Specifically, the reviewer suggests parsing version components individually rather than assuming they are adjacent, using flexible regular expressions instead of exact string replacement for updating package metadata, and refining the regex used to strip the scikit-build metadata block to prevent premature matching.
IvanaGyro
commented
May 31, 2026
IvanaGyro
force-pushed
the
release-nightly-and-prod-pypi
branch
from
5db13b9 to
f5dcea7
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #866 +/- ##
==========================================
+ Coverage 29.05% 29.49% +0.44%
==========================================
Files 241 241
Lines 35519 35524 +5
Branches 14807 14780 -27
==========================================
+ Hits 10319 10477 +158
+ Misses 18039 17791 -248
- Partials 7161 7256 +95
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
IvanaGyro
force-pushed
the
release-nightly-and-prod-pypi
branch
from
f5dcea7 to
80fa62c
Compare
IvanaGyro
changed the title
IvanaGyro
commented
May 31, 2026
IvanaGyro
commented
May 31, 2026
Two related cleanups to dependency declarations in pyproject.toml:
1. The `dev` aggregate under `[project.optional-dependencies]`
previously duplicated the contents of the `test` and `coverage`
groups verbatim, so any change to either leaf group required a
second edit to keep `dev` in sync. Replace the duplicated dep
list with self-references:
dev = ["cytnx[test]", "cytnx[coverage]"]
`pip install --editable .[dev]` resolves the references and pulls
the same packages as before, but the source of truth for each
group is now its own definition.
2. Add a new PEP 735 `[dependency-groups]` table with a
`release-tools` group containing `tomlkit`. The group is consumed
by release pipelines that programmatically rewrite pyproject.toml
(e.g. tools/prepare_nightly_release.py).
`[dependency-groups]` is used rather than
`[project.optional-dependencies]` because tools listed here are
needed at *build pipeline* time only, not at install or run time
of cytnx itself. `pip install --group release-tools` installs the
listed packages without invoking the project build backend, so a
release pipeline can install the helpers on a runner without
accidentally triggering a scikit-build-core compile of cytnx
before cibuildwheel runs in its own isolated environment.
Co-Authored-By: Claude <noreply@anthropic.com>
`set_target_properties(VERSION ...)`, `project(VERSION ...)`, and the shared-library SONAME require a strict MAJOR.MINOR.PATCH, so CYTNX_VERSION cannot itself carry a PEP 440 dev/local suffix. Introduce a separate string `CYTNX_VERSION_FULL`, initialised to the numeric `CYTNX_VERSION` and extended with the contents of the `CYTNX_VERSION_TAG` environment variable when that variable is set and non-empty. Use `CYTNX_VERSION_FULL` for the `CYTNX_VERSION` compile definition consumed by pybind, which becomes the runtime `cytnx.__version__`. Numeric-only consumers (project version, target VERSION/SOVERSION, libname suffix) keep reading `CYTNX_VERSION`. With no environment variable set, `CYTNX_VERSION_FULL` and `CYTNX_VERSION` are identical, so non-release builds are unaffected and `pytests/version_test.py` continues to compare `cytnx.__version__` against the numeric version.cmake values without change. Also append `CYTNX_VERSION_TAG` to the cibuildwheel Linux `environment-pass` list so the variable, when set on the host runner, is forwarded into the manylinux container that performs the actual wheel compilation. On macOS the variable is inherited from the runner environment directly and needs no allow-list entry. The variable itself is never set by a normal contributor or by any existing CI job in this commit; nothing downstream consumes it yet. A subsequent commit adds the release tooling that produces the suffix. Co-Authored-By: Claude <noreply@anthropic.com>
IvanaGyro
force-pushed
the
release-nightly-and-prod-pypi
branch
from
1c0abab to
d91029c
Compare
Collaborator
Author
|
I plan to follow SPEC 4 to release the nightly build. This is adopted by numpy, scipy and some scientific Python packages. I am requesting access of Scientific Python Nightly Wheels to release the nightly build on scientific-python/upload-nightly-action#167. We will not merge this PR until we get access. |
IvanaGyro
force-pushed
the
release-nightly-and-prod-pypi
branch
from
d91029c to
38a7dd9
Compare
IvanaGyro
changed the title
IvanaGyro
force-pushed
the
release-nightly-and-prod-pypi
branch
from
38a7dd9 to
75913cf
Compare
IvanaGyro
changed the title
IvanaGyro
force-pushed
the
release-nightly-and-prod-pypi
branch
from
75913cf to
69559a6
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69559a69d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Collaborator
Author
They rejected our application, so I decide to publish to a new organization. |
Add `tools/prepare_nightly_release.py`, a single-shot script intended
to run in CI before cibuildwheel on every push to `master`. It
- reuses the regex declared in
`[tool.scikit-build.metadata.version]` of pyproject.toml as the
sole parser for version.cmake, so the two never disagree on what
"the version" is;
- derives a PEP 440 dev version of the form
MAJOR.MINOR.PATCH.devYYYYMMDDHHMM (UTC), giving each push a
monotonically increasing, deterministic identifier;
- rewrites pyproject.toml in place to a static `version = "..."`
on the `cytnx` project, removing the `dynamic = ["version"]`
entry from `[project]` and the now-redundant
`[tool.scikit-build.metadata.version]` table; and
- appends `CYTNX_VERSION_TAG=.devYYYYMMDDHHMM` to `$GITHUB_ENV`
when running under GitHub Actions, so the surrounding CI job can
forward the suffix into the cibuildwheel build via the
`CYTNX_VERSION_TAG` hook in CMakeLists.txt, keeping the wheel
filename's version string and `cytnx.__version__` in lockstep.
The rewrite uses `tomlkit` (declared in the `release-tools`
dependency-group) for round-trip formatting preservation, so the
comments and section ordering of pyproject.toml survive the stamping
intact.
The script is idempotent only against a clean checkout; CI is
expected to run it once on a fresh tree before cibuildwheel.
No workflow invokes the script yet; the nightly publishing workflow
is added in a follow-up commit so that this commit can be reviewed
on its own.
Co-Authored-By: Claude <noreply@anthropic.com>
The workflow previously published a non-version-suffixed wheel set to TestPyPI on every pull request and every push to master, and re-uploaded tagged releases to TestPyPI rather than PyPI, so the tag-push wheels never reached production PyPI. Make this the canonical production release pipeline: - The publish step uploads to production PyPI (the `repository-url: https://test.pypi.org/legacy/` override is removed). - Publishing is gated to `v*` tag pushes and manual `workflow_dispatch`. Pull-request and master-push runs still build the full wheel matrix as a build-health check, but no longer upload anything (previously every PR and master push published to TestPyPI). The `ReleasePyPI` job's `if:` enforces this. - Job name renamed from `ReleaseTestPyPI` (`ReleaseWheel-TestPyPI`) to `ReleasePyPI` to reflect the destination. - The pull_request-keyed and push-keyed ccache cache steps are collapsed into a single step. The key is derived from `github.ref_name` with a `ccache-wheel-${OS}-` restore-keys fallback, so a first build of a new ref (including a PR's merge ref) warms from the most recent cache for the same OS. - Drop `defaults.run.shell: bash -el {0}`; the workflow does not rely on login-shell `.bashrc` initialisation, so the implicit `bash` shell is sufficient and avoids the extra interactive shell setup cost per step. - The PR "Merge with latest target branch" step is retained so that the release wheels are validated against the current target branch, and is given an explicit committer identity because a non-fast-forward merge creates a merge commit. - Pin every third-party action to a full commit SHA with the upstream tag retained as a trailing comment, so an upstream tag re-point cannot silently change what runs in a release job that has PyPI publish permissions. The wheel build matrix itself is unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
Fold the nightly publishing pipeline into release_pypi.yml so the
cibuildwheel matrix, ccache configuration, and PR merge-with-target
logic are defined once and shared between the stable and nightly
release paths, instead of being duplicated across two workflow files
that would each run a full build for the same source tree.
Behaviour by trigger:
* `push` of a `v*` tag → stable channel: publishes `cytnx X.Y.Z`
to production PyPI exactly as before.
* `push` to `master` → nightly channel: runs
tools/prepare_nightly_release.py to stamp a static
MAJOR.MINOR.PATCH.devYYYYMMDDHHMM version into pyproject.toml,
forwards the same suffix into CMake so `cytnx.__version__`
aligns with the wheel filename, builds the matrix, and uploads
the wheels to the `cytnx-nightly-wheels` organisation channel
on anaconda.org.
* `pull_request` → nightly channel with publishing disabled: the
PR is merged with its target branch and the nightly wheels are
built (validating the nightly release path end to end), but no
upload job runs.
* `workflow_dispatch` → a `channel` input chooses `stable`,
`nightly`, or `auto` (the default, which infers from
`github.ref`: tag → stable, otherwise nightly).
Implementation:
* A new `DetermineChannel` job emits three outputs consumed by
the other jobs: `channel` (stable/nightly), `publish`
(true/false), and `dev_tag` (the `.devYYYYMMDDHHMM` suffix, or
the empty string for stable). Computing `dev_tag` in this
single job rather than re-deriving it inside each matrix entry
ensures that the four parallel `BuildWheel` jobs stamp the
same version even when their runners cross a minute boundary;
otherwise the per-OS wheels for one workflow run could carry
different `.devN` versions and break the anaconda.org index's
per-version resolution.
* `BuildWheel` gains a `needs: DetermineChannel` dependency,
gates the `Install nightly stamping helper` and
`Stamp pyproject.toml for nightly release` steps on
`channel == 'nightly'`, and sources `CYTNX_VERSION_TAG` for
both the stamp step and the cibuildwheel step from
`needs.DetermineChannel.outputs.dev_tag`. The stable path
skips the stamp step entirely so production wheels keep the
numeric version straight from version.cmake.
* The `ReleasePyPI` job's gate changes from the event-based
expression to `channel == 'stable' && publish == 'true'`, and a
`PublishNightlyAnaconda` job is added with the mirror condition
`channel == 'nightly' && publish == 'true'`. The two publish
jobs are mutually exclusive, and the `publish == 'true'` clause
keeps pull-request runs (channel nightly, publish false) from
uploading anything.
Nightlies are published to a self-hosted anaconda.org channel
rather than to PyPI so that the production PyPI project only ever
holds tagged stable releases. Wheels keep the `cytnx` distribution
name in both channels, so the nightly install command is:
pip install --pre \
--extra-index-url \
https://pypi.anaconda.org/cytnx-nightly-wheels/simple \
cytnx
`--pre` selects the dev version; `--extra-index-url` (not
`--index-url`) lets cytnx's runtime dependencies continue to
resolve from PyPI.
Upload is performed by `scientific-python/upload-nightly-action`
pinned to its 0.6.4 commit SHA, matching the SHA-pinning policy of
the other third-party actions in this file. The action runs
`anaconda upload --force` under the hood so workflow re-runs
overwrite the same-named file safely. Authentication uses an
anaconda.org API token supplied through the
`ANACONDA_ORG_UPLOAD_TOKEN` repository secret.
Co-Authored-By: Claude <noreply@anthropic.com>
IvanaGyro
force-pushed
the
release-nightly-and-prod-pypi
branch
from
69559a6 to
5dee678
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Single
release_pypi.ymlworkflow that builds wheels once with cibuildwheel and routes them by a resolved channel:v*tag push, publishescytnx X.Y.Zto production PyPI via OIDC trusted publishing.master(every merged PR), stamps a PEP 440 dev version (MAJOR.MINOR.PATCH.devYYYYMMDDHHMM, UTC) into pyproject.toml and uploads the wheels to thecytnx-nightly-wheelsorganisation channel on anaconda.org.channelinput (auto/stable/nightly);autoinfers fromgithub.ref(tag → stable, otherwise nightly).A single
DetermineChanneljob emits two outputs —channel(stable/nightly) andpublish(true/false) — consumed by the build steps and the two mutually-exclusive publish jobs.The wheels keep the
cytnxdistribution name in both channels. Stable users do nothing new (pip install cytnx). Nightly users add the anaconda.org index:--preselects the.devNpre-release;--extra-index-url(not--index-url) lets cytnx's runtime deps (numpy, graphviz, beartype) keep resolving from PyPI.Why anaconda.org instead of PyPI for nightlies
The original design uploaded nightlies to PyPI as same-project
.devNreleases (numpy-style:pip install cytnx→ stable,pip install --pre cytnx→ nightly). We then applied for hosting on the shared scientific-python-nightly-wheels anaconda.org channel (the channel numpy, scipy, scikit-learn, matplotlib, … use for their nightlies). That application was rejected (scientific-python/upload-nightly-action#167), so we self-host the nightly index on our own anaconda.org organisation,cytnx-nightly-wheels.This keeps the production PyPI project (
cytnx) reserved for tagged stable releases only and isolates per-merge dev wheels on a dedicated index. The publish action (scientific-python/upload-nightly-action) is the same one the scientific-python ecosystem uses — only the target organisation differs.Why one workflow file (and why it builds on PRs)
The stable and nightly paths share the cibuildwheel matrix, ccache configuration, and the PR merge-with-target step; only the publish target differs. Folding both into one file means a change to the build matrix is made in one place, and a same-commit tag push + master merge no longer run two redundant builds.
Pull requests build the (nightly) release wheels but skip publishing, so a PR that breaks the release build — including the nightly version-stamping path — is caught before merge.
DetermineChannelresolves PRs tochannel=nightly, publish=false; thepublish == 'true'clause on both publish jobs keeps PR runs from uploading.Commits
build: refactor optional-deps; add release-tools dependency-group— collapsedevextras tocytnx[test] + cytnx[coverage](one source of truth per leaf group) and add a PEP 735[dependency-groups]table with arelease-toolsgroup (tomlkit). A dependency-group, not an optional-dependency, sopip install --group release-toolsinstalls the helper without going through scikit-build-core (whichpip install .[release-tools]would, compiling cytnx on the host).build: add CYTNX_VERSION_TAG env hook for dev-version suffixes—CMakeLists.txtgainsCYTNX_VERSION_FULL= numericCYTNX_VERSION+$ENV{CYTNX_VERSION_TAG}when set. Used only for theCYTNX_VERSIONcompile def (i.e.cytnx.__version__); numeric-only consumers (project(VERSION ...), SOVERSION, libname) untouched. Also addsCYTNX_VERSION_TAGto the cibuildwheel Linuxenvironment-pass. No-op when unset.build: add nightly-release pyproject stamping helper—tools/prepare_nightly_release.pyusestomlkitto rewrite pyproject.toml (dynamic = ["version"]removed, staticversion = "X.Y.Z.devYYYYMMDDHHMM",[tool.scikit-build.metadata.version]removed) and appendsCYTNX_VERSION_TAGto$GITHUB_ENV. Reuses the regex from[tool.scikit-build.metadata.version].regex, so the regex lives in one place.ci: switch release_pypi.yml from TestPyPI to production PyPI— publish destination TestPyPI → production PyPI. PR and master-push runs still build the full matrix as a build-health check but no longer upload anything (previously every PR/master push went to TestPyPI); publishing is gated tov*tags + dispatch. Job renamedReleaseTestPyPI→ReleasePyPI. PR merge-with-target step retained (with an explicit committer identity for the merge commit). Login-shelldefaultsdropped; two ccache steps collapsed to one; third-party actions SHA-pinned.ci: extend release_pypi.yml with nightly anaconda.org path— adds theDetermineChanneljob (channel+publishoutputs), the nightly stamping steps (gated onchannel == 'nightly'), theworkflow_dispatch.channelinput, and aPublishNightlyAnacondajob uploading to anaconda.org viascientific-python/upload-nightly-action(SHA-pinned 0.6.4) withANACONDA_ORG_UPLOAD_TOKEN.ReleasePyPI's gate moves from the event-based expression tochannel == 'stable' && publish == 'true'.Version stamping — why timestamp instead of incremental
A monotonic timestamp (
devYYYYMMDDHHMM) needs no external state. An incrementaldevNwould require querying the index for the last.devNor storing a counter — both race under parallel merges and break on repo moves. Minute resolution handles multiple merges per day; a same-minute collision is overwritten byanaconda upload --forceon re-run.One-time setup required before merge
cytnxproject) — add a trusted publisher: repositoryCytnx-dev/Cytnx, workflowrelease_pypi.yml, jobReleasePyPI, no environment.cytnx-nightly-wheelsorg) — create an API token with upload scope for the org, and store it as theANACONDA_ORG_UPLOAD_TOKENrepository secret (Settings → Secrets and variables → Actions). The upload step reads it viasecrets.ANACONDA_ORG_UPLOAD_TOKEN.Until these are configured, the respective publish jobs will fail; this is intentional and safe. PR builds need neither secret (they don't publish).
Test plan
cytnxproject.ANACONDA_ORG_UPLOAD_TOKENrepository secret created with upload scope forcytnx-nightly-wheels.