blosc2.random — chunk-parallel NumPy-quality random constructors#678
Open
FrancescAlted wants to merge 7 commits into
Open
blosc2.random — chunk-parallel NumPy-quality random constructors#678FrancescAlted wants to merge 7 commits into
FrancescAlted wants to merge 7 commits into
Conversation
Spawns one SeedSequence stream per chunk and generates chunks concurrently
in a thread pool (NumPy Generator fill loops release the GIL), giving full
PCG64 quality with ~3.2x measured speedup over asarray(np.random...) while
staying seed-reproducible regardless of thread count.
Covers default_rng().{random,integers,normal,uniform} for now.
Fills out the easy-win half of numpy.random.Generator's surface: every distribution whose numpy signature is (params..., size=None) drops straight into the existing _fill/_bounded_map machinery as a one-line wrapper. Still out: choice/permutation/shuffle/permuted (sequential, not per-chunk independent), the multivariate draws and dirichlet (vector-valued per draw), and bytes (not an NDArray). gamma/standard_gamma rename numpy's distribution-shape parameter to shape_param (positional-only) since it collides with blosc2's array-shape `shape` keyword.
choice (replace=True, 1-D/scalar `a`) drops straight into the existing per-chunk-independent machinery, same as any scalar distribution; replace=False and multi-dim `a` raise NotImplementedError since they'd need cross-chunk coordination this module doesn't do. dirichlet/multinomial/multivariate_hypergeometric/multivariate_normal each draw a whole length-k vector per element, so output shape is shape + (k,) with the trailing dim pinned to k and never split across chunks. Factored the shared chunk-loop out of _fill into _fill_chunks so _fill_vector can reuse it with a different chunk-shape function.
doc/reference/random.rst now documents the module's consistent deviations from numpy.random.Generator (shape vs size, no out=, gamma's shape_param rename, choice's replace=True restriction, vector-output trailing dim) plus a full coverage table for all 43 Generator methods (39 implemented). examples/ndarray/random-constructor.py is a runnable walkthrough: basic draws, reproducibility semantics, blosc2 kwargs passthrough, a few distributions including a vector-valued one, and a timing comparison against asarray(np.random...). RELEASE_NOTES.md entry updated to match the current 39/43 coverage.
Unlike the rest of the module these are not chunk-parallel: whole-array shuffling is inherently sequential, so each call materializes the full array into memory and shuffles it single-threaded with one spawned SeedSequence stream, then writes the result back out. shuffle additionally requires its argument to already be a blosc2.NDArray, since it mutates in place and returns None to match numpy's contract. Brings coverage of numpy.random.Generator's public methods to 42/43 (only bytes, which returns raw bytes rather than an NDArray, remains out of scope). Docs, coverage table, and RELEASE_NOTES updated to match.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new blosc2.random module that provides seedable, NumPy-API-shaped random NDArray constructors which generate each chunk independently (via SeedSequence.spawn) and fill chunks concurrently in a thread pool for true parallel generation.
Changes:
- Introduce
src/blosc2/random.pyimplementing aGeneratorplusdefault_rng()with chunk-parallel filling for manynumpy.random.Generatormethods, and documented sequential exceptions (permutation,permuted,shuffle). - Add a comprehensive new test suite for reproducibility, sanity checks, and key API deviations.
- Document the feature (reference docs + release notes) and add an example script; export the module via
blosc2.random.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/blosc2/random.py |
New chunk-parallel random generator implementation (Generator, default_rng, chunk filling helpers). |
tests/test_random.py |
New tests covering reproducibility, distribution sanity, vector-valued distributions, and shuffling behavior. |
src/blosc2/__init__.py |
Expose blosc2.random at the package top level. |
RELEASE_NOTES.md |
Announce new blosc2.random feature and scope/limitations. |
examples/ndarray/random-constructor.py |
Add example usage + basic performance comparison snippet. |
doc/reference/random.rst |
New reference documentation and compatibility table. |
doc/reference/array_operations.rst |
Add random to the reference toctree. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+213
to
+216
| def test_vector_dist_scalar_leading_shape(): | ||
| a = blosc2.random.default_rng(0).dirichlet([1.0, 2.0, 3.0], shape=()) | ||
| assert a.shape == (3,) | ||
|
|
The WASM/Pyodide CI jobs (PR #678) reported a real nthreads count but can't actually spawn OS threads, so ThreadPoolExecutor.submit() raised "RuntimeError: can't start new thread" on the very first chunk, crashing test_random_distribution_sanity[2] and test_scheduling_independent_of_nthreads. _bounded_map now catches that RuntimeError and falls back to serial generation instead of propagating it. Since the failure happens on the very first submit (thread creation is refused outright, not exhausted), nothing has been yielded yet, so restarting fully serial is safe and correct.
Copilot review on PR #678 (src/blosc2/random.py:98): _fill_vector normalized shape with isinstance(shape, int), but blosc2.empty() and the rest of this module accept np.integer too. shape=np.int64(5) fell into tuple(shape) and raised "'numpy.int64' object is not iterable" since numpy scalars aren't iterable. Widened the check to int | np.integer, matching how permutation() already handles its own int-vs-array-like input, and added the regression test Copilot also flagged as missing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What: A new blosc2.random module providing seedable random NDArray constructors that mirror numpy.random.Generator, but generate each chunk independently in a thread pool instead of serially — giving full PCG64 statistical quality with genuinely parallel generation (measured ~3x faster than asarray(np.random.default_rng(...).random(...)) on a 100M-element array).
Why: A prior hash-based DSL random generator was fast but statistically weak. NumPy's SeedSequence.spawn() exists precisely to hand out independent, high-quality streams to parallel workers — this hands one stream per chunk and fills chunks concurrently (NumPy Generator fill loops release the GIL, so this actually parallelizes, unlike a lazyudf-based approach which was measured to serialize generation through a single Python thread).
Coverage: 42 of numpy.random.Generator's 43 public methods (full compatibility table in doc/reference/random.rst):
API deviations from numpy (all documented in doc/reference/random.rst): shape replaces size and is required + keyword-only (except on random); no out=; gamma/standard_gamma rename numpy's colliding shape parameter to shape_param; reproducibility depends on (seed, call order, shape, chunks) since chunk layout determines stream assignment.