Skip to content

Reuse reserved memory when resuming an early-stopped chain - #577

Open
antonsoo wants to merge 1 commit into
dfm:mainfrom
antonsoo:fix/resume-preallocated-memory
Open

antonsoo wants to merge 1 commit into
dfm:mainfrom
antonsoo:fix/resume-preallocated-memory

Conversation

@antonsoo

Copy link
Copy Markdown

Stopping a sample loop early leaves reserved space in the default memory backend. Continuing with fewer steps than the unused reservation raises ValueError: negative dimensions are not allowed:

import emcee
import numpy as np

coords = np.random.RandomState(0).normal(size=(16, 2))
sampler = emcee.EnsembleSampler(16, 2, lambda p: -0.5 * np.dot(p, p))
for i, state in enumerate(sampler.sample(coords, iterations=1000)):
    if i == 9:
        break
sampler.run_mcmc(state, 20)
assert sampler.iteration == 30

Reuse existing storage until more is needed, and initialize a new blob array across the full reserved capacity. The regressions compare an early-stopped/resumed run with an uninterrupted chain, including structured blobs and thinning, and retain the HDF backend as a control.

Validation on Linux/Python 3.14.7: all 147 tests and repository-wide pre-commit hooks pass. On 107 NIST ultrasonic calibration measurements, all 72 continuation configurations match uninterrupted runs exactly, including positions, log probabilities, blobs, accepted-step counts and RNG state. This repairs all 36 failing memory-backend runs; all 36 HDF controls remain unchanged. This is continuation correctness, not a new convergence or statistical-efficiency claim.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant