- Changed the package license to Apache License (>= 2).
-
Remove unused
CallrFutureError(). -
Adjust internal code on how to best clean up callr processes.
- The 'callr' backend was holding on to temporary callr files longer than necessary. Such files were only removed when the future object itself was removed. This would result in a large number of temporary files accumulating where there were many futures processed. Now the 'callr' backend finalizes the 'callr' process as soon as the future results have been collected, which results in removing temporary files created by callr sooner. Previously, the finalizer was only run when the future object was removed and garbage collected.
resolved()for callr futures could throw aFutureErrorin some cases, e.g. aFutureInterruptErrorif the future was terminated abruptly.
-
Now 'callr' futures can be canceled using
cancel(), which also interrupts them by default, which in turn frees up compute resources sooner. Map-reduce API such as future.apply, doFuture, and furrr can take advantage of this by canceling all non-resolved futures whenever they detect an error in one of the futures. Also, canceled futures can bereset()and thereafter relaunched, possibly on another future backend. -
A future that failed due to the 'callr' worker process being terminated is now considered interrupted, which for instance means that it can be
reset(). -
Now 'callr' futures relay
immediateCondition:s in near real-time, e.g.progressionconditions signals by the progressr package.
- If a 'callr' future failed, because the parallel process crashed, the corresponding parallel-worker slot was never released.
run()forCallrFuturewould update the RNG state, becausecallr::r_bg(), used to launch the future, does so. This would compromise numeric reproducibility, where thecallrfuture backend would not give the same random numbers as other future backends. Nowrun()launches the future in stealth RNG mode, i.e. givesr_bg()a semi-random initial seed to work with (by removing.Random.seed) and then undoes the RNG state at the very end.
-
Now
resolved()supports early signaling. -
Now
result()andvalue()gives a slightly more informative error message in case the callr process failed with a non-zero exit code.
-
Add option to configure callr futures to be "supervised" by the callr package when setting up the plan, i.e.
plan(callr, supervise = TRUE). -
Now callr-specific orchestration errors are of class
CallrFutureError, which provides information also on the future that failed.
- A callr future that produces a large amount of standard error (stderr) could stall forever when collecting its results. The exact reason is unknown but the symptom is currently that the underlying processx process never terminates, resulting in a never-ending wait for the results. Since futures don't capture stderr in other backends, the workaround for now is to discard all stderr output. Note that messages, warnings, etc. are still captured and relayed.
- Removed S3 generic function
await(), which was used for internal purposes.
- Add
nbrOfFreeWorkers().
- Removed S3 generic function
await(), which was used for internal purposes.
resolved()forCallrFuturewill launch lazy futures [future (>= 1.15.0)].
- Debug messages are now prepended with a timestamp.
- Now the future label is exposed in the process information
(e.g.
top) via a dummy--future-label="<label>"argument in the callr system call.
plan(callr, workers)whereworkersbeing a function would result in an error when a future was created.
- The 'callr' backend supports the handling of the standard output as implemented in future (>= 1.9.0).
- Callr futures did not protect against recursive parallelism,
e.g. with
plan(list(callr, callr))the second layer of futures would use the same number of workers as the first layer.
-
Argument
workersof future strategies may now also be a function, which is called without argument when the future strategy is set up and used as is. For instance,plan(callr, workers = halfCores)wherehalfCores <- function() { max(1, round(availableCores() / 2)) }will use half of the number of available cores. This is useful when using nested future strategies with remote machines. -
Gathering of results from background processes is made a little bit more robust against slow file systems by retrying a few times before accepting an error as an error.
- Prepared code to gather a richer set of results from futures.
- Callr futures did not acknowledge timeout option
future.wait.timeout.
- Producing errors of class
FutureErrorwhere applicable.
- Minor updates to the vignette related to the callr package.
- Number of workers in
plan(callr, workers = n)was not respected.
- Added 'callr' futures. Use
plan(callr)orplan(callr, workers = 4L).