Skip to content

dask doesn't support rolling over multiple partitions #48

Description

@dwr-psandhu

Getting this error from error_detect which uses dask method.

              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\ProgramData\miniconda3\envs\dms_datastore\Lib\site-packages\vtools\functions\error_detect.py", line 279, in median_test_twoside
    filt = dds.rolling(filt_len,center=True).apply(lambda x: np.nanmedian(x[medseq]),raw=True,engine='numba').compute()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\ProgramData\miniconda3\envs\dms_datastore\Lib\site-packages\dask_expr\_collection.py", line 481, in compute
    return DaskMethodsMixin.compute(out, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\ProgramData\miniconda3\envs\dms_datastore\Lib\site-packages\dask\base.py", line 372, in compute
    (result,) = compute(self, traverse=False, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\ProgramData\miniconda3\envs\dms_datastore\Lib\site-packages\dask\base.py", line 660, in compute
    results = schedule(dsk, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\ProgramData\miniconda3\envs\dms_datastore\Lib\site-packages\dask_expr\_expr.py", line 1043, in _combined_parts
    raise NotImplementedError(msg)
NotImplementedError: Partition size is less than overlapping window size. Try using ``df.repartition`` to increase the partition size.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions