v0.5.0 — robust remote caching, pandas 3.0 support, comprehensive tutorials & demo GIF

[jeremymanning]

Overview

Release v0.5.0 of pydata-wrangler: a robust remote-caching fix, a batch of correctness fixes, a full lint/docs/test cleanup, and resolution of four tracked issues (#5, #8, #29, #30). Everything is green — full test suite, flake8, and a warning-free Sphinx build — and every major public function has been exercised on real inputs with the outputs examined.

Closes #5. Closes #8. Closes #29. Closes #30.

Headline fix — remote-file caching

get_extension() now strips URL query strings/fragments, so Dropbox/Drive-style links like …/minipedia.npz?dl=1 (including the built-in text corpora) cache under a clean, stable name and reload from cache. Previously they were cached as <hash>.npz?dl=1 and then failed to reload (ValueError: Unknown datatype: npz?dl=1), silently re-downloading each time.

Tracked issues

#30 — pandas 3.0 breaks type detection → stack/unstack fail

is_dataframe / is_multiindex_dataframe / panda_handler matched pandas' internal __module__ strings, which pandas 3.0 flattened (pandas.core.frame → pandas). Replaced with isinstance checks (stable across pandas 2.x/3.0); also dropped DataFrame.backfill/.pad from dataframe_like (removed in pandas 3.0).

Direct evidence — isolated pandas 3.0.0 / Python 3.12 venv:

type(pd.DataFrame()).__module__ = 'pandas'   # was 'pandas.core.frame' in 2.x
old __module__ checks -> False (would break is_dataframe / is_multiindex / dataframe_like)
FIXED: is_dataframe(df)=True, dataframe_like(df)=True, stack->MultiIndex,
       is_multiindex_dataframe=True, unstack->2 frames (values match)
pytest core subset under pandas 3.0.0: 30 passed

Also green on pandas 2.3.3. Regression test added (test_issue30_pandas_type_detection).

#29 — __version__ reported the wrong version

__version__ is now read from the installed package metadata (importlib.metadata), making setup.py the single source of truth (with a literal fallback only for un-installed source checkouts). Also fixed the silently-broken bumpversion config that caused the original drift. dw.__version__ == metadata.version('pydata-wrangler') == 0.5.0. Regression test added.

#8 — demo GIF for the README

scripts/make_demo_gif.py renders a terminal-style animated GIF from real datawrangler calls/outputs (array→DataFrame, Polars backend, sentence-transformers embeddings, @funnel). docs/images/demo.gif (900×560, 134 frames, ~0.5 MB), embedded in the README.

#5 — comprehensive tutorials

Filled every gap in the issue checklist across 7 tutorial notebooks (each supported datatype in-memory + from file; sklearn vectorization vs. sentence-transformers embedding; built-in + HuggingFace corpora; interpolation/imputation; all decorators incl. stack/unstack + apply_stacked/unstacked; io local/remote/save; util; core/config.ini). Fixed 3 pre-existing NameErrors that broke top-to-bottom execution. All 7 edited notebooks execute cleanly (0 error outputs) and re-verified with an independent kernel.

Correctness fixes (from a full package audit)

• apply_defaults no longer KeyErrors for names absent from config.ini (had broken sklearn TSNE/MDS/Isomap).
• apply_stacked no longer crashes on return_model=True with unstacked input.
• Removed a dead npz-unwrap branch and fixed two missing f-string prefixes in zoo.text.
• Declared pyarrow in requirements (polars↔pandas conversion needs it; surfaced by the clean pandas-3.0 env).

Tests, lint, docs

• Tests: full suite 61 passed, 0 failed (added 16 tests: caching regressions, return_model contract, Polars helpers, backend config, io.save/load round-trips, value-based assertions, and datawrangler.__version__ reports incorrect version #29/pandas 3.0 support: type(x).__module__ string-matching breaks is_dataframe / is_multiindex_dataframe (→ stack/unstack fail) #30 regressions).
• flake8: 912 → 0 findings (max-line-length = 120 set explicitly; removed genuinely-unused imports, type==→is, bare-except→except Exception, etc.).
• Docs: warning-free sphinx -E -a build (docstrings, RST, notebook lexer/heading fixes, new API stubs). Removed two dead duplicate modules (text_lazy.py, text_original.py).
• Added hierarchical AGENTS.md docs and a runnable scripts/verify_functions.py smoke test.

Comprehensive direct evidence (real calls, outputs examined)

python scripts/verify_functions.py exercises every major public function; representative outputs:

wrangle: array->(2,3) exact values | df preserved | None->empty | polars backend->(3,2)
         image->(1400,5760) | text sklearn LDA->(3,50) | ST embed->(2,384) | mixed->['array','text','dataframe']
decorate: funnel->[2.0,3.0] | list_generalizer 9/[4,9,16] | stack->(2,2)/multiindex | unstack->2 frames
          apply_unstacked->multiindex | interpolate(impute)->no NaN
io: load csv->(7,5) | remote URL->(7,6) | save+load pickle round-trip OK
util: btwn T/F | array_like/dataframe_like T | depth 0/1/2
core: get_default_options OK | apply_defaults stop_words='english' | set/get/reset backend OK
corpus: get_corpus('sotus')->29 docs | get_corpus('cam-cst/cbt','raw')->108 docs (byte-identical to legacy)

🤖 Generated with Claude Code

[jeremymanning]

Manual verification — every user-facing surface (input → output)

Method. Beyond the automated pytest suite, I exercised each public surface by hand with real calls and examined the actual outputs. The exact input (>>>) and its real output are shown below, and it is reproducible any time with python scripts/verify_functions.py. Two environments were used: the main dev env (pandas 2.3.3 / numpy 2.2.6 / Python 3.10) for the full API, and an isolated pandas-3.0 env (pandas 3.0.0 / numpy 2.5.0 / Python 3.12) for issue #30.

---

1) Full public API — pandas 2.3.3

Top-level API — dw.wrangle (each supported datatype)

>>> dw.wrangle(np.array([[1, 2, 3], [4, 5, 6]]))
<DataFrame shape=(2, 3)>
   0  1  2
0  1  2  3
1  4  5  6    # array -> DataFrame
>>> _df = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})
>>> dw.wrangle(_df)
<DataFrame shape=(2, 2)>
   a  b
0  1  3
1  2  4    # DataFrame preserved
>>> dw.wrangle(None)
<empty DataFrame shape=(0, 0)>    # null -> empty DataFrame
>>> dw.wrangle(np.arange(6).reshape(3, 2), backend='polars')
<polars.DataFrame shape=(3, 2)>    # Polars backend
>>> dw.wrangle(dw.io.load(f'{R}/wrangler.jpg')).shape
(1400, 5760)    # image -> DataFrame shape
>>> dw.wrangle(['the cat sat', 'dogs run', 'cats and dogs'], text_kwargs={'model': 'CountVectorizer'}).shape
(3, 1220)    # text sklearn VECTORIZATION
>>> dw.wrangle(['the cat sat', 'dogs run', 'cats and dogs'], text_kwargs={'model': ['CountVectorizer', 'LatentDirichletAllocation'], 'corpus': 'minipedia'}).shape
(3, 50)    # text sklearn LDA (trained on minipedia)
>>> dw.wrangle(['hello world', 'data wrangler'], text_kwargs={'model': 'all-MiniLM-L6-v2'}).shape
(2, 384)    # text sentence-transformers EMBEDDING
>>> dw.wrangle([np.array([1, 2, 3]), 'free text', _df], text_kwargs={'model': 'all-MiniLM-L6-v2'}, return_dtype=True)[1]
list(len=3): ['array', 'text', 'dataframe']    # mixed list -> detected dtypes
>>> dw.wrangle(np.array([[1, 2], [3, 4]]), return_model=True)[1].keys()
dict_keys(['model', 'args', 'kwargs'])    # return_model contract
>>> dw.__version__
'0.5.0'    # issue #29: single-sourced from metadata

Decorators

>>> @dw.funnel
def colmeans(x):
    return x.mean(axis=0)
>>> colmeans(np.array([[1, 2], [3, 4]])).values.tolist()
list(len=2): [2.0, 3.0]    # funnel: raw array -> DataFrame in
>>> @dw.decorate.list_generalizer
def square(x):
    return x ** 2
>>> (square(3), square([2, 3, 4]))
(9, [4, 9, 16])    # list_generalizer
>>> _d1, _d2 = _df.iloc[:1], _df.iloc[1:]
>>> dw.stack([_d1, _d2]).shape
(2, 2)    # stack
>>> dw.zoo.is_multiindex_dataframe(dw.stack([_d1, _d2]))
True    # stacked is MultiIndex
>>> [f.shape for f in dw.unstack(dw.stack([_d1, _d2]))]
list(len=2): [(1, 2), (1, 2)]    # unstack round-trips
>>> @dw.decorate.apply_unstacked
def per_frame(x):
    return pd.DataFrame(x.mean(axis=0)).T
>>> dw.zoo.is_multiindex_dataframe(per_frame(dw.stack([_d1, _d2])))
True    # apply_unstacked
>>> _imp = _df.astype(float).copy(); _imp.loc[0, 'a'] = np.nan
>>> @dw.decorate.interpolate
def _ident(x):
    return x
>>> _ident(_imp, interp_kwargs={'impute_kwargs': {'model': 'IterativeImputer'}}).isna().any().any()
np.False_    # interpolate/impute -> no NaN remain

IO

>>> dw.io.load(f'{R}/testdata.csv', index_col=0).shape
(7, 5)    # load local csv
>>> dw.io.load(f'{R}/home_on_the_range.txt')[:34]
'O give me a home where the buffalo'    # load local txt
>>> dw.io.load_dataframe(f'{R}/testdata.csv').shape
(7, 6)    # load_dataframe (extension auto-detected)
>>> dw.io.save('evidence://obj.pkl', {'x': [1, 2, 3]}, dtype='pickle')
>>> from datawrangler.io.io import get_local_fname
>>> dw.io.load(get_local_fname('evidence://obj.pkl'), dtype='pickle')
{'x': [1, 2, 3]}    # save+load round-trip
>>> dw.io.load('https://raw.githubusercontent.com/ContextLab/data-wrangler/main/tests/resources/testdata.csv').shape
(7, 6)    # load REMOTE url (issue: query-string caching fix)

Util

>>> (dw.util.btwn(np.array([1, 2, 3]), 0, 5), dw.util.btwn(np.array([1, 9]), 0, 5))
(np.True_, np.False_)    # btwn in/out of range
>>> (dw.util.array_like([1, 2, 3]), dw.util.dataframe_like(_df))
(True, True)    # array_like / dataframe_like
>>> (dw.util.depth(5), dw.util.depth([1, 2, 3]), dw.util.depth([[1], [2]]))
(0, np.int64(1), np.int64(2))    # depth

Core / config

>>> all(k in dw.core.get_default_options() for k in ['CountVectorizer', 'text', 'data'])
True    # get_default_options
>>> from sklearn.feature_extraction.text import CountVectorizer
>>> dw.core.apply_defaults(CountVectorizer)().get_params()['stop_words']
'english'    # apply_defaults injects config.ini
>>> dw.core.update_dict({'a': 1, 'b': 2}, {'b': 9, 'c': 3})
{'a': 1, 'b': 9, 'c': 3}    # update_dict
>>> dw.core.set_dataframe_backend('polars')
>>> dw.core.get_dataframe_backend()
'polars'    # set/get backend
>>> dw.core.reset_dataframe_backend()
>>> dw.core.get_dataframe_backend()
'pandas'    # reset backend

Zoo type predicates + text corpora/helpers

>>> (dw.zoo.is_array(np.array([1])), dw.zoo.is_dataframe(_df), dw.zoo.is_text('hi'), dw.zoo.is_null([]))
(True, True, True, True)    # is_<type>
>>> dw.zoo.to_str_list(['a', 'b'])
list(len=2): ['a', 'b']    # to_str_list
>>> len(dw.zoo.text.get_corpus('sotus'))
29    # get_corpus built-in (dropbox ?dl=1 -> caching fix)
>>> len(dw.zoo.text.get_corpus('cam-cst/cbt', 'raw'))
108    # get_corpus HuggingFace (namespaced; datasets>=4)

_All of the above executed on pandas 2.3.3 / numpy 2.2.6 / Python 3.10.12 _

---

2) Issue #30 — pandas 3.0 (isolated venv): old checks demonstrably broken, new code works

Issue #30 — under pandas 3.0.0 / numpy 2.5.0 / Python 3.12.10 (isolated venv)

>>> type(pd.DataFrame()).__module__          # pandas 3.0 flattened this
'pandas'    # was 'pandas.core.frame' in 2.x
>>> 'pandas.core.frame' in [...]             # what the OLD is_dataframe checked
False   <- OLD check now BROKEN
>>> 'indexes.multi' in type(mi).__module__   # what the OLD is_multiindex checked
False   <- OLD check now BROKEN
>>> (hasattr(df, 'backfill'), hasattr(df, 'pad'))  # methods dataframe_like required
(False, False)   <- removed in pandas 3.0

>>> # ---- the FIXED (isinstance-based) code ----
>>> dw.zoo.is_dataframe(df)
True   (expected True)
>>> dw.util.dataframe_like(df)
True   (expected True)
>>> s = dw.stack([df, df]); type(s.index).__name__
'MultiIndex'
>>> dw.zoo.is_multiindex_dataframe(s)
True   (expected True)
>>> [f.shape for f in dw.unstack(s)]          # the cascade the issue reported broken
[(3, 2), (3, 2)]  values match: True
>>> dw.wrangle(np.arange(6).reshape(3, 2)).shape   # array
(3, 2)
>>> dw.wrangle(df, backend='polars').__class__.__name__   # Polars backend
DataFrame
>>> len(dw.wrangle(None))                      # null
0
>>> @dw.funnel def colmeans(x): return x.mean(axis=0); colmeans(np.array([[1,2],[3,4]])).values.tolist()
[2.0, 3.0]
>>> dw.__version__
'0.5.0'

---

3) Cross-version CI (Python 3.9–3.12) — all green

build (3.9)  pass     build (3.10) pass     build (3.11) pass     build (3.12) pass

Python 3.11 and 3.12 install pandas 3.0, so CI itself independently re-runs the whole suite against pandas 3.0 (61 passed / 0 failed on every version). Run: https://github.com/ContextLab/data-wrangler/actions/runs/28683627973

4) Issue #5 — tutorials execute end-to-end

All 7 edited notebooks were re-executed via jupyter nbconvert --execute with a kernel bound to this interpreter: 0 error outputs across 205 cells (core, util, io, decorators1, decorators2, interpolation_and_imputation, wrangling_basics).

5) Issue #8 — demo GIF

docs/images/demo.gif is generated by scripts/make_demo_gif.py from real dw calls/outputs; frames were visually inspected. (Being upgraded to larger / real-world examples per the follow-up request.)

[jeremymanning]

Demo GIF upgraded (issue #8, follow-up)

docs/images/demo.gif now uses larger datasets and real, interesting examples — every number is measured/computed live by scripts/make_demo_gif.py, nothing is faked:

1. Scale + speed: wrangle np.random.rand(20_000_000, 3) (60M values) to the Polars backend, then sort 20M rows — pandas 2842 ms vs polars 339 ms → 8.4× faster (rendered live).
2. Real NLP: wrangle four news headlines into (4, 384) sentence embeddings, then cosine similarity shows the two finance and two cooking sentences cluster by topic (finance↔finance 0.48, food↔food 0.30, cross-topic −0.07).
3. @funnel: a function written for DataFrames applied to a raw numpy array (z-score).
4. ML imputation: IterativeImputer filling gaps in sensor readings.

205 frames, ~1.0 MB, 900×560. Regenerate any time with python scripts/make_demo_gif.py.