Expose prefill/decode context caching via fit_mode="fit_with_cache" (JAX)#61
Open
jjovalle99 wants to merge 1 commit into
Open
Expose prefill/decode context caching via fit_mode="fit_with_cache" (JAX)#61jjovalle99 wants to merge 1 commit into
jjovalle99 wants to merge 1 commit into
Conversation
…JAX) Adds an opt-in fit_mode="fit_with_cache" to TabFMClassifier and TabFMRegressor that runs model.prefill once per ensemble-member chunk at fit time and serves predict/predict_proba through model.decode against the cached context, instead of re-encoding [context + queries] on every call. Decode runs eagerly: jit-ing it copies the multi-GB caches into the executable arena and fuses transposes that exhaust memory at real context lengths. Default fit_mode="fit" is unchanged. Caches are dropped on pickle and rebuilt lazily. Raises NotImplementedError under a multi-device data mesh (cached path is single-device for now) and for non-JAX backends. Parity vs the default path: ~1e-7 (fp32 unit tests, incl. the ensemble preset, categorical permutation, row subsampling); bf16 1.6B checkpoint max prob delta ~0.012 on a 10k-row comparison.
jjovalle99
requested review from
abhidas,
erzel,
rajatsen91,
siriuz42,
tamannarayan and
weihaokong
as code owners
July 8, 2026 13:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs: #60
What
Adds an opt-in
fit_mode="fit_with_cache"toTabFMClassifierandTabFMRegressor(JAX backend):fit()runsmodel.prefillonce per ensemble member view, chunked bybatch_size, and stores the context caches on the estimator.predictandpredict_probarunmodel.decodefor the query rows againstthe cached context, instead of re-encoding context plus queries on every
call.
fit_mode="fit"is unchanged. The cached path reuses the sameclass-shift correction and logit post-processing as the default path.
Why
Predict currently re-processes the full training context on every call, once
per ensemble member.
prefill/decodealready exist in the JAX model(covered by
model_test.py::test_prefill_decode_consistency) and were builtto move that work to fit time. This PR exposes them through the sklearn API.
Steady-state measurements on the released 1.6B classification checkpoint
(RTX 5090 32 GB, bf16, creditcard dataset from OpenML,
n_estimators=1,10k-row context; the one-time prefill inside
fittakes 28.9 s):fit_mode="fit"fit_mode="fit_with_cache"The default path's per-call cost grows faster than linearly with context
length: per member, about 2.5 s at 10k rows, 9 s at 20k, 35 s at 40k. The
cached path's cost does not depend on context length, so the benefit grows
exactly where TabFM is slowest. The table stops at 10k because larger cached
contexts hit decode's memory limits (see Known limitations).
Equivalence
difference about 8e-8 between the cached and default paths, including the
ensemble()preset (NNLS plus calibration), categorical permutation, androw subsampling.
to 2e-4. Max difference 0.012 with
n_estimators=4, and 0.134 on isolatedrows with
n_estimators=1, where no ensemble averaging damps bf16 noise.Both sit within the atol=0.2 that
test_prefill_decode_consistencyusesfor bf16-scale numerics. The paths are numerically equivalent, not bitwise
identical.
Implementation notes
decoderuns eagerly, matching howmodel_test.pyuses it. Wrapping it innnx.jitcopies the multi-GB caches into the executable arena (about 2xpeak memory) and fuses a transpose across all layers that runs out of
memory or fails XLA autotuning at real context lengths. The caches returned
by
prefillare passed todecodeunmodified, so the wrapper never touchesICLearningCacheinternals.EnsembleGenerator.context_tensors()builds the fitted context-onlymember tensors directly from generator state. The public
transformpathand its sklearn validation are untouched. A unit test checks its output,
element for element, against the transform-and-slice derivation under
permute_categorical=Trueandmax_num_rows.batch_sizeat prefill and decode, and caches are stored per chunk.following the existing
_COMPILED_PREDICT_CACHE_ATTRSpattern.CHANGELOG.mdis left untouched; it appears to be maintainer-managedthrough the Copybara sync.
Known limitations
NotImplementedErrorunder a multi-devicedatamesh. The column cache'sleading axis is the layer axis, not batch, so the default sharding spec
does not apply. Sharding support is a follow-up.
card, cached predict works at a 10k context with one member, and runs out
of memory at 10k with four members and at 20k or more. The cause is
decode's internal per-head key/value tensor spanning all 24 layers
(
[heads, layers, B, T, head_dim], about 1 GB per 10k context rows permember); at 40k the same tensor also fails XLA autotuning. This is a
property of the existing
decode, not of this change: the same shapesappear when calling
decodedirectly. Chunking that tensor per layer wouldunlock larger contexts. We can file this separately.
prefill/decodeyet, so this is JAX only.Tests
CachedPredictTest(13 tests): probability and label parity againstthe default mode, including categorical permutation, row subsampling,
single-chunk, and
ensemble()preset configs; regressor value parity;pickle round-trips for both estimators; refit rebuilds the cache; a
positive check that the decode path actually runs; invalid
fit_mode; andthe non-JAX and multi-device guards.
EnsembleGeneratorTest::test_context_tensors_matches_dummy_query_derivation.model_test.pyuntouched and passing.