Repository navigation
feat: add tweak, which picks the fastest cut depth - #56
Draft
lslusarczyk wants to merge 9 commits into
Draft
lslusarczyk wants to merge 9 commits into
lslusarczyk wants to merge 9 commits into
Conversation
A module without a strategy is now tuned with tt.default_strategy(device). It is HighestThroughputStrategy over default_backends(device). On XPU these are XPU graph, Torch Inductor, OpenVINO, and Torch Eager. On CPU these are Torch Inductor, OpenVINO, and Torch Eager. TorchAO is left out because quantization changes the results. The old default was one FirstWinsStrategy instance shared by every module. FirstWins keeps the first backend that builds, not the fastest one. The shared instance also carried state from one module to the next. Each graph now gets its own strategy, built for the device it is tuned on. FirstWinsStrategy and HighestThroughputStrategy without backends use the same default_backends(device). FirstWinsStrategy keeps its order, XPU graph first. HighestThroughputStrategy() now also measures OpenVINO. On CPU neither strategy tries XPU graph. Without backends, a device other than XPU or CPU raises ValueError. On an Arc Pro B70 the default strategy was the fastest or close to it on five of six models. OpenVINO wins on whole ViT and DistilBERT models and loses on small modules, so it is measured, not excluded. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
`auto_tune` tunes the model at every depth from `min_depth` to `max_depth` and keeps the fastest cut. Modules shared between depths are tuned once. The shallowest depth within the tie tolerance wins. The model is left untuned when no depth beats the untuned model. The default strategy does not search the maximum batch size. Modules that fail to tune keep their original forward, and a depth where every module fails is skipped. Weights that a backend moved to CPU go back to their device before the next depth. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
Every parameter after `dataset` is keyword-only. `print(result)` shows the latency of every depth and the selected one. The docs say that `obj` stays untuned when the whole model wins. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
`auto_tune` starts no new depth after `time_budget` seconds. The depth in progress finishes. Depths not started are reported with the error "time budget used up". Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
`auto_tune` uses `tt.default_strategy(device)` when no strategy is given, without the search for the maximum batch size. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
The first measurement can run before the device reaches its full clock. On an Arc Pro B70 the same untuned T5 encoder measured 2.8 ms to 8.3 ms across runs, which favored tuning. The untuned model is measured again after the last depth, and the lower latency of every sample counts. The second measurement is skipped when no depth was measured. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
The example tunes a four-layer transformer encoder with `auto_tune`. A dry run shows how many modules every depth wraps. `print(result)` shows the latency of every depth against the untuned model. The winner is saved, loaded, and checked against the untuned output. CI runs it next to the minimal example. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
The most complete entry point carries the name of the project. `tt.auto_tune` is now `tt.tweak`, and `AutoTuneResult` is `TweakResult`. The module, guide, and example move to `tweaking.py`, `docs/guides/tweak.md`, and `examples/Tweak`. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
`tweak` picks the cut depth and the backend for every module. `inspect`, `wrap`, and `tune` tune one cut at the depth the user chooses, and are faster. The README, the learn pages, the guides, the assistant skill, and the docstrings now say so. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
lslusarczyk
force-pushed
the
feat/auto-tune
branch
from
October 29, 2026 16:36
b25cb2d to
ee61312
Compare
lslusarczyk
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tt.tweakis the main entry point of Torch Tweak.It tunes a model at every cut depth and keeps the fastest one.
Depends on #55.
The first commit belongs to #55. Review from
feat: add auto_tuneon.tweakinspects the model once and wraps each distinct cut frommin_depthtomax_depth.Every cut is tuned and measured end to end on the dataset.
A module tuned at one depth is reused at deeper depths.
The shallowest depth within
tie_toleranceof the fastest wins.The model stays untuned when no depth beats it.
datasetare keyword-only.print(result)shows the latency of every depth.time_budgetstops the search between depths once the budget is used up.tt.default_strategy(device), without the search for the maximum batch size.inspect,wrap, andtunestay.They tune one cut at the depth the user chooses, and are faster.
The README, the learn pages, the guides, and the assistant skill now present
tweakfirst.examples/Tweaktunes a four-layer transformer encoder.On an Arc Pro B70, depth 0 is 2.5x faster than eager and depth 2 (32 modules) is 0.66x.