Skip to content

feat(tune-strategy): tune with every backend for the device by default - #55

Open
lslusarczyk wants to merge 2 commits into
intel:mainfrom
lslusarczyk:feat/default-strategy
Open

lslusarczyk wants to merge 2 commits into
intel:mainfrom
lslusarczyk:feat/default-strategy

Conversation

@lslusarczyk

@lslusarczyk lslusarczyk commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

A module without a strategy is now tuned with tt.default_strategy(device).
It measures every backend that runs on the device and keeps the fastest.

tt.default_strategy(device) is HighestThroughputStrategy over default_backends(device):

Device Backends
XPU XPU graph, Torch Inductor, OpenVINO, Torch Eager
CPU Torch Inductor, OpenVINO, Torch Eager

TorchAO is left out because quantization changes the results.

The old default was one FirstWinsStrategy instance shared by every module.
FirstWins keeps the first backend that builds, not the fastest one.
The shared instance also carried state from one module to the next.
Each graph now gets its own strategy, built for the device it is tuned on.

FirstWinsStrategy and HighestThroughputStrategy without backends use the same default_backends(device).
FirstWinsStrategy keeps its order, XPU graph first.
HighestThroughputStrategy() now also measures OpenVINO.
On CPU neither strategy tries XPU graph.

On an Arc Pro B70 the default strategy was the fastest or close to it on five of six models.
OpenVINO wins on whole ViT and DistilBERT models and loses on small modules, so it is measured, not excluded.

Breaking:

  • Default tuning takes longer, because it measures every backend.
  • Without backends, tuning on a device other than XPU or CPU raises ValueError.

@lslusarczyk
lslusarczyk force-pushed the feat/default-strategy branch from e27f664 to 6edd20b Compare October 29, 2026 14:47
@lslusarczyk
lslusarczyk marked this pull request as ready for review October 29, 2026 14:47
@lslusarczyk
lslusarczyk requested a review from a team as a code owner October 29, 2026 14:47
A module without a strategy is now tuned with tt.default_strategy(device).
It is HighestThroughputStrategy over default_backends(device).
On XPU these are XPU graph, Torch Inductor, OpenVINO, and Torch Eager.
On CPU these are Torch Inductor, OpenVINO, and Torch Eager.
TorchAO is left out because quantization changes the results.

The old default was one FirstWinsStrategy instance shared by every module.
FirstWins keeps the first backend that builds, not the fastest one.
The shared instance also carried state from one module to the next.
Each graph now gets its own strategy, built for the device it is tuned on.

FirstWinsStrategy and HighestThroughputStrategy without backends use the same default_backends(device).
FirstWinsStrategy keeps its order, XPU graph first.
HighestThroughputStrategy() now also measures OpenVINO.
On CPU neither strategy tries XPU graph.
Without backends, a device other than XPU or CPU raises ValueError.

On an Arc Pro B70 the default strategy was the fastest or close to it on five of six models.
OpenVINO wins on whole ViT and DistilBERT models and loses on small modules, so it is measured, not excluded.

Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
@lslusarczyk lslusarczyk changed the title feat: tune with every backend for the device by default feat(tune-strategy): tune with every backend for the device by default Oct 29, 2026
@lslusarczyk
lslusarczyk force-pushed the feat/default-strategy branch from 6edd20b to 6fa7a28 Compare October 29, 2026 14:50
@lslusarczyk
lslusarczyk enabled auto-merge (squash) October 29, 2026 16:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant