Skip to content

fix(checkpoint): save tuned modules nested in pipeline components - #42

Merged
lslusarczyk merged 3 commits into
intel:mainfrom
lslusarczyk:fix/checkpoint-components-with-tuned-modules
Oct 29, 2026
Merged

lslusarczyk merged 3 commits into
intel:mainfrom
lslusarczyk:fix/checkpoint-components-with-tuned-modules

Conversation

@lslusarczyk

@lslusarczyk lslusarczyk commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

save now stores tuned modules nested inside pipeline components.

save stored only pipeline attributes that are tuned modules.
A tuned module inside a component, such as vae.decoder, was left out.
Save then failed with "No tuned modules found".

Each tuned module is stored under its dotted path, e.g. vae.decoder.
Modules that are not tuned are not saved, so the checkpoint does not carry the weights of the whole component.
A module that the pipeline holds under several paths is saved once and restored as one module.
Older checkpoints with one entry per alias load the module once.

save warns about tuned modules it does not save.
A module held in a plain list, dict or object is still not saved.
A TODO marks the search to extend, the same way ModuleInspector walks containers.

The code follows ai-dynamo/aitune#108, which merged Konrad Nowicki's fix.
One walk maps every module path, and load restores through the same map.
A path to a non-module attribute is still reported as not a module.

Tests tune real modules on XPU, save, load into a fresh pipeline, and check the result.

@lslusarczyk
lslusarczyk force-pushed the fix/checkpoint-components-with-tuned-modules branch from c0d3894 to ad1c11c Compare October 23, 2026 10:29
@lslusarczyk lslusarczyk changed the title fix(checkpoint): save pipeline components that contain tuned modules fix(checkpoint): save tuned modules nested in pipeline components Oct 23, 2026
@lslusarczyk
lslusarczyk force-pushed the fix/checkpoint-components-with-tuned-modules branch 3 times, most recently from 37712e2 to 50e3d28 Compare October 28, 2026 11:33
`save` stored only pipeline attributes that are tuned modules.
A tuned module inside a pipeline component was left out, and save failed with "No tuned modules found".

Each tuned module is now stored under its dotted path, e.g. `vae.decoder`.
Modules that are not tuned are not saved.
A module that the pipeline holds under several paths is saved once and restored as one module.
Older checkpoints with one entry per alias load the module once.

`save` warns about tuned modules it does not save.
A module held in a plain list, dict or object is still not saved.

Follows ai-dynamo/aitune#108: one walk maps every module path, and load reuses it.
A path to a non-module attribute is still reported as not a module.

Co-authored-by: Konrad Nowicki <knowicki@nvidia.com>
Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
@lslusarczyk
lslusarczyk force-pushed the fix/checkpoint-components-with-tuned-modules branch from 50e3d28 to 9f592b6 Compare October 28, 2026 13:27
@lslusarczyk
lslusarczyk marked this pull request as ready for review October 28, 2026 13:27
@lslusarczyk
lslusarczyk requested a review from a team as a code owner October 28, 2026 13:27
@lslusarczyk
lslusarczyk enabled auto-merge (squash) October 29, 2026 08:13
Comment thread torch_tweak/torch/checkpoint/torch_checkpoint.py Outdated
A module reached again, e.g. through an alias or a module that holds itself, keeps its path but its children are not walked again.
A module that holds itself no longer causes a RecursionError.

Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
@lslusarczyk
lslusarczyk merged commit 9e31362 into intel:main Oct 29, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants