fix(trainer): fix resume_from_checkpoint logic bug (issue #47375)#47391
Open
varneeth01 wants to merge 1 commit into
Open
fix(trainer): fix resume_from_checkpoint logic bug (issue #47375)#47391varneeth01 wants to merge 1 commit into
varneeth01 wants to merge 1 commit into
Conversation
…#47375) The condition was always True when resume_from_checkpoint is not None, even when it's an empty string. Changed to so empty string is properly handled as "no checkpoint". Also fix batch size calculation for evaluation loss in several examples to use actual batch size from input tensors instead of configured batch size, correctly handling the last partial batch.
Contributor
|
Thank you for your contribution 🤗! CI Security Gate — automatic approval blockedThis PR was not automatically approved for CI because the security gate failed. Possible reasons:
See the workflow run for the exact violations. A maintainer can review and manually approve CI if a finding is a false positive. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #47375
This PR fixes two correctness issues in the examples/pytorch//run__no_trainer.py scripts.
The previous condition:
if args.resume_from_checkpoint is not None or args.resume_from_checkpoint != "":
This produced incorrect metrics for the final partial batch.
This PR updates those examples to use the actual batch size when computing evaluation loss and perplexity.