Skip to content

fix(trainer): fix resume_from_checkpoint logic bug (issue #47375)#47391

Open
varneeth01 wants to merge 1 commit into
huggingface:mainfrom
varneeth01:fix/trainer-resume-47375
Open

fix(trainer): fix resume_from_checkpoint logic bug (issue #47375)#47391
varneeth01 wants to merge 1 commit into
huggingface:mainfrom
varneeth01:fix/trainer-resume-47375

Conversation

@varneeth01

Copy link
Copy Markdown

What does this PR do?
Fixes #47375

This PR fixes two correctness issues in the examples/pytorch//run__no_trainer.py scripts.

  1. Fix resume_from_checkpoint logic

The previous condition:

if args.resume_from_checkpoint is not None or args.resume_from_checkpoint != "":

  • prevented the automatic checkpoint discovery path from being reached when an empty string was provided.
  • This PR changes the condition to require both checks so an empty string correctly triggers automatic checkpoint discovery.
  1. Fix evaluation loss normalization
  • Several no_trainer examples normalized evaluation loss using the configured batch size instead of the actual batch size from the input tensors.

This produced incorrect metrics for the final partial batch.
This PR updates those examples to use the actual batch size when computing evaluation loss and perplexity.

…#47375)

The condition
was always True when resume_from_checkpoint is not None, even when it's an empty string.
Changed to  so empty string is properly handled as "no checkpoint".

Also fix batch size calculation for evaluation loss in several examples to use actual
batch size from input tensors instead of configured batch size, correctly handling
the last partial batch.
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

  • The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
  • A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
  • A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant