Fix double-shifted training loss in GitForCausalLM

[Yigtwxx]

What does this PR do?

GitForCausalLM.forward shifts the labels manually and then passes them positionally to self.loss_function, so they bind to labels and shift_labels stays None. ForCausalLMLoss (loss_utils.py:61-64) then shifts a second time, and position t is trained to predict token t + 2.

Because the manual shift flattens the labels to 1-D before the call, the internal pad-and-slice keeps the shape consistent (N → N+1 → N), so nothing raises — and each batch row's final target is silently pulled in from the next row.

This regressed in #35875, which swapped CrossEntropyLoss (no shift) for self.loss_function (shifts) while leaving the manual shift in place. First released in v4.49.0. Same mechanism as the Moonshine fix in #46784.

Unlike Moonshine, GIT cannot just drop the manual shift: logits carries num_image_tokens leading image positions that must be sliced off regardless. So this PR passes shift_labels explicitly to suppress the internal shift, following the convention already used by csm (modeling_csm.py:619-621), and keeps the tensors 2-D since ForCausalLMLoss flattens them itself — which also removes the cross-row leak.

Verified with the repro from the issue (CPU, no pretrained weights). Before:

model loss            : 4.584834575653076
aligned CE (expected) : 4.646134853363037
matches aligned       : False
matches double-shift  : True
row 0 correct targets : [28, 71, 8, 58, 61, 15]
row 0 actual targets  : [71, 8, 58, 61, 15, 19]
row 1 labels          : [19, 42, 30, 58, 94, 67]

After:

model loss            : 4.646134853363037
aligned CE (expected) : 4.646134853363037
matches aligned       : True
matches double-shift  : False

Also checked that the num_items_in_batch path used by Trainer under gradient accumulation still matches the mean reduction when all targets are valid.

Added test_training_loss_no_double_shift to tests/models/git/test_modeling_git.py, covering both plain labels and a -100-padded copy. It asserts the loss equals the aligned cross-entropy and is not equal to the double-shifted one, so it fails on main (AssertionError: False is not true) and passes with this change.

$ pytest tests/models/git/
190 passed, 297 skipped, 2208 subtests passed

ruff check and ruff format --check (pinned 0.14.10) are clean on both files. No # Copied from blocks reference this code and GIT has no modular file, so nothing needs regenerating.

Fixes #47394

Disclosure: this fix was implemented with the assistance of a code agent, per my proposal in the linked issue, and reviewed and validated by me.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline and the
  Pull Request checks?
• Was this discussed/approved via a Github issue? GitForCausalLM double-shifts the training loss and leaks targets across batch rows (regression from #35875) #47394
• Did you make sure to update the documentation with your changes according to the guidelines?
• Did you write any new necessary tests?

Who can review?

@zucchini-nlp

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: git

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 29637186287:2
Result: success | Jobs: 5 | Tests: 485 | Failures: 0 | Duration: 2m 44s

[Sanjays2402]

since forward now pre-shifts and passes shift_labels with labels=None, the shift_labels[..., 1:] step inside ForCausalLMLoss is skipped, which is the fix. but the manual shifted_logits[:, :-1] + labels[:, 1:] pattern here predates that helper. worth checking Git-derived models (e.g. GitForSequenceClassification path or any copy) for the same double-shift, or is git the only one hitting it?

[Yigtwxx]

Good question. I checked, and git is the only model hitting this specific bug.

Git-derived models: there are none. modeling_git.py defines only GitVisionModel, GitModel and GitForCausalLM — there is no GitForSequenceClassification, and no modular_*.py file imports from models/git, so there is no copy to keep in sync.

The manual pattern elsewhere is safe. Models that still do shift_logits = logits[..., :-1, :] / shift_labels = labels[..., 1:] (gpt2, imagegpt, openai, blip_2, clvp, mamba, falcon_mamba, xlstm, gemma3, llama4, qwen2_audio, granite_speech, modernbert_decoder, ...) pass the result to a plain nn.CrossEntropyLoss / fixed_cross_entropy, not to self.loss_function. ForCausalLMLoss is never involved, so there is no second shift. Those are pre-loss_function leftovers, not bugs.

One other model does hit it: moshi. MoshiForCausalLM.forward (src/transformers/models/moshi/modeling_moshi.py#L1003-L1014) pre-shifts, flattens, and then calls self.loss_function(shift_logits, shift_labels, vocab_size=...) positionally — shift_labels= is never passed as a keyword, so ForCausalLMLoss pads and shifts again (loss/loss_utils.py#L61-L64). Because the tensors are already flattened, the shapes stay valid and it fails silently, exactly like git did.

I left that out of this PR to keep the diff scoped to git; happy to open a separate PR for moshi (or fold it in here if a maintainer prefers).

[zucchini-nlp]

I'd prefer to keep a sinlge PR to fix all models and add a proper tests. A contrib was working on it in #46903 (will nudge them again) so let's close this PR

Fixing and reviewing the same issue in all models one-by-one takes too much time for maintainers and for contribs

[Yigtwxx]

Thanks for the review, and understood on wanting a single consolidated PR — that makes sense from a maintenance standpoint.

One thing worth flagging before this gets folded into #46903: the two cases have different root paths, and I don't think #46903 as currently written would cover GIT.

#46903 targets encoder-decoder models, where decoder_input_ids is built via shift_tokens_right so the logits are already aligned with labels. Its regression test (test_encoder_decoder_loss_no_double_shift) is gated on config.is_encoder_decoder, so it skips GIT entirely.

GIT is decoder-only. The double shift here comes from GitForCausalLM.forward shifting manually — which it has to do anyway, because logits carries num_image_tokens leading image positions that must be sliced off — and then passing the result positionally, so it binds to labels and ForCausalLMLoss shifts again. Dropping the manual shift isn't an option the way it was for Moonshine (#46784); the fix is passing shift_labels explicitly.

So a genuinely model-agnostic PR would need to cover both the encoder-decoder path and the decoder-only-with-prefix-tokens path, with a test that isn't gated on is_encoder_decoder.

Happy to go either way:

• close this and port the GIT case + an ungated test into Fix PPFormulaNet training-loss double-shift (train against labels, not labels[..., 1:]) #46903 (coordinating with @OmkumarSolanki), or
• leave this open if you'd rather keep the decoder-only variant separate.

Just let me know which you prefer and I'll act on it.