Repository navigation
Conversation
|
Hi @nrenvo, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice. Once the PR links an issue (or gets the |
|
I ran AI-assisted testing and write-up. |
…izer step When AMP skips an optimizer step at the last batch of an epoch, the static num_train_epochs ceiling causes the outer epoch loop to exit with global_step < max_train_steps. Replace the for-epoch loop with while global_step < max_train_steps so training always completes the requested number of successful steps regardless of where in an epoch the overflow happens.
68b9429 to
b4f1cc8
Compare
When AMP skips an optimizer step at the last batch of an epoch, global_step is not incremented and the for epoch in range loop exits before reaching max_train_steps.
Replace the for-loop with while global_step < args.max_train_steps so training always completes the requested number of steps regardless of where in an epoch the overflow happens.
Adds test_text_to_image_ema_skipped_step_at_epoch_boundary to pin this behaviour.
Splits out from #15173 as requested.
Fixes #15112