GPU usage for training

[LennoxWei]

Great Work! Could you please tell me some more details about GPU usage for training? How many A100-80Gs did you use for training? And how long did the training last? Thank you so much.

[FangHeng]

Thanks for your interest. Our main training runs used 8 NVIDIA A100 80GB GPUs. We trained for 100,000 steps, and a full run took about 3 days in our setup.

[LennoxWei]

I only have 4 A100-80G GPUs, will that be enough to reproduce your excellent work? Thank you.

[FangHeng]

Thanks for your question. We used DeepSpeed ZeRO-2 for full-parameter fine-tuning. In principle, 4 A100 80GB GPUs may still be feasible, but we have not officially tested this setting, so we cannot guarantee exact reproducibility. With fewer GPUs, the memory usage on each GPU will be slightly higher because the optimizer states are sharded across fewer devices.

A practical option is to keep the effective global batch size aligned with our original setup by gradient accumulation. If needed, you can also further enable gradient checkpointing and slightly lower the batch size for stability. You may also try a smaller base model on top of our training recipe. Dynamic manipulation is still a relatively open problem, and we believe there is still substantial room for further exploration in model scale and training design.

Please let us know if you have any further questions.