torchtitan

Fork of PyTorch TorchTitan that adds AWS Trainium support with TorchNeuron

Pre-training one LLM across supercomputers at once - AMD/ROCm and NVIDIA/CUDA, different queues, no shared filesystem. DiLoCo outer steps merged by a GPU-less VM via Flower/FedMom; DARL leases the corpus so no token is trained twice; membership is elastic, so a site stuck in the Slurm queue costs nothing.

Reproducible FP4 Llama training: MXFP4, NVFP4, localCTA, hybrid recipes, kernels, evaluations, and paper

UE5M3 FP4 pretraining: Triton kernels, TorchTitan recipes, quantized evaluation, and audited paper artifacts