Accelerating Fine-Tuning of Transformers with NVIDIA NeMo AutoModel

NVIDIA NeMo AutoModel is an open-source library within the NVIDIA NeMo framework for building custom generative AI models at scale. It builds on top of Transformers v5, adding Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels, while leveraging dynamic weight loading from v5 to apply these optimizations across a wide range of model families. The result is a 3.4–3.7x increase in training throughput and a 29–32% reduction in GPU memory usage when fine-tuning MoE models compared to native Transformers v5 using the same from_pretrained() API — simply change one import line without modifying the rest of the code. NeMoAutoModelForCausalLM is a subclass of AutoModelForCausalLM, so any code that works with Hugging Face models also works with AutoModel. For popular MoE architectures (Qwen3, NVIDIA Nemotron, GPT-OSS, DeepSeek V3), NeMo AutoModel uses specialized implementations with TransformerEngine attention, fused linear layers, and custom expert kernels. For other models, an optimized fallback with Liger kernels is applied. Performance measurements show: during full fine-tuning of the Nemotron 3 Ultra 550B A55B model on 16 nodes (128 GPUs), NeMo AutoModel achieved 815 tokens/second per GPU and 293 TFLOPs per GPU with peak memory consumption of 58.2 GB, while Transformers v5 failed to run at this scale due to out-of-memory errors. On single-node tests with 8 GPUs for Qwen3-30B-A3B, NeMo AutoModel delivered 11,340 tokens/second per GPU (versus 3,075 for v5, a 3.69x speedup) and peak memory of 48.1 GB (29% less than v5's 68.2 GB). For Nemotron 3 Nano 30B A3B, results were: 15,421 tokens/second (versus 4,583, a 3.36x speedup) and 42.5 GB memory (32% less than v5's 62.1 GB). The speedup comes from three factors: Expert Parallelism reduces memory pressure, DeepEP overlaps communication with computation, and TransformerEngine kernels accelerate core operations.