ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

Authors: Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He (Microsoft Corporation)

Abstract: We present ZeRO-Infinity, a novel heterogeneous system technology that leverages GPU, CPU and NVMe memory to allow for unprecedented model scale on limited resources without requiring model code refactoring. At the same time it achieves excellent training throughput and scalability, unencumbered by the limited CPU or NVMe bandwidth. ZeRO-Infinity can fit models with tens and even hundreds of trillions of parameters for training on current generation GPU clusters. It can be used to fine-tune trillion parameter models on a single NVIDIA DGX-2 node, making large models more accessible. In terms of training throughput and scalability, it sustains over 25 petaFLOPS on 512 NVIDIA V100 GPUs (40% of peak), while also demonstrating superlinear scalability.


Presentation: file


Back to Technical Papers Archive Listing