The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop:LLVM-HPC2021: The Seventh Workshop on the LLVM Compiler Infrastructure in HPC
Authors: William S. Moses and Valentin Churavy (Massachusetts Institute of Technology (MIT))
Abstract: Derivatives are key to algorithms in scientific computing and machine learning such as optimization, uncertainty quantification, and stability analysis. Enzyme is a LLVM compiler plugin for reverse-mode automatic differentiation (AD) and thus generates fast gradients of programs in a variety of languages (C/C++, Fortran, Julia, Rust, etc) and architectures (CPU, CUDA, ROCm). Unlike existing tools which must operate at the source level, Enzyme can differentiate after optimization, which allows for asymptotically faster gradients. The need to optimize first is accentuated when differentiating parallel and specifically GPU programs, where data races and complex memory hierarchies can dramatically change runtimes. In addition to describing the importance that optimizations have on AD, this talk will preview the challenges that arise when synthesizing parallel gradient functions.
For more details on GPU AD and GPU/AD-specific optimization, come to our SC talk!