Enzyme: Fast, Language Agnostic, Differentiation of Parallel Programs in LLVM
Parallel Programming Systems
TimeSunday, 14 November 20212:39pm - 2:46pm CST
DescriptionDerivatives are key to algorithms in scientific computing and machine learning such as optimization, uncertainty quantification, and stability analysis. Enzyme is a LLVM compiler plugin for reverse-mode automatic differentiation (AD) and thus generates fast gradients of programs in a variety of languages (C/C++, Fortran, Julia, Rust, etc) and architectures (CPU, CUDA, ROCm). Unlike existing tools which must operate at the source level, Enzyme can differentiate after optimization, which allows for asymptotically faster gradients. The need to optimize first is accentuated when differentiating parallel and specifically GPU programs, where data races and complex memory hierarchies can dramatically change runtimes. In addition to describing the importance that optimizations have on AD, this talk will preview the challenges that arise when synthesizing parallel gradient functions.
For more details on GPU AD and GPU/AD-specific optimization, come to our SC talk!