Bayesian Optimization for Auto-Tuning GPU kernels
Event Type
Workshop
Online Only
Accelerator-based Architectures
Applications
Computational Science
Emerging Technologies
Extreme Scale Comptuing
File Systems and I/O
Heterogeneous Systems
Parallel Programming Languages and Models
Performance
Scientific Computing
Software Engineering
W
TimeMonday, 15 November 20214:30pm - 5pm CST
LocationOnline
DescriptionFinding optimal parameter configurations for tunable GPU kernels is a non-trivial exercise for large search spaces, even when automated. This poses an optimization task on a non-convex search space, using an expensive-to-evaluate function with unknown derivative. These characteristics make a good candidate for Bayesian optimization, which previously has not been applied to this problem.
The application of Bayesian optimization to this problem, however, is challenging. We demonstrate how to deal with the rough, discrete, constrained search spaces, containing invalid configurations. We introduce a novel contextual variance exploration factor, as well as new acquisition functions with improved scalability, combined with an informed acquisition function selection mechanism. By comparing the performance of our Bayesian optimization implementation on various test cases to the existing search strategies in Kernel Tuner, as well as other Bayesian optimization implementations, we demonstrate that our search strategies generalize well and consistently outperform other search strategies by a wide margin.
The application of Bayesian optimization to this problem, however, is challenging. We demonstrate how to deal with the rough, discrete, constrained search spaces, containing invalid configurations. We introduce a novel contextual variance exploration factor, as well as new acquisition functions with improved scalability, combined with an informed acquisition function selection mechanism. By comparing the performance of our Bayesian optimization implementation on various test cases to the existing search strategies in Kernel Tuner, as well as other Bayesian optimization implementations, we demonstrate that our search strategies generalize well and consistently outperform other search strategies by a wide margin.
