Exploration of Multi-accelerator Neural Network Inference in Diversely Heterogeneous Embedded Systems

Workshop:RSDHA: Redefining Scalability for Diversely Heterogeneous Architectures

Authors: Ismet Dagli and Mehmet Belviranli (Colorado School of Mines)

Abstract: Neural network inference (NNI) is commonly used in mobile and autonomous systems for latency-sensitive critical operations such as obstacle detection and avoidance. In addition to latency, energy consumption is also an important factor in such workloads, since battery is a shared and limited resource in such systems. Energy and latency demands can vary based on the physical system state. For example, the remaining energy on a low-running battery should be prioritized for motor consumption in a quadcopter, whereas latency-aware execution becomes a priority if the quadcopter is flying through obstacles. Many recent mobile and autonomous system-on-chips embed a diverse range of accelerators with varying power and performance characters which can be utilized to achieve this fine trade-off between energy and latency. In this paper, we investigate Multi-accelerator Execution(MAE) on diversely heterogeneous embedded systems, where sub-components of a given workload, such as NNI, can be assigned to different type of accelerators to achieve a desired latency or energy goal. We first analyze the energy and performance characteristics of execution of neural network layers on different type of accelerators. We then explore energy/performance trade-offs via layer-wise scheduling for NNI by considering different layer-to-PE mappings. We finally propose a customizable metric, called multi-accelerator execution gain (MAEG), in order to measure the energy or performance benefits of MAE of a given workload. Our empirical results on Jetson Xavier SoCs show that our methodology can provide up to \%28 energy/performance trade-off benefit compared to the case where all layers are assigned to a single PE.

Back to RSDHA: Redefining Scalability for Diversely Heterogeneous Architectures Archive Listing

Back to Full Workshop Archive Listing