Exploration of Multi-Accelerator Neural Network Inference in Diversely Heterogeneous Embedded Systems
Event Type
Workshop
Architectures
Extreme Scale Comptuing
Heterogeneous Systems
W
TimeFriday, 19 November 202111:10am - 11:23am CST
Location229
DescriptionNeural network inference (NNI) is commonly used in mobile and autonomous systems for latency-sensitive critical operations such as obstacle detection and avoidance. In addition to latency, energy consumption is also an important factor in such workloads, since battery is a shared and limited resource in such systems.
Energy and latency demands can vary based on the physical system state. For example, the remaining energy on a low-running battery should be prioritized for motor consumption in a quadcopter, whereas latency-aware execution becomes a priority if the quadcopter is flying through obstacles.
Many recent mobile and autonomous system-on-chips embed a diverse range of accelerators with varying power and performance characters which can be utilized to achieve this fine trade-off between energy and latency.
In this paper, we investigate Multi-accelerator Execution(MAE) on diversely heterogeneous embedded systems, where sub-components of a given workload, such as NNI, can be assigned to different type of accelerators to achieve a desired latency or energy goal. We first analyze the energy and performance characteristics of execution of neural network layers on different type of accelerators. We then explore energy/performance trade-offs via layer-wise scheduling for NNI by considering different layer-to-PE mappings. We finally propose a customizable metric, called multi-accelerator execution gain (MAEG), in order to measure the energy or performance benefits of MAE of a given workload. Our empirical results on Jetson Xavier SoCs show that our methodology can provide up to \%28 energy/performance trade-off benefit compared to the case where all layers are assigned to a single PE.
Energy and latency demands can vary based on the physical system state. For example, the remaining energy on a low-running battery should be prioritized for motor consumption in a quadcopter, whereas latency-aware execution becomes a priority if the quadcopter is flying through obstacles.
Many recent mobile and autonomous system-on-chips embed a diverse range of accelerators with varying power and performance characters which can be utilized to achieve this fine trade-off between energy and latency.
In this paper, we investigate Multi-accelerator Execution(MAE) on diversely heterogeneous embedded systems, where sub-components of a given workload, such as NNI, can be assigned to different type of accelerators to achieve a desired latency or energy goal. We first analyze the energy and performance characteristics of execution of neural network layers on different type of accelerators. We then explore energy/performance trade-offs via layer-wise scheduling for NNI by considering different layer-to-PE mappings. We finally propose a customizable metric, called multi-accelerator execution gain (MAEG), in order to measure the energy or performance benefits of MAE of a given workload. Our empirical results on Jetson Xavier SoCs show that our methodology can provide up to \%28 energy/performance trade-off benefit compared to the case where all layers are assigned to a single PE.
