Exploration of vector-array architecture with heterogeneity-aware scheduling for multi-user/multi-DNN workloads

Abstract

With the dominance of machine learning and artificial intelligence in today's technology, designing an accelerator platform for fast and efficient completion of inference workloads in datacenters is becoming essential. General-purpose processors such as CPU and GPU have been mainly used in datacenters, but they are not suitable for ML inference workloads due to low performance and high power consumption. This paper proposes a vector-array architecture with heterogeneity-aware scheduling for multi-user/multi-DNN workloads in datacenters. It features a load balancer and multiple vector-array clusters, where each cluster consists of a scheduler, array processors, and vector processors. The main contribution is threefold. First, we devise the unified model format (UMF) to describe DNN models in a hardware-amenable packet form. Second, we propose a scheduling algorithm that efficiently allocates the concurrent tasks to available resources at run-time by estimating the computation and external memory access time. Third, we implement an analysis framework based on the implementation results of the proposed architecture. Using this framework, we conduct a design space exploration for this architecture and provide insights for advanced ML accelerator design. As a result, the proposed heterogeneity-aware scheduling algorithm improves the throughput and energy efficiency by 82% and 21%, respectively, compared to a standard round-robin algorithm. This research is conducted in collaboration with Jung-Hoon Kim, a master's student at KAIST.