(A) scalable heterogeneous vector-array architecture with resource scheduling for multi-user/multi-DNN workloads

Abstract

As artificial intelligence and machine learning technologies disrupt a wide range of industries, cloud datacenters face ever-increasing demand in inference workloads. However, conventional CPU-based servers cannot handle excessive computational requirements of deep neural network (DNN) models, while GPU-based servers suffer from huge power consumption and high cost. In this paper, we present a scalable heterogeneous vector-array architecture that can cope with dynamically changing multi-user/multi-DNN workloads in cloud datacenters. We propose a heterogeneous architecture that features a load balancer that performs a high-level workload distribution and multiple vector-array clusters. Each cluster consists of a programmable scheduler, throughput-oriented array processors, function-oriented vector processors, and shared memories, in which the scheduler is responsible for allocating layer/sub-layer tasks in the cluster. The array processor is designed for matrix/convolution operations and the vector processor is designed for various algorithmic functions required in DNN models. For supporting multi-user/multi-DNN in our proposed architecture, we devise a lightweight DNN model description format that enables general model representation with user description. We also propose a novel resource scheduling algorithm that enables concurrent execution of runtime tasks while maximizing hardware utilization based on computation and memory access time estimation. Finally, we build an architecture and algorithm simulation framework based on actual synthesis and place-and-route implementation results. As a result, the proposed heterogeneous vector-array architecture achieves 10.9x higher throughput performance and 30.17x higher energy efficiency than a compatible GPU on various ML workloads. This research is conducted in collaboration with Sungyeob, Yoo, a master's student of the laboratory.