DNN Model Deployment on Distributed Edges

Abstract

Deep learning-based visual analytic applications have drawn attention by suggesting fruitful combinations with Deep Neural Network (DNN) models and visual data sensors. Because of the high cost of DNN inference, most systems adopt offloading techniques utilizing a high-end cloud. However, tasks that require real-time streaming often suffer from the problem of an imbalanced pipeline due to the limited bandwidth and latency between camera sensors and the cloud. Several DNN slicing approaches show that effectively utilizing the edge computing paradigm effectively lowers the frame drop rate and overall latency, but recent research has primarily focused on building a general framework that only considers a few fixed settings. However, we observed that the optimal split strategy for DNN models can vary significantly based on application requirements. Hence, we focus on the characteristics and explainability of split points derived from various application goals. First, we propose a new simulation framework for flexible software-level configuration, including latency and bandwidth, using dockercompose, and we experiment on a 14-layered Convolutional Neural Network (CNN) model with diverse layer types. We report the results of the total process time and frame drop rate of 50 frames with three different configurations and further discuss recommendations for providing proper decision guidelines on split points, considering the target goals and properties of the CNN layers.