(An) empirical study on DNN inference

Abstract

The need for reduction of computation resources consumed by deep learning applications has motivated researches to introduce several techniques that modify pre-trained DNNs. Pruning is one of said techniques, and is particularly suitable for modifying model which will be executed on GPUs and common DL frameworks. Since the introduction of pruning, a grate number of pruning approaches have been developed, each better than the previous one at reducing the inference computations cost while loosing smaller percentages of the original model’s accuracy. However, not much has been said about the degree to which layers of a DNN should be pruned, a particularly tricky task since it has been proven that different layers present a distinct sensitivity to pruning. The aim of this work is to study, experiment with and improve the means by which a proper pruning degree can be determined. To this end, we introduce a mathematical model for pruning sensitivity, as well as a schema for generating pruning degree assignations specific to each f the model’s layers based on their pruning sensitivity characteristic. Subsequent to this, we explore the usefulness of our proposed schema on a variety of models and datasets, providing thus a holistic view of the potential benefits of pruning. We performed a side-by-side comparison between models pruned according to our schema and models pruned according to literature-based pruning degree assignation. In terms of performance and accuracy, our approach allowed us to prune models which have less than 1% accuracy drop in comparison with those pruned according to the literature, while achieving from 17% to 22% more compute cost reduction. Additionally, we pruned a variety of models and achieved more than 30% compute cost reduction without loosing much more than 2.3% accuracy in the majority of cases. Such benefits were obtained thanks to the mathematical base used for modeling the pruning sensitivity of DNN layers. Said model allowed us to reduce considerably extra weights from robust layers while preserving more weights on the sensitive ones, thus achieving a good compromise between inference performance and accuracy.