Cost-effective, SLO-aware machine learning inference system for heterogeneous instances in public cloud

Abstract

Recently, several cloud companies have released various types of hardware accelerator cloud products. For example, AWS GPU-type instances and inference-specific instances were released, and Google released dedicated instances such as TPU (Tensor Processing Unit). Cloud instances for inference are becoming more diverse as the years go by, and heterogeneity is also emerging in the cloud environment that constitutes a group of instances. In addition, various Machine Learning (ML) models are emerging in various fields for text analysis, text generation, and sound classification as well as image classification. In situations where there is a need for large-scale ML, it is necessary to analyze the performance/cost correlation between various cloud instances, and various ML models, for efficiency. This study introduces the StageH system. StageH was implemented in a distributed and heterogeneous cloud environment. Various ML models (e.g., RESNET, BERT, GPT, YAMNET, INCEPTI ON) keep the SLO as much as possible in the cloud environment where it is executed. In addition, cost-effective autoscaling algorithm saves costs in a cloud environment.