LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference

Cited 0 time in webofscience Cited 0 time in scopus
  • Hit : 37
  • Download : 0
DC FieldValueLanguage
dc.contributor.authorChoi, Yujeongko
dc.contributor.authorKim, Yunseongko
dc.contributor.authorRhu, Minsooko
dc.date.accessioned2021-06-10T06:50:17Z-
dc.date.available2021-06-10T06:50:17Z-
dc.date.created2021-06-09-
dc.date.issued2021-03-02-
dc.identifier.citationThe 27th IEEE International Symposium on High-Performance Computer Architecture (HPCA-27), pp.493 - 506-
dc.identifier.issn1530-0897-
dc.identifier.urihttp://hdl.handle.net/10203/285733-
dc.description.abstractIn cloud ML inference systems, batching is an essential technique to increase throughput which helps optimize total-cost-of-ownership. Prior graph batching combines the individual DNN graphs into a single one, allowing multiple inputs to be concurrently executed in parallel. We observe that the coarse-grained graph batching becomes suboptimal in effectively handling the dynamic inference request traffic, leaving significant performance left on the table. This paper proposes LazyBatching, an SLA-Aware batching system that considers both scheduling and batching in the granularity of individual graph nodes, rather than the entire graph for flexible batching. We show that LazyBatching can intelligently determine the set of nodes that can be efficiently batched together, achieving an average 15\times, 1.5\times, and 5.5\times improvement than graph batching in terms of average response time, throughput, and SLA satisfaction, respectively.-
dc.languageEnglish-
dc.publisherIEEE Computer Society-
dc.titleLazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference-
dc.typeConference-
dc.identifier.scopusid2-s2.0-85104996265-
dc.type.rimsCONF-
dc.citation.beginningpage493-
dc.citation.endingpage506-
dc.citation.publicationnameThe 27th IEEE International Symposium on High-Performance Computer Architecture (HPCA-27)-
dc.identifier.conferencecountryKO-
dc.identifier.conferencelocationVirtual-
dc.identifier.doi10.1109/HPCA51647.2021.00049-
dc.contributor.localauthorRhu, Minsoo-
dc.contributor.nonIdAuthorKim, Yunseong-
Appears in Collection
EE-Conference Papers(학술회의논문)
Files in This Item
There are no files associated with this item.

qr_code

  • mendeley

    citeulike


rss_1.0 rss_2.0 atom_1.0