DC Field | Value | Language |
---|---|---|
dc.contributor.advisor | Lee, Juho | - |
dc.contributor.advisor | 이주호 | - |
dc.contributor.author | Nam, Giung | - |
dc.date.accessioned | 2023-06-22T19:31:18Z | - |
dc.date.available | 2023-06-22T19:31:18Z | - |
dc.date.issued | 2022 | - |
dc.identifier.uri | http://library.kaist.ac.kr/search/detail/view.do?bibCtrlNo=1008202&flag=dissertation | en_US |
dc.identifier.uri | http://hdl.handle.net/10203/308199 | - |
dc.description | 학위논문(석사) - 한국과학기술원 : 김재철AI대학원, 2022.8,[iii, 24 p. :] | - |
dc.description.abstract | Deep ensembles excel in large-scale image classification tasks both in terms of prediction accuracy and calibration. Despite being simple to train, the computation and memory cost of deep ensembles limits their practicability. While some recent works propose to distill an ensemble model into a single model to reduce such costs, there is still a performance gap between the ensemble and distilled models. We propose a simple approach for reducing this gap, i.e., making the distilled performance close to the full ensemble. Our key assumption is that a distilled model should absorb as much function diversity inside the ensemble as possible. We first empirically show that the typical distillation procedure does not effectively transfer such diversity, especially for complex models that achieve near-zero training error. To fix this, we propose a perturbation strategy for distillation that reveals diversity by seeking inputs for which ensemble member outputs disagree. We empirically show that a model distilled with such perturbed samples indeed exhibits enhanced diversity, leading to improved performance. | - |
dc.language | eng | - |
dc.publisher | 한국과학기술원 | - |
dc.subject | Deep Ensemble▼aEnsemble Learning▼aKnowledge Distillation | - |
dc.subject | 딥 앙상블▼a앙상블 학습▼a지식 증류 | - |
dc.title | Diversity matters when learning from ensembles | - |
dc.title.alternative | 다양성을 고려한 앙상블 학습 | - |
dc.type | Thesis(Master) | - |
dc.identifier.CNRN | 325007 | - |
dc.description.department | 한국과학기술원 :김재철AI대학원, | - |
dc.contributor.alternativeauthor | 남기웅 | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.