DC Field | Value | Language |
---|---|---|
dc.contributor.advisor | Kim, Junmo | - |
dc.contributor.advisor | 김준모 | - |
dc.contributor.author | Yun, Juseung | - |
dc.date.accessioned | 2023-06-23T19:33:39Z | - |
dc.date.available | 2023-06-23T19:33:39Z | - |
dc.date.issued | 2023 | - |
dc.identifier.uri | http://library.kaist.ac.kr/search/detail/view.do?bibCtrlNo=1030543&flag=dissertation | en_US |
dc.identifier.uri | http://hdl.handle.net/10203/309092 | - |
dc.description | 학위논문(박사) - 한국과학기술원 : 전기및전자공학부, 2023.2,[iv, 41 p. :] | - |
dc.description.abstract | Modern deep neural networks are equipped with normalization layers such as batch normalization or layer normalization to enhance and stabilize training dynamics. If a network contains such normalization layers, the optimization objective is invariant to the scale of the neural network parameters. The scale-invariance induces the neural network's output to be only affected by the weights' direction and not the weights' scale. We address the tuning of hyperparameters and their applications in such scale-invariant neural networks. As a first application, hyperparameter tuning in active learning is dealt with. In active learning, the number of labeled training data continues to increase as learning progresses. We propose a weight decay scheduling method suitable for active learning based on the analysis of the relationship between the number of training data and weight decay. We also propose a method of distilling knowledge from the low-performing network used in the previous round and apply it to active learning. We validate our methods on the MNIST, CIFAR-10, and CIFAR-100 datasets using convolutional neural networks of various sizes. Second, we find a common feature of good hyperparameter combinations on such a scale-invariant network, including learning rate, weight decay, number of data samples, and batch size. Our key observation is that hyperparameter setups that lead to good performance show similar degrees of angular update during one epoch. Using a stochastic differential equation, we analyze the angular update and show how each hyperparameter affects it. With this relationship, we can derive a simple hyperparameter tuning method and apply it to an efficient hyperparameter search. | - |
dc.language | eng | - |
dc.publisher | 한국과학기술원 | - |
dc.subject | Active learning▼aWeight decay▼aKnowledge distillation▼aScale-invariant network▼aNormalization▼aAngular update▼aHyperparameter tuning | - |
dc.subject | 능동 학습▼a가중치 감소▼a지식 증류▼a스케일 불변 네트워크▼a정규화▼a각도 업데이트▼a초매개변수 조율 | - |
dc.title | On the hyperparameter tuning of a scale-invariant network and its applications | - |
dc.title.alternative | 스케일 불변 네트워크의 초매개변수 조율과 그 응용 | - |
dc.type | Thesis(Ph.D) | - |
dc.identifier.CNRN | 325007 | - |
dc.description.department | 한국과학기술원 :전기및전자공학부, | - |
dc.contributor.alternativeauthor | 윤주승 | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.