DC Field | Value | Language |
---|---|---|
dc.contributor.advisor | Chang, Dong Eui | - |
dc.contributor.advisor | 장동의 | - |
dc.contributor.author | Bu, Fanchen | - |
dc.date.accessioned | 2022-04-27T19:31:34Z | - |
dc.date.available | 2022-04-27T19:31:34Z | - |
dc.date.issued | 2021 | - |
dc.identifier.uri | http://library.kaist.ac.kr/search/detail/view.do?bibCtrlNo=963415&flag=dissertation | en_US |
dc.identifier.uri | http://hdl.handle.net/10203/296046 | - |
dc.description | 학위논문(석사) - 한국과학기술원 : 전기및전자공학부, 2021.8,[iv, 40 p. :] | - |
dc.description.abstract | The optimization with orthogonality has been proven to be useful in training deep neural networks (DNNs). To impose orthogonality on DNNs, existing algorithms either utilize hard constraints or soft constraints. However, the methods using hard constraints are computationally expensive, and those based on soft constraints can hardly maintain the orthogonality during the whole training process. To this end, we propose a novel method, named Feedback Gradient Descent (FGD), that induces orthogonality based on the simple Euler discretization of a continuous-time dynamical system on the tangent bundle of the Stiefel manifold, showing high efficiency and stability simultaneously. Rather than using time-consuming structure-preserving discretization methods such as variational or symplectic integrators, we employ the framework of feedback integrators for the discretization. Namely, a continuous-time dynamical system is constructed in a Euclidean space containing the tangent bundle of the Stiefel manifold such that the tangent bundle becomes a local exponential attractor of the system. Since the system is in a Euclidean space, the stability of the tangent bundle is carried over to its discretized system with any off-the-shelf discretization method such as Euler, yielding the FGD algorithm that is fast in speed and stable in the preservation of the tangent bundle of the Stiefel manifold. We conduct extensive image classification experiments on popular benchmark datasets, e.g., CIFAR-10/100 and ImageNet, using various models, e.g., WideResNet and ResNet, where FGD comprehensively outperforms the existing state-of-the-art methods in terms of accuracy, efficiency, and stability. | - |
dc.language | eng | - |
dc.publisher | 한국과학기술원 | - |
dc.subject | optimization of deep neural networks▼aoptimization with orthogonality▼anumerical integration | - |
dc.subject | 심층 신경망의 최적화▼a직교성을 가진 최적화▼a수치적분 | - |
dc.title | (A) novel optimization algorithm with orthogonality for deep neural networks inspired by feedback integrators | - |
dc.title.alternative | 피드백 적분기에 영감을 받은 직교성을 가진 심층 신경망의 새로운 최적화 알고리즘 | - |
dc.type | Thesis(Master) | - |
dc.identifier.CNRN | 325007 | - |
dc.description.department | 한국과학기술원 :전기및전자공학부, | - |
dc.contributor.alternativeauthor | 복범진 | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.