DC Field | Value | Language |
---|---|---|
dc.contributor.author | Lee, Hae Beom | ko |
dc.contributor.author | Lee, Hayeon | ko |
dc.contributor.author | Shin, Jae Woong | ko |
dc.contributor.author | Yang, Eunho | ko |
dc.contributor.author | Hospedales, Timothy | ko |
dc.contributor.author | Hwang, Sung Ju | ko |
dc.date.accessioned | 2022-12-05T05:02:08Z | - |
dc.date.available | 2022-12-05T05:02:08Z | - |
dc.date.created | 2022-12-05 | - |
dc.date.issued | 2022-04-25 | - |
dc.identifier.citation | 10th International Conference on Learning Representations, ICLR 2022 | - |
dc.identifier.uri | http://hdl.handle.net/10203/301655 | - |
dc.description.abstract | Many gradient-based meta-learning methods assume a set of parameters that do not participate in inner-optimization, which can be considered as hyperparameters. Although such hyperparameters can be optimized using the existing gradient-based hyperparameter optimization (HO) methods, they suffer from the following issues. Unrolled differentiation methods do not scale well to high-dimensional hyperparameters or horizon length, Implicit Function Theorem (IFT) based methods are restrictive for online optimization, and short horizon approximations suffer from short horizon bias. In this work, we propose a novel HO method that can overcome these limitations, by approximating the second-order term with knowledge distillation. Specifically, we parameterize a single Jacobian-vector product (JVP) for each HO step and minimize the distance from the true second-order term. Our method allows online optimization and also is scalable to the hyperparameter dimension and the horizon length. We demonstrate the effectiveness of our method on three different meta-learning methods and two benchmark datasets. | - |
dc.language | English | - |
dc.publisher | International Conference on Learning Representations, ICLR | - |
dc.title | Online Hyperparameter Meta-Learning with Hypergradient Distillation | - |
dc.type | Conference | - |
dc.type.rims | CONF | - |
dc.citation.publicationname | 10th International Conference on Learning Representations, ICLR 2022 | - |
dc.identifier.conferencecountry | US | - |
dc.identifier.conferencelocation | Virtual | - |
dc.contributor.localauthor | Yang, Eunho | - |
dc.contributor.localauthor | Hwang, Sung Ju | - |
dc.contributor.nonIdAuthor | Shin, Jae Woong | - |
dc.contributor.nonIdAuthor | Hospedales, Timothy | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.