DC Field | Value | Language |
---|---|---|
dc.contributor.author | Lee, Sangmin | ko |
dc.contributor.author | Sim, Byeongsu | ko |
dc.contributor.author | Ye, Jong Chul | ko |
dc.date.accessioned | 2024-08-29T02:00:09Z | - |
dc.date.available | 2024-08-29T02:00:09Z | - |
dc.date.created | 2024-07-25 | - |
dc.date.issued | 2024-10 | - |
dc.identifier.citation | NEURAL NETWORKS, v.178 | - |
dc.identifier.issn | 0893-6080 | - |
dc.identifier.uri | http://hdl.handle.net/10203/322448 | - |
dc.description.abstract | Understanding the training dynamics of deep ReLU networks is a significant area of interest in deep learning. However, there remains a lack of complete elucidation regarding the weight vector dynamics, even for single ReLU neurons. To bridge this gap, our study delves into the training dynamics of the gradient flow w(t) for single ReLU neurons under the square loss, dissecting it into its magnitude ||w(t)|| and angle phi(t) components. Through this decomposition, we establish upper and lower bounds on these components to elucidate the convergence dynamics. Furthermore, we demonstrate the empirical extension of our findings to general two-layer multi-neuron networks. All theoretical results are generalized to the gradient descent method and rigorously verified through experiments. | - |
dc.language | English | - |
dc.publisher | PERGAMON-ELSEVIER SCIENCE LTD | - |
dc.title | Magnitude and angle dynamics in training single ReLU neurons | - |
dc.type | Article | - |
dc.identifier.wosid | 001266414300001 | - |
dc.identifier.scopusid | 2-s2.0-85197494452 | - |
dc.type.rims | ART | - |
dc.citation.volume | 178 | - |
dc.citation.publicationname | NEURAL NETWORKS | - |
dc.identifier.doi | 10.1016/j.neunet.2024.106435 | - |
dc.contributor.localauthor | Ye, Jong Chul | - |
dc.description.isOpenAccess | N | - |
dc.type.journalArticle | Article | - |
dc.subject.keywordAuthor | Single ReLU neurons | - |
dc.subject.keywordAuthor | Magnitude and angle dynamics | - |
dc.subject.keywordAuthor | Gradient flow | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.