DC Field | Value | Language |
---|---|---|
dc.contributor.advisor | 서민준 | - |
dc.contributor.author | Jang, Joel | - |
dc.contributor.author | 장요엘 | - |
dc.date.accessioned | 2024-07-25T19:30:49Z | - |
dc.date.available | 2024-07-25T19:30:49Z | - |
dc.date.issued | 2023 | - |
dc.identifier.uri | http://library.kaist.ac.kr/search/detail/view.do?bibCtrlNo=1045744&flag=dissertation | en_US |
dc.identifier.uri | http://hdl.handle.net/10203/320556 | - |
dc.description | 학위논문(석사) - 한국과학기술원 : 김재철AI대학원, 2023.8,[ii, 31 p. :] | - |
dc.description.abstract | Large Language Models (LMs) are known to encode world knowledge in their parameters as they pretrain on a vast amount of web corpus, which is often utilized for performing knowledge-dependent downstream tasks such as question answering, fact-checking, and open dialogue. In real-world scenarios, the world knowledge stored in the LMs can quickly become outdated as the world changes, but it is non-trivial to avoid catastrophic forgetting and reliably acquire new knowledge while preserving invariant knowledge. To push the community towards better maintenance of ever-changing LMs, we formulate a new continual learning (CL) problem called Continual Knowledge Learning (CKL). We construct a new benchmark and metric to quantify the retention of time-invariant world knowledge, the update of outdated knowledge, and the acquisition of new knowledge. We adopt applicable recent methods from literature to create several strong baselines. Through extensive experiments, we find that CKL exhibits unique challenges that are not addressed in previous CL setups, where parameter expansion is necessary to reliably retain and learn knowledge simultaneously. By highlighting the critical causes of knowledge forgetting, we show that CKL is a challenging and important problem that helps us better understand and train ever-changing LMs. | - |
dc.language | eng | - |
dc.publisher | 한국과학기술원 | - |
dc.subject | 지속적인 학습▼a대규모 언어 모델▼a지식 습득▼a치명적인 망각▼a자연어처리 | - |
dc.subject | Continual learning▼aLarge language models▼aKnowledge acquisition▼aCatastrophic forgetting▼aNatural language processing | - |
dc.title | Towards continual knowledge learning of language models | - |
dc.title.alternative | 언어 모델의 지속적인 지식 학습 | - |
dc.type | Thesis(Master) | - |
dc.identifier.CNRN | 325007 | - |
dc.description.department | 한국과학기술원 :김재철AI대학원, | - |
dc.contributor.alternativeauthor | Seo, Minjoon | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.