DC Field | Value | Language |
---|---|---|
dc.contributor.advisor | 서민준 | - |
dc.contributor.author | Chang, Hoyeon | - |
dc.date.accessioned | 2024-07-30T19:30:36Z | - |
dc.date.available | 2024-07-30T19:30:36Z | - |
dc.date.issued | 2024 | - |
dc.identifier.uri | http://library.kaist.ac.kr/search/detail/view.do?bibCtrlNo=1096049&flag=dissertation | en_US |
dc.identifier.uri | http://hdl.handle.net/10203/321344 | - |
dc.description | 학위논문(석사) - 한국과학기술원 : 김재철AI대학원, 2024.2,[iv, 27 p. :] | - |
dc.description.abstract | The recent discovery that language models can store substantial factual knowledge within their parameters during pre-training has led to extensive research into understanding the factual knowledge acquired by pre-trained language models. However, relatively little research has been conducted on the specific mechanisms of how and why language models acquire factual knowledge during pre-training, despite its importance. This study addresses this gap by examining how these models acquire factual knowledge during pre-training. Through a series of targeted analytical experiments, I evaluated language models at individual factual knowledge points and monitored their progress throughout training. The findings reveal microscopic dynamics of acquisition and forgetting during training, akin to a 'tug-of-war', occurring within these models. Notably, the ability of these models to acquire and maintain factual knowledge does not show improvement throughout the progress of pre-training. This research contributes to a deeper understanding of the acquisition of factual knowledge in language models, paving the way for future advancements in their design and application. | - |
dc.language | eng | - |
dc.publisher | 한국과학기술원 | - |
dc.subject | 자연어처리▼a언어 모델▼a사전학습▼a사실적 지식의 습득 | - |
dc.subject | Natural language processing▼aLanguage model▼aPre-training▼aFactual knowledge acquisition | - |
dc.title | On the knowledge acquisition in language model pre-training | - |
dc.title.alternative | 언어모델 사전학습에서의 지식 습득에 대한 연구 | - |
dc.type | Thesis(Master) | - |
dc.identifier.CNRN | 325007 | - |
dc.description.department | 한국과학기술원 :김재철AI대학원, | - |
dc.contributor.alternativeauthor | 장호연 | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.