On the knowledge acquisition in language model pre-training

Abstract

The recent discovery that language models can store substantial factual knowledge within their parameters during pre-training has led to extensive research into understanding the factual knowledge acquired by pre-trained language models. However, relatively little research has been conducted on the specific mechanisms of how and why language models acquire factual knowledge during pre-training, despite its importance. This study addresses this gap by examining how these models acquire factual knowledge during pre-training. Through a series of targeted analytical experiments, I evaluated language models at individual factual knowledge points and monitored their progress throughout training. The findings reveal microscopic dynamics of acquisition and forgetting during training, akin to a 'tug-of-war', occurring within these models. Notably, the ability of these models to acquire and maintain factual knowledge does not show improvement throughout the progress of pre-training. This research contributes to a deeper understanding of the acquisition of factual knowledge in language models, paving the way for future advancements in their design and application.