Distill your own knowledge: towards ever-evolving language models via online self-distillation

Abstract

Recent studies have found that language models (LMs) pretrained on enormous amounts of text corpora can store world knowledge in their internal parameters. Despite its benefits, however, the knowledge stored in LMs can be easily outdated as the world evolves over time. In this study, we introduce a novel framework for lifelong pretraining of LMs, based on the concept of knowledge distillation. Specifically, our framework adjusts how much to distill the knowledge from a teacher’s prediction by considering its reliability. Furthermore, we demonstrate that the student model can effectively serve as its own teacher, generating highly valuable labels for training. Experiments on multiple benchmarks confirm the effectiveness and validity of our framework.