Language model adaptation with neural mask generator and domain knowledge graph

Abstract

In this thesis, we address methods for language model adaptation which handles the case where the pre-trained language model is transferred to the specific domain. First, we introduce the Neural Mask Generator (NMG) which generates the adaptive word maskings for language model adaptation with further pre-training based on the Masked Language Model. We empirically show that the NMG shows better adaptation performance on Natural Language Understanding (NLU) tasks across various domains compared to rule-based baselines. Second, we propose the framework that integrates the domain knowledge graph in the fine-tuning stage of the language model adaptation. We introduce the novel layer named knowledge-guided attention for integrating the knowledge into the language model. We experimentally show that the use of our framework achieves comparable or better performance on NLU tasks across various domains compared to the existing knowledge-based method and further pre-training in the language model adaptation scenario.