MaghBERT: pre-trained language models for the Maghrebi dialects

Abstract

Random Masked Language Models have brought a significant performance boost to various natural language processing tasks. However, their performance is bounded by the raw corpus domain. As such, target domains that shift notably from the source domain generally perform poorly, even if they share a considerable amount of semantics. A particular case of such a paradigm can be seen with Modern Standard Arabic Language and the Arabic Dialects. Although both share a large proportion of semantics, a pre-trained model on the standardized variant fails to perform adequately on the latter. A typical solution to address this problem is to introduce the pre-trained model to the target domain through another round of pre-training, a process known as domain adaptation. However, recent domain adaptation techniques fail to deal with noisy target data. Thus limiting their learnability, all while harming the representation of the source domain. To address these issues, we propose a semi-supervised masking strategy that leverages a relatively small set of supervised signals to extract various Term Weighting schemes such as Information Gain and Odds Ratio. During domain adaptive pre-training, sentence-level weights are merged using an ensemble ranking approach and then used to pick masking candidates over a non-uniform distribution. Furthermore, we show that at inference level, a pre-trained model and a target test corpus can be effectively used to find adequate collection frequencies before any domain adaptation or pre-training. The overall effectiveness of our approach is further reflected in various downstream tasks against multiple pre-trained dialectal models, as well as current domain-adaptation strategies.