Unsupervised domain adaptation in neural machine translation

Abstract

Domain mismatch between the training and test data is a well-known challenge in neural machine translation and leads to performance degradation. As the domain shifted, unknown words come out, certain words represent different meanings, and word co-occurrence statistics are changed. We focus on the problems mentioned earlier on the source language side and suggest two unsupervised domain adaptation methods that utilize an additional source language monolingual data. First, we propose a Joint Masked Sequence to Sequence (JMSS) model that shares the parameter of the conditional masked language model’s encoder and masked language model’s encoder. JMSS exploits a masked language model, and it ensures that the latent representation of the source sentence becomes robust to the source language. Next, we introduce the Sequence Margin Disparity Discrepancy (SMDD), a conditional masked language model with an auxiliary classifier for learning domain invariant representation using adversarial training. SMDD attempts to extend the unsupervised domain adaptation algorithm previously limited to classification problem to sequential problem. We also suggest the model selection method in domain adaptation in neural machine translation. We conduct domain adaptation experiments in five domains and demonstrate performance improvement in the domain adaptation tasks. Finally, we show that we can surpass Domain Adaptation by Lexicon Induction (DALI) using only source-side monolingual data.