Paraphrase identification models construction through fine-tuning structure transformation of BERT

Abstract

This study proposes a fine-tuning approach to effectively compare two sentences for paraphrase identification (PI) by modifying the fine-tuning structure of bidirectional encoder representations from transformers (BERT). We take the BERT and Sentence-bert models as the baseline and overcome the disadvantages of the structures used when fine-tuning the PI task. Our approach first defines three layers, namely extraction, aggregation, and attention, for the fine-tuning structure of the models, then investigates the combinations of these layers by proposing three models: Siamese-BERT$^{+}$, Siamese-BERT+Attention, and Sentence Comparison-BERT (SCBERT). Further, the advantages and disadvantages of these three models are analyzed through experiments to discover SCBERT to be the most suitable. Contrary to our initial belief, it turns out that the Siamese-BERT$^{+}$ and the Siamese-BERT+Attention models have shown lower performance than the original BERT. However, SCBERT model outperformed BERT by a 3.6% point on MRPC and 0.4% point on QQP which we believe competitive over state-of-the-art models with the same pretrained model.