Boosting Adapter Transfer Learning via Weak Parameter Sharing

Abstract

Adapter tuning is a parameter-efficient way of transfer learning for natural language processing, where adapter modules are inserted in a pretrained model and only the modules are updated, with the pretrained model fixed. This also allows easier continual learning, as the number of parameter linearly grows with the number of tasks. In this paper, we apply the concept of adapter tuning and further improve the method by sharing parameters among adapters plugged in different encoder layers. We attempt a lot of different configurations of parameter sharing to find the optimal setup for each GLUE task and find the settings where the model is on par with or even outperforms the default adapter tuning with just 1, 2, 3, or 4 adapters in a BERT-base model. In the experiment, we also analyze the training results to find some patterns among the configurations and their meanings in the perspective of improving transfer learning.