NASH: on structured pruning for encoder-decoder language models

Abstract

Even though generative language models are getting popular, previous pruning studies only focused on the pruning for encoder-only models rather than generative language models. This paper investigates the considerations for structured pruning on encoder-decoder models, one of the generative language models. First, we demonstrate that the straightforward application of existing structured pruning methods to encoder-decoder models is ineffective regarding inference acceleration. In addition, we suggest two design philosophies to be considered when applying structured pruning to the encoder-decoder models: 1) the decoder depth and encoder width are the essential factor for accelerating inference and enhancing output quality, respectively 2) mitigating the training instability is important. Based on the philosophies, we propose a novel framework called NASH\,(NArrow encoder SHallow decoder) to accelerate inference of the encoder-decoder model. Extensive experiments on diverse generation and inference tasks validate the effectiveness of our method in both speedup and output quality. NASH offers a practical and efficient solution for accelerating encoder-decoder language models, enhancing their deployability in resource-constrained environments.