Continually updating generative retrieval on dynamic corpora

Abstract

Generative retrieval has recently gained significant amount of recognition by the research community for its simplicity, high performance, and the ability to fully leverage the power of autoregressive models. However, the majority of prior work on generative retrieval does not consider realistic applications where temporal knowledge is accumulated over time. In this paper, we present DynamicGR, a parameter-efficient continual pre-training method that integrates dynamically changing corpora into the generative retrieval model. We conduct a comprehensive evaluation of the performance and efficiency of generative retrieval models against strong bi-encoder baselines on the StreamingQA benchmark. With the DynamicGR pretraining strategy, we demonstrate a promising performance in generative retrieval, showing an improvement of 7% over conventional parameter-efficient updates with low-rank adaptation (LoRA) on attention parameters. Furthermore, we show that DynamicGR can make generative retrieval as competitive as the bi-encoder approaches when considering performance and efficiency in dynamic scenarios. Our work will be open-sourced.