Improving performance of quantitative question answering by reducing distraction in argument recognition

Abstract

Quantitative question answering, which utilizes long-form documents containing tables and textual data, is being actively studied in finance. To deal with long-form documents, such as financial statements, an architecture combining a retriever and a generator is typically used. The retriever finds evidence sentences in a given document, while the generator recognizes the proper arguments and produces an answer program. However, argument recognition suffers from a distraction problem since evidence sentences retrieved from financial statements contain many numerical data that could be candidate arguments. To address this problem, it is necessary to supervise which of the candidate arguments are required to the answer program during the generator's training process. In this paper, we propose an approach for training a generator in argument recognition by focusing on the probabilities in a candidate generation so that the arguments comprising the ground-truth have higher weights. The proposed approach consists of an argument aggregator to model the probabilities in each candidate generation, and an argument set loss to compute the cross-entropy between that probabilities and the candidates' existence in the ground-truth in terms of the argument set. In our experiments, we show performance improvements of 3.62% and 3.98% in execution accuracy and program accuracy, respectively, over the existing FinQANet model based on a financial quantitative question answering dataset. Also, we observed that the similarity of argument sets between the generated program and the ground truth improved by about 2.9%, indicating a mitigation of the distraction problem.