Exploiting numerical-contextual knowledge to improve numerical reasoning over text in question answering

Abstract

Numerical reasoning over text is a challenging subtask in question answering (QA) that requires both the understanding of texts and numbers. However, existing language models that are often used as encoders in these numerical reasoning QA models tend to overly rely on the pre-existing parametric knowledge instead of relying on the given context to interpret such numbers in the given text. Our work proposes a novel attention masked reasoning model, the NC-BERT, that learns to leverage the number-related contextual knowledge to enhance the numerical reasoning capabilities of the QA model. The empirical results suggest that understanding of numbers in their context, and refining numerical information in the number embeddings lead to improved numerical reasoning accuracy and performance in DROP, a numerical QA dataset.