Nearest Neighbor Search using Metric-Preserving Function for Retrieval-based Dialogue System

Abstract

The Nearest Neighbor Algorithm is commonly used in Retrieval-based Dialogue Systems to find the next response candidate for Contexts. In this paper, We experiment with two metric functions for the metric tree-based Nearest Neighbor Search Algorithm to utilize in the dialog system. (1) we experiment with leveraging the embeddings of a pretrained model using cosine distance to extract contextual embeddings of dialog responses and applying them to a Nearest Neighbor Algorithm defined in metric space using metricpreserving cosine distance. (2) We measure the performance of a metric-preserving function that combines Jaccard distance for navigational terms such as specific entity names. Through this, we show the potential improvement of the response retrieval by using various metrics together in the metric treebased Nearest Neighbor Search.