Speaker oriented conversation model and its evaluation

Abstract

This thesis presents new methods for generating appropriate responses of open-domain conversations and evaluating the quality of the generated responses. Open-domain conversations are casual conversations between people without limited topics. Many neural network based open-domain conversation models have seen successes in recent years. Despite these recent successes, the open-domain conversation models still have challenges to imitate the human-level conversations. One of the challenges is the consideration of speakers in the conversations.

The main contribution of this thesis is suggesting a novel speaker oriented conversation model and its evaluation metric. First, I build a new and large open-domain conversation corpus, Twitter conversation corpus. The corpus has three characteristics. First, the conversations are open-domain, as opposed to specific topics such as forum comments. Second, the conversations are naturally-occurring, as opposed to authored such as movie scripts. Third, Twitter users are socially connected among them, as opposed to task-oriented corpora.

I propose a speaker oriented conversation model, Variational Hierarchical User-based Conversation Model (VHUCM). VHUCM generates appropriate responses for given conversations and personalized responses. With the new conversation corpus, VHUCM outperforms baselines for most of the automated metrics. VHUCM also generates personalized responses based on the speakers. VHUCM solves the new user cold-start problem.

I suggest new evaluation metrics for open-domain conversation responses, Speaker Sensitive Responses Evaluation Model (SSREM). SSREM examines the conversational context and ground truth response together. It learns the model parameters from the unlabeled Twitter conversation corpus. The main idea of the approach is that it considers the speakers in defining the different levels of similar context. Experiments show that SSREM outperforms the other existing evaluation metrics in terms of high correlation with human annotation scores. I also show that SSREM trained on Twitter can be applied to movie dialogues without any additional training.