Prototype selection for interpreting decision-making of deep temporal neural networks

Abstract

Recently, deep neural networks have widely been used in classification tasks for sequential data. However, it has not been lightened sufficiently why a neural network makes such decisions. In this paper, we suggest a new framework that selects temporal patterns from sequential data and visualizes them in order to explain the decision-making process of a deep temporal neural network. It does not require prior information such as hand-crafted segmentation labels. Our framework finds internal nodes that are highly activated in the trained temporal convolutional neural network. Subsequences that contribute to activating those nodes are characterized as representative patterns by the prototype selection method, which minimizes Maximum Mean Discrepancy between the prototypes and the total subsequences. In this process, the use of a Gaussian kernel, which is a general choice for similarity measurement, causes problems in grouping time series data. Thus, we propose a Gram kernel matrix using feature vectors in the neural network to improve the quality of selected prototypes in time series domains. The representative temporal patterns, which we call Prototypes of Temporally Activated Patterns (PTAP), show key regions in sequential data to interpret the decision-making of deep neural networks. Moreover, we analyze and visualize the role of each channel by observing input attribution with selected prototypes.