New fine-tuning techniques to improve the capabilities of deep neural networks

Abstract

Human can learn new knowledge better by utilizing their previous experience or knowledge. In this study, we deal with techniques that can make better use of previously learned knowledge when deep neural networks acquire new knowledge like human. One of the most popular techniques in deep learning to deal with previously learned information is a fine-tuning technique. Such fine-tuning techniques have been usually used to compensate for a lack of training data of new task.

We will further develop these fine-tuning techniques and suggest ways to use the previously learned information effectively. First, we propose a technique that can be used when a fine-tuning technique fuses two already-learned networks with different characteristics. Additionally, we propose a technique to solve a catastrophic forgetting problem, which is an important issue to be solved in deep neural networks.

Firstly, our joint fine-tuning technique uses temporal information, which has useful features for recognizing facial expressions. In general, to manually design useful features requires a lot of effort. In this thesis, to reduce this effort, a deep learning technique, which is regarded as a tool to automatically extract useful features from raw data, is adopted. Our deep network is based on two different models. The first deep network extracts temporal appearance features from image sequences, while the other deep network extracts temporal geometry features from temporal facial landmark points. These two models are combined using a joint fine-tuning method in order to boost the performance of the facial expression recognition. Through several experiments, we show that the two models cooperate with each other. As a result, we achieve superior performance to other state-of-the-art methods in the CK+ and Oulu-CASIA datasets. Furthermore, we show that our new integration method gives more accurate results than traditional methods, such as a weighted summation and a feature concatenation method.

The second fine-tuning technique is a less-forgetful learning to overcome a catastrophic problem in deep neural networks. Expanding the domain that deep neural network has already learned without accessing old domain data is a challenging task because deep neural networks forget previously learned information when learning new data from a new domain. In this thesis, we propose a less-forgetful learning method for the domain expansion scenario. While existing domain adaptation techniques solely focused on adapting to new domains, the proposed technique focuses on working well with both old and new domains without needing to know whether the input is from the old or new domain. First, we present two naive approaches which will be problematic, then we provide a new method using two proposed properties for less-forgetful learning. Finally, we prove the effectiveness of our method through experiments on image classification tasks. Also, we present a new version of LF that complements the disadvantages of the original LF. The new version of LF uses EWC to update the learnable parameters of the top layer. Finally, we have achieved good results in the continual learning scenario.