Searching for an activation function of CNN with a heuristic evolutionary algorithm

Abstract

All neural networks require a non-linear activation function to have effective expressive power. Then, the performance of the neural network is greatly affected by the selection of the activation function. Many researchers have attempted to find the most suitable activation function for neural networks. In this paper, we study the activation function that the convolutional neural network truly desires to have. We mainly design a novel function generator with well-known activation functions and several simple operators, and suggest the automated search method with heuristic evolutionary algorithm. Our search method found $LeakySwish$, a new activation function better than well-generalized baseline ReLU, and LeakySwish outperforms the existing activation functions under CIFAR datasets, CNN architectures, batch sizes, and data augmentation techniques. In addition, we propose two methods for parameterizing the activation function, and the shape of the learned activation function during the training process was analyzed and compared with LeakySwish. We identified the characteristics of the activation function that CNNs require, which LeakySwish also has. Through LeakySwish and the shape of the learned activation function, we expect that it will be able to propose a new activation function with a simple structure that is most suitable for CNNs.