Evolutionary optimization for generalized mixture neural network architecture

Abstract

Theoretical studies have shown that neural networks are capable of approximating any continuous function on a compact domain to any degree of accuracy. However, constructing a optimal neural network needs to find a good tradeoff between fitting the training data and keeping the network architecture simple. A simple neural network is not only more comprehensive and more manageable, but also less likely to overfit the training data, that is, has a higher capacity to generalize the unexperienced data. Up to now, architecture design of neural networks has been a human expert``s job for the most part. It depends heavily on the expert experience and it often needs a tedious trial-and-error process. Although considerable research on growing and pruning methods has pursued the automatic design of architectures, most works are susceptible to becoming trapped at structural local minima and those investigations are likely to be restricted to a topological subsets rather than the complete class of network architectures. Furthermore, few principled approaches exist in the literature because of the lack of a well-defined optimality criterion. In this thesis, a promising solution, a new evolutionary optimization method is proposed to provide a general, systematic, and unified way to design an optimal neural network of the generalized mixture neural network class. The generalized mixture neural network class is newly defined here to include various kinds of neural networks with a linear-in-the-parameters structure. In the evolutionary process, a use of statistical information criteria in the fitness evaluation provides a principled way of finding an optimal tradeoff between the two conflicting modelling objectives. Furthermore, a practically available Pareto optimal selection is presented to provide the "best practically available networks" for the developers of neural networks, instead of recommending the only one network regarded as the best one. Therefore, the developers ca...