Neural networks based on hidden-layer error for improved performance and hardware implemention

Abstract

In this thesis we have developed two neural network models, which not only provide robust hetero-association and classification performance but also are suitable for hardware implementation. The multi-layer bidirectional associational associative memory (MBAM) is an extension of Kosko``s bidirectional associative memory (BAM) into multi-layer architecture, of which synapses are defined as sum of vector outer products. By storing the binary vectors, one can emulate multi-valued synapses with only binary storages for easy hardware implementations. To train the MBAM one need adjust the binary vectors by genetic algorithm or simulated annealing. A new error term is added to push the hidden-layer activations into saturation regions, which reduces effects of hard-limiting nonlinearity, instead of Sigmoid nonlinearity, at practical hardware implementations. Computer simulation and VLSI hardware implementation demonstrate much better performance than single-layer BAM. Also the bidirectional recall process provides much better error correction capability than multi-layer Perceptron. The additional hidden-layer error is also introduced to train multi-layer Perceptron for good generalization and robust classification. Good generalization is achievable only with proper combination of the training data size, the underlying problem complexity, and the network complexity. Unlike other methods which reduce network complexity to improve generalization by putting restrictions on synaptic weights, this algorithm increases complexity of the underlying problem by imposing appropriate additional requirements on hidden-layer neurons, i.e. low output sensitivity to the input values or equivalently saturation of hidden-layer activations. Learning is based on gradient-descent error minimization, and the additional gradient-descent term turns out to be Hebbian. Hence this new algorithm incorporates both the error back-propagation and Hebbian learning rules. The developed hybrid learning ...