Merging back-propagation and Hebbian learning rules for robust classifications

Abstract

By imposing saturation requirements on hidden-layer neural activations, a new learning algorithm is developed to improve robustness on classification performance of a multi-layer Perceptron. Derivatives of the sigmoid functions at hidden-layers are added to the standard output error with relative significance factors, and the total error is minimized by the steepest-descent method. The additional gradient-descent terms become Hebbian, and this new algorithm merges two popular learning algorithms, i.e., error back-propagation and Hebbian learning rules. Only slight modifications are needed for the standard back-propagation algorithm, and additional computational requirements are negligible. This saturation requirement effectively reduces output sensitivity to the input, which results in improved robustness and better generalization for classifier networks. Also distributed representations at hidden-layers are successfully suppressed to accomplish efficient utilization of hidden neurons. Computer simulations demonstrates much faster learning convergence as well as improved robustness for classifications and hetero-associations of binary patterns. Copyright (C) 1996 Elsevier Science Ltd