Neural Network Training Algorithm with Positive Correlation

Md. SHAHJAHAN  Kazuyuki MURASE  

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E88-D   No.10   pp.2399-2409
Publication Date: 2005/10/01
Online ISSN: 
DOI: 10.1093/ietisy/e88-d.10.2399
Print ISSN: 0916-8532
Type of Manuscript: PAPER
Category: Biocybernetics, Neurocomputing
Keyword: 
artificial neural networks,  correlation,  positive correlation,  information theory,  entropy,  hidden unit activity,  generalization,  

Full Text: PDF(776KB)>>
Buy this Article




Summary: 
In this paper, we present a learning approach, positive correlation learning (PCL), that creates a multilayer neural network with good generalization ability. A correlation function is added to the standard error function of back propagation learning, and the error function is minimized by a steepest-descent method. During training, all the unnecessary units in the hidden layer are correlated with necessary ones in a positive sense. PCL can therefore create positively correlated activities of hidden units in response to input patterns. We show that PCL can reduce the information on the input patterns and decay the weights, which lead to improved generalization ability. Here, the information is defined with respect to hidden unit activity since the hidden unit plays a crucial role in storing the information on the input patterns. That is, as previously proposed, the information is defined by the difference between the uncertainty of the hidden unit at the initial stage of learning and the uncertainty of the hidden unit at the final stage of learning. After deriving new weight update rules for the PCL, we applied this method to several standard benchmark classification problems such as breast cancer, diabetes and glass identification problems. Experimental results confirmed that the PCL produces positively correlated hidden units and reduces significantly the amount of information, resulting improved generalization ability.