Analysis of Momentum Term in Back-Propagation

Masafumi HAGIWARA  Akira SATO  

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E78-D   No.8   pp.1080-1086
Publication Date: 1995/08/25
Online ISSN: 
DOI: 
Print ISSN: 0916-8532
Type of Manuscript: PAPER
Category: Bio-Cybernetics and Neurocomputing
Keyword: 
back-propagation,  momentum term,  learning,  

Full Text: PDF(464.7KB)>>
Buy this Article



Summary: 
The back-propagation algorithm has been applied to many fields, and has shown large capability of neural networks. Many people use the back-propagation algorithm together with a momentum term to accelerate its convergence. However, in spite of the importance for theoretical studies, theoretical background of a momentum term has been unknown so far. First, this paper explains clearly the theoretical origin of a momentum term in the back-propagation algorithm for both a batch mode learning and a pattern-by-pattern learning. We will prove that the back-propagation algorithm having a momentum term can be derived through the following two assumptions: 1) The cost function is Enαn-µEµ, where Eµ is the summation of squared error at the output layer at the µth learning time and a is the momentum coefficient. 2) The latest weights are assumed in calculating the cost function En. Next, we derive a simple relationship between momentum, learning rate, and learning speed and then further discussion is made with computer simulation.