For Full-Text PDF, please login, if you are a member of IEICE,|
or go to Pay Per View on menu list, if you are a nonmember of IEICE.
Novel Superlinear First Order Algorithms
IEICE TRANSACTIONS on Fundamentals of Electronics, Communications and Computer Sciences
Publication Date: 2004/06/01
Print ISSN: 0916-8508
Type of Manuscript: PAPER
Category: Neural Networks and Bioengineering
first order optimization, steepest descent, conjugate gradient, line search subproblem, classification framework, neural networks,
Full Text: PDF>>
Applying the formerly proposed classification framework for first order line search optimization techniques we introduce novel superlinear first order line search methods. Novelty of the methods lies in the line search subproblem. The presented line search subproblem features automatic step length and momentum adjustments at every iteration of the algorithms realizable in a single step calculation. This keeps the computational complexity of the algorithms linear and does not harm the stability and convergence of the methods. The algorithms have none or linear memory requirements and are shown to be convergent and capable of reaching the superlinear convergence rates. They were practically applied to artificial neural network training and compared to the relevant training methods within the same class. The simulation results show satisfactory performance of the introduced algorithms over the standard and previously proposed methods.