Differential and Algebraic Geometry of Multilayer Perceptrons

Shun-ichi AMARI  Tomoko OZEKI  

IEICE TRANSACTIONS on Fundamentals of Electronics, Communications and Computer Sciences   Vol.E84-A   No.1   pp.31-38
Publication Date: 2001/01/01
Online ISSN: 
Print ISSN: 0916-8508
Type of Manuscript: INVITED PAPER (Special Section on the 10th Anniversary of the IEICE Transactions of Fundamentals: "Last Decade and 21st Century")
information geometry,  multilayer perceptron,  singularities,  learning,  natural gradient,  

Full Text: PDF(385.7KB)>>
Buy this Article

Information geometry is applied to the manifold of neural networks called multilayer perceptrons. It is important to study a total family of networks as a geometrical manifold, because learning is represented by a trajectory in such a space. The manifold of perceptrons has a rich differential-geometrical structure represented by a Riemannian metric and singularities. An efficient learning method is proposed by using it. The parameter space of perceptrons includes a lot of algebraic singularities, which affect trajectories of learning. Such singularities are studied by using simple models. This poses an interesting problem of statistical inference and learning in hierarchical models including singularities.