
For FullText PDF, please login, if you are a member of IEICE,
or go to Pay Per View on menu list, if you are a nonmember of IEICE.

Generalization Performance of Subspace Bayes Approach in Linear Neural Networks
Shinichi NAKAJIMA Sumio WATANABE
Publication
IEICE TRANSACTIONS on Information and Systems
Vol.E89D
No.3
pp.11281138 Publication Date: 2006/03/01
Online ISSN: 17451361 Print ISSN: 09168532 Type of Manuscript: PAPER Category: Algorithm Theory Keyword: empirical Bayes, variational Bayes, neural networks, reducedrank regression, JamesStein, unidentifiable,
Full Text: PDF(330.2KB) >>Buy this Article
Summary:
In unidentifiable models, the Bayes estimation has the advantage of generalization performance over the maximum likelihood estimation. However, accurate approximation of the posterior distribution requires huge computational costs. In this paper, we consider an alternative approximation method, which we call a subspace Bayes approach. A subspace Bayes approach is an empirical Bayes approach where a part of the parameters are regarded as hyperparameters. Consequently, in some threelayer models, this approach requires much less computational costs than Markov chain Monte Carlo methods. We show that, in threelayer linear neural networks, a subspace Bayes approach is asymptotically equivalent to a positivepart JamesStein type shrinkage estimation, and theoretically clarify its generalization error and training error. We also discuss the domination over the maximum likelihood estimation and the relation to the variational Bayes approach.

