Learning Curves in Learning with NoiseAn Empirical Study

Hanzhong GU  Haruhisa TAKAHASHI  

IEICE TRANSACTIONS on Information and Systems   Vol.E80-D   No.1   pp.78-85
Publication Date: 1997/01/25
Online ISSN: 
Print ISSN: 0916-8532
Type of Manuscript: PAPER
Category: Bio-Cybernetics and Neurocomputing
concept learning,  noisy learning,  generalization,  learning curves,  neural networks,  

Full Text: PDF>>
Buy this Article

In this paper, we apply the method of relating learning to hypothesis testing [6] to study average generalization performance of concept learning from noisy random training examples. A striking aspect of the method is that a learning problem with a so-called ill-disposed learning algorithm can equivalently be reduced to a simple one, and for this simple problem, even though a direct and exact calculation of the learning curves might still be impossible, a thorough empirical study can easily be performed. One of the main advantages of using the illdisposed algorithm is that it well models lower quality learning in real situations, and hence the result can provide useful implications as far as reliable generalization is concerned. We provide empirical formulas for the learning curves by simple functions of the noise rate and the sample size from a thorough empirical study, which smoothly incorporates the results from noise-free analysis and are quite accurate and adequate for practical applications when the noise rate is relatively small. The resulting learning curve bounds are directly related to the number of system weights and are not pessimistic in practice, and apply to learning settings not necessarily within the Bayesian framework.