
For FullText PDF, please login, if you are a member of IEICE,
or go to Pay Per View on menu list, if you are a nonmember of IEICE.

Polynomial Learnability of Stochastic Rules with Respect to the KLDivergence and Quadratic Distance
Naoki ABE Junichi TAKEUCHI Manfred K. WARMUTH
Publication
IEICE TRANSACTIONS on Information and Systems
Vol.E84D
No.3
pp.299316 Publication Date: 2001/03/01 Online ISSN:
DOI: Print ISSN: 09168532 Type of Manuscript: PAPER Category: Theory of Automata, Formal Language Theory Keyword: PAClearning, KLdivergence, quadraticdistance, stochastic rules, pconcepts,
Full Text: PDF(561KB)>>
Summary:
We consider the problem of efficient learning of probabilistic concepts (pconcepts) and more generally stochastic rules in the sense defined by Kearns and Schapire and by Yamanishi. Their models extend the PAClearning model of Valiant to the learning scenario in which the target concept or function is stochastic rather than deterministic as in Valiant's original model. In this paper, we consider the learnability of stochastic rules with respect to the classic 'KullbackLeibler divergence' (KL divergence) as well as the quadratic distance as the distance measure between the rules. First, we show that the notion of polynomial time learnability of pconcepts and stochastic rules with fixed range size using the KL divergence is in fact equivalent to the same notion using the quadratic distance, and hence any of the distances considered in [6] and [18]: the quadratic, variation, and Hellinger distances. As a corollary, it follows that a wide range of classes of pconcepts which were shown to be polynomially learnable with respect to the quadratic distance in [6] are also learnable with respect to the KL divergence. The sample and time complexity of algorithms that would be obtained by the above general equivalence, however, are far from optimal. We present a polynomial learning algorithm with reasonable sample and time complexity for the important class of convex linear combinations of stochastic rules. We also develop a simple and versatile technique for obtaining sample complexity bounds for learning classes of stochastic rules with respect to the KLdivergence and quadratic distance, and apply them to produce bounds for the classes of probabilistic finite state acceptors (automata), probabilistic decision lists, and convex linear combinations.

