Computationally Efficient Estimation of Squared-Loss Mutual Information with Multiplicative Kernel Models

Tomoya SAKAI  Masashi SUGIYAMA  

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E97-D   No.4   pp.968-971
Publication Date: 2014/04/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.E97.D.968
Type of Manuscript: LETTER
Category: Fundamentals of Information Systems
Keyword: 
squared-loss mutual information,  least-squares mutual information,  density ratio estimation,  multiplicative kernel models,  independence test,  

Full Text: PDF>>
Buy this Article




Summary: 
Squared-loss mutual information (SMI) is a robust measure of the statistical dependence between random variables. The sample-based SMI approximator called least-squares mutual information (LSMI) was demonstrated to be useful in performing various machine learning tasks such as dimension reduction, clustering, and causal inference. The original LSMI approximates the pointwise mutual information by using the kernel model, which is a linear combination of kernel basis functions located on paired data samples. Although LSMI was proved to achieve the optimal approximation accuracy asymptotically, its approximation capability is limited when the sample size is small due to an insufficient number of kernel basis functions. Increasing the number of kernel basis functions can mitigate this weakness, but a naive implementation of this idea significantly increases the computation costs. In this article, we show that the computational complexity of LSMI with the multiplicative kernel model, which locates kernel basis functions on unpaired data samples and thus the number of kernel basis functions is the sample size squared, is the same as that for the plain kernel model. We experimentally demonstrate that LSMI with the multiplicative kernel model is more accurate than that with plain kernel models in small sample cases, with only mild increase in computation time.