Stochastic Divergence Minimization for Biterm Topic Models

Zhenghang CUI  Issei SATO  Masashi SUGIYAMA  

IEICE TRANSACTIONS on Information and Systems   Vol.E101-D   No.3   pp.668-677
Publication Date: 2018/03/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2017EDP7310
Type of Manuscript: PAPER
Category: Data Engineering, Web Information Systems
short text,  topic model,  biterm,  stochastic inference algorithm,  

Full Text: PDF(605.7KB)
>>Buy this Article

As the emergence and the thriving development of social networks, a huge number of short texts are accumulated and need to be processed. Inferring latent topics of collected short texts is an essential task for understanding its hidden structure and predicting new contents. A biterm topic model (BTM) was recently proposed for short texts to overcome the sparseness of document-level word co-occurrences by directly modeling the generation process of word pairs. Stochastic inference algorithms based on collapsed Gibbs sampling (CGS) and collapsed variational inference have been proposed for BTM. However, they either require large computational complexity, or rely on very crude estimation that does not preserve sufficient statistics. In this work, we develop a stochastic divergence minimization (SDM) inference algorithm for BTM to achieve better predictive likelihood in a scalable way. Experiments show that SDM-BTM trained by 30% data outperforms the best existing algorithm trained by full data.