Modeling Content Structures of Domain-Specific Texts with RUP-HDP-HSMM and Its Applications

Youwei LU  Shogo OKADA  Katsumi NITTA  

IEICE TRANSACTIONS on Information and Systems   Vol.E100-D    No.9    pp.2126-2137
Publication Date: 2017/09/01
Publicized: 2017/06/09
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2017EDP7043
Type of Manuscript: PAPER
Category: Artificial Intelligence, Data Mining
hidden semi-Markov models,  content structure,  local features,  text mining,  rapid switching,  

Full Text: PDF>>
Buy this Article

We propose a novel method, built upon the hierarchical Dirichlet process hidden semi-Markov model, to reveal the content structures of unstructured domain-specific texts. The content structures of texts consisting of sequential local contexts are useful for tasks, such as text retrieval, classification, and text mining. The prominent feature of our model is the use of the recursive uniform partitioning, a stochastic process taking a view different from existing HSMMs in modeling state duration. We show that the recursive uniform partitioning plays an important role in avoiding the rapid switching between hidden states. Remarkably, our method greatly outperforms others in terms of ranking performance in our text retrieval experiments, and provides more accurate features for SVM to achieve higher F1 scores in our text classification experiments. These experiment results suggest that our method can yield improved representations of domain-specific texts. Furthermore, we present a method of automatically discovering the local contexts that serve to account for why a text is classified as a positive instance, in the supervised learning settings.