Determining Indexing Strings with Statistical Analysis

Yoshiyuki TAKEDA  Kyoji UMEMURA  Eiko YAMAMOTO  

IEICE TRANSACTIONS on Information and Systems   Vol.E86-D   No.9   pp.1781-1787
Publication Date: 2003/09/01
Online ISSN: 
Print ISSN: 0916-8532
Type of Manuscript: Special Section PAPER (Special Issue on Text Processing for Information Access)
agglutinative language,  word segmentation,  adaptation,  

Full Text: PDF>>
Buy this Article

Determining indexing strings is an important factor in information retrieval. Ideally, the strings should be words that represent documents or queries. Although any single word may be the first candidate for indexing strings for an English corpus, it may not be ideal due to the existence of compound nouns, which are often good indexing strings, and which often depend on the genre of the corpus used. The situation is even worse in Japanese or Chinese where the words are not separated by spaces. In this paper, we propose a method of determining indexing strings based on statistical analysis. The novel features of our method are to make the most of the statistical measure called "adaptation" and not to use language-dependent resources such as dictionaries and stop word lists. In evaluating our method using a Japanese test collection, we found that it actually improves the precision of information retrieval systems.