For Full-Text PDF, please login, if you are a member of IEICE,|
or go to Pay Per View on menu list, if you are a nonmember of IEICE.
Determining Indexing Strings with Statistical Analysis
Yoshiyuki TAKEDA Kyoji UMEMURA Eiko YAMAMOTO
IEICE TRANSACTIONS on Information and Systems
Publication Date: 2003/09/01
Print ISSN: 0916-8532
Type of Manuscript: Special Section PAPER (Special Issue on Text Processing for Information Access)
agglutinative language, word segmentation, adaptation,
Full Text: PDF>>
Determining indexing strings is an important factor in information retrieval. Ideally, the strings should be words that represent documents or queries. Although any single word may be the first candidate for indexing strings for an English corpus, it may not be ideal due to the existence of compound nouns, which are often good indexing strings, and which often depend on the genre of the corpus used. The situation is even worse in Japanese or Chinese where the words are not separated by spaces. In this paper, we propose a method of determining indexing strings based on statistical analysis. The novel features of our method are to make the most of the statistical measure called "adaptation" and not to use language-dependent resources such as dictionaries and stop word lists. In evaluating our method using a Japanese test collection, we found that it actually improves the precision of information retrieval systems.