For Full-Text PDF, please login, if you are a member of IEICE,|
or go to Pay Per View on menu list, if you are a nonmember of IEICE.
An Objective Method for Evaluating Speech Translation System: Using a Second Language Learner's Corpus
Keiji YASUDA Fumiaki SUGAYA Toshiyuki TAKEZAWA Genichiro KIKUI Seiichi YAMAMOTO Masuzo YANAGIDA
IEICE TRANSACTIONS on Information and Systems
Publication Date: 2005/03/01
Print ISSN: 0916-8532
Type of Manuscript: Special Section PAPER (Special Section on Corpus-Based Speech Technologies)
Category: Speech Corpora and Related Topics
evaluation of speech translation systems, BLEU, TOEIC,
Full Text: PDF>>
In this paper we propose an objective method for assessing the capability of a speech translation system. It automates the translation paired comparison method, which gives a simple, easy to understand TOEIC score proposed by Sugaya et al., to succinctly evaluate a speech translation system. To avoid the expensive evaluation cost of the original method where large manual effort is required, the new objective method automates the procedure by employing an objective metric such as BLEU and DP-based measure. The evaluation results obtained by the proposed method are similar to those of the original method. Also, the proposed method is used to evaluate the usefulness of a speech translation system. It is then found that our speech translation system is useful in general, even to users with higher TOEIC score than the system's.