For Full-Text PDF, please login, if you are a member of IEICE,|
or go to Pay Per View on menu list, if you are a nonmember of IEICE.
Hybrid Method of Data Collection for Evaluating Speech Dialogue System
Shu NAKAZATO Ikuo KUDO Katsuhiko SHIRAI
IEICE TRANSACTIONS on Information and Systems
Publication Date: 1996/01/25
Print ISSN: 0916-8532
Type of Manuscript: PAPER
Category: Speech Processing and Acoustics
spoken-dialogue system, Wizard of OZ, data collection, evaluation,
Full Text: PDF(518.2KB)>>
In this paper, we propose a new method of dialogue data collection which can be used to evaluate modules of a spoken dialogue system. To evaluate the module, it is necessary to use suitable data. Human-human dialogue data have not been appropriate to module evaluation, because spontaneous data usually include too much specific phenomena such as fillers, restarts, pauses, and hesitations. Human-machine dialogue data have not been appropriate to module evaluation, because the dialogue was unnatural and the available vocabularies were limited. Here, we propose 'Hybrid method' for the collection of spoken dialogue data. The merit is that, the collected data can be used as test data for the evaluation of a spoken dialogue system without any modification. In our method a human takes the role of some modules of the system and the system, also, works as the other part of the system together. For example, humans works as the speech recognition module and the dialogue management and a machine does the other part, response generation module. The collected data are good for the evaluation of the speech recognition and the dialogue management modules. The reasons are as follows. (1) Lexicon: The lexicon was composed of limited words and dependent on the task. (2) Grammar: The intention expressed by the subjects were concise and clear. (3) Topics: There were few utterances outside the task domain. The collected data can be used test data for the evaluation of a spoken dialogue system without any modification.