Multimodal Interface for Drawing Diagrams that Does not Interfere with Natural Talking and Drawing

Xingya XU  Hirohito SHIBATA  

IEICE TRANSACTIONS on Electronics   Vol.E102-C   No.5   pp.408-415
Publication Date: 2019/05/01
Online ISSN: 1745-1353
DOI: 10.1587/transele.2018ECP5054
Type of Manuscript: PAPER
Category: Electronic Displays
multimedia interface,  interaction technique,  speech recognition,  gesture recognition,  

Full Text: PDF>>
Buy this Article

The aim of this research is to support real-time drawingin talking by using multimodal user interface technologies. In this situation, if talking and drawing are considered as commands by mistake during presentation, it will disturb users' natural talking and drawing. To prevent this problem, we introduce two modes of a command mode and a free mode, and explore smooth mode switching techniques that does not interfere with users' natural talking and drawing. We evaluate four techniques. Among them, a technique that specifies the command mode after actions using a pen gesture was the most effective. In this technique, users could quickly draw diagrams, and specifying mode switching didn't interfere with users' natural talk.