Hotspot Modeling of Hand-Machine Interaction Experiences from a Head-Mounted RGB-D Camera

Longfei CHEN  Yuichi NAKAMURA  Kazuaki KONDO  Walterio MAYOL-CUEVAS  

IEICE TRANSACTIONS on Information and Systems   Vol.E102-D   No.2   pp.319-330
Publication Date: 2019/02/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2018EDP7146
Type of Manuscript: PAPER
Category: Human-computer Interaction
egocentric vision,  machine operation experiences,  hotspots,  RGB-D,  task modeling,  

Full Text: PDF(1.7MB)
>>Buy this Article

This paper presents an approach to analyze and model tasks of machines being operated. The executions of the tasks were captured through egocentric vision. Each task was decomposed into a sequence of physical hand-machine interactions, which are described with touch-based hotspots and interaction patterns. Modeling the tasks was achieved by integrating the experiences of multiple experts and using a hidden Markov model (HMM). Here, we present the results of more than 70 recorded egocentric experiences of the operation of a sewing machine. Our methods show good potential for the detection of hand-machine interactions and modeling of machine operation tasks.