A GUI-Interaction Aiding System for Cut-and-Paste Operation Based on Image Processing for the Visually Impaired

Alberto TOMITA,Jr.  Tsuyoshi EBINA  Rokuya ISHII  

IEICE TRANSACTIONS on Information and Systems   Vol.E81-D   No.9   pp.1019-1024
Publication Date: 1998/09/25
Online ISSN: 
Print ISSN: 0916-8532
Type of Manuscript: PAPER
Category: Image Processing,Computer Graphics and Pattern Recognition
human interface,  GUI,  visual impairment,  image processing,  

Full Text: PDF(553KB)>>
Buy this Article

In this paper we propose a method to aid a visually impaired person in the operation of a computer running a graphical user interface (GUI). It is based on image processing techniques, using images taken by a color camera placed over a Braille display. The shape of the user's hand is extracted from the image by analyzing the hue and saturation histograms. The orientation of the hand, given by an angle θ with the vertical axis, is calculated based on central moments. The image of the hand is then rotated to a normalized position. The number of pixels in each column of the normalized image is counted, and the result is put in a histogram. By analyzing the coefficient of asymmetry of this histogram, it can be determined whether the thumb is positioned along the pointing finger, or whether it is far from the other fingers. These two positions define two states that correspond to a mouse button up or down. In this way, by rotating the hand and moving the thumb, we can emulate the acts of moving a scroll bar and depressing a mouse button, respectively. These operations can be used to perform tasks in a GUI, such as cut-and-paste, for example. Experimental results show that this method is fast and efficient for the proposed application.