Developing a smart camera for gesture recognition in HCI applications

Smart cameras are becoming more popular in human computer interaction (HCI). One of HCI research areas, multimodal user interface (MMUI) allows user to interact with a computer by using his or her natural communication modalities, such as speech, pen, touch, gestures, eye gaze, and facial expression. This paper presents the hardware and software co-design and implementation of KLT (Kanade Lucas Tomasi) tracking algorithm in a FPGA-based smart camera prototype for recognize simple hand gestures consisting of a CMOS image sensor capture unit and FPGA main video processor. This tracking system uses face and hand detections as a tool to detect and track gesture (face and hand motion). This gesture tracking system that are based on Harris keypoint detection algorithm and KLT tracking algorithm has been successfully implemented in FPGA and shows promising result in real time performance.