Kodály Musical Hand Signs Recognition without Visual Background Modeling

In this study, we develop a novel vision-based Kodály musical hand signs recognition system to recognize the gestures of the musical notes. Vision-based gesture recognitions often face the following problems. First, the illumination change can influence the hand detections. Second, the hand tracking will become difficult under the complex background. To overcome the aforementioned problems, we propose several novel technologies to overcome these problems. The first one is the block-based foreground detection method in which the difference between consecutive frames of moving hand can be identified. The second one is the dual foregrounds fusion method that can generate the precise hand regions. The third one is the texture-based fist tracking method that can locate the fist position precisely without the influence of illumination variations. After the fist locating, the skin color detection is applied to extract the complete hand region and then the various kind of Kodály musical hand signs can be recognized with the moment invariants and support vector machines. The experimental results show that the hand can be tracked with the accuracy 95.71% and efficiency 20 fps under the complex background. The recognition accuracy for the Kodály musical gestures is about 97%.