Unobtrusive measurement of subtle nonverbal behaviors with the Microsoft Kinect

We describe two approaches for unobtrusively sensing subtle nonverbal behaviors using a consumer-level depth sensing camera. The first signal, respiratory rate, is estimated by measuring the visual expansion and contraction of the user´s chest cavity during inhalation and exhalation. Additionally, we detect a specific type of fidgeting behavior, known as “leg jiggling,” by measuring high-frequency vertical oscillations of the user´s knees. Both of these techniques rely on the combination of skeletal tracking information with raw depth readings from the sensor to identify the cyclical patterns in jittery, low-resolution data. Such subtle nonverbal signals may be useful for informing models of users´ psychological states during communication with virtual human agents, thereby improving interactions that address important societal challenges in domains including education, training, and medicine.