An audio-visual approach to learning salient behaviors in couples´ problem solving discussions

We present a method for characterizing salient behavioral events from audio-visual data of dyadic human interactions. This behavioral signal processing work is aimed at supporting observational analysis of domain experts such as psychologists and clinicians. We extract prosodic and spectral speech features as well as visual motion vector features on overlapping windows from a multimodal corpus. We then apply a technique called multiple instance learning to detect salient audio and visual instances for predicting human expert annotated behavior ratings. We demonstrate the performance gains achieved through multimodal fusion in characterizing complex behavior patterns of interest such as blame and acceptance in recordings of couples´ problem solving discussions during marital therapy.