Nonnegative CCA for Audiovisual Source Separation

We present a method for finding correlated components in audio and video signals. The new technique is applied to the task of identifying sources in video and separating them in audio. The concept of canonical correlation analysis is reformulated such that it incorporates nonnegativity and sparsity constraints on the coefficients of projection directions. Nonnegativity ensures that projections are compatible with an interpretation as energy signals. Sparsity ensures that coefficient weight concentrates on individual sources. By finding multiple conjugate directions we finally obtain a component based decomposition of both data modalities. Experiments effectively demonstrate the potential and benefits of this approach.