Unsupervised Stream-Weights Computation in Classification and Recognition Tasks

In this paper, we provide theoretical results on the problem of optimal stream weight selection for the two stream classification problem. It is shown that in the presence of estimation or modeling errors using stream weights can decrease the total classification error. Specifically, we show that stream weights should be selected to be proportional to the feature stream reliability and informativeness. Next, we turn our attention to the problem of unsupervised stream weights computation in real tasks. Based on the theoretical results we propose to use models and ldquoanti-modelsrdquo (class-specific background models) to estimate stream weights. A nonlinear function of the ratio of the inter- to intra-class distance is proposed for stream weight estimation. The resulting unsupervised stream weight estimation algorithm is evaluated on both artificial data and on the problem of audiovisual speech classification. Finally, the proposed algorithm is extended to the problem of audiovisual speech recognition. It is shown that the proposed algorithms achieve results comparable to the supervised minimum-error training approach for classification tasks under most testing conditions.