Handling session mismatch by fusion-based co-training: An empirical study using face and speech multimodal biometrics

Semi-supervised learning has been shown to be a viable training strategy for handling the mismatch between training and test samples. For multimodal biometric systems, classical semi-supervised learning strategies such as self-training and co-training may not have fully exploited the advantage of a multimodal fusion, notably due to the fusion module. For this reason, we explore a novel semi-supervised training strategy known as fusion-based co-training that generalizes the classical co-training such that it can use a trainable fusion classifier. Our experiments on the BANCA face and speech database show that this proposed strategy is a viable approach. In addition, we also address the resolved issue of how to select the decision threshold for adaptation. In particular, we find that a strong classifier, including a multimodal system, may benefit better from a more relaxed threshold whereas a weak classifier may benefit better from a more stringent one.