Enhancing RGB CNNs with depth

Most current approaches for recognition in RGB-D images fall in either the late fusion or the early fusion category. A drawback of the early fusion scheme is its inapplicability when one of the modalities is absent at test time. On the other hand, a late fusion of features does not allow the correlated nature of modalities to be exploited effectively. Recent approaches using Deep Learning are not immune to these problems either. In this work, we propose a simple, yet elegant method towards combining early and late fusion of colour and depth information when training deep Convolutional Neural Networks (CNNs). We show that when fine-tuning CNNs, an intermediate depth pre-training step provides a significant jump in colour recognition accuracy. The trends are observed consistently over several benchmark RGB-D datasets.