Combining computer vision and video processing to achieve immersive mobile videoconferencing

In this paper we provide a technical perspective in support of the view that mobile videoconferencing is evolving towards immersiveness, with a detailed discussion of the technological readiness of the most critical components, namely the integration of depth and computer vision into the traditional video processing engine. Immersive user experiences are achieved by adding new dimensions to the perceptual space of interaction. Adding depth creates a sense of presence, and providing additional points of view creates a visual flexibility associated with real-life interaction. As the quality of service improves along with the computational capabilities of mobile platforms, we expect to see an evolution towards immersive mobile videoconferencing. Depth capture or extraction is the first element that allows 3D experiences as well as view synthesis. Depth maps or 3D models may be obtained by multiple methods but without application-driven post-processing there is no guarantee that the synthesized outputs provided to the end users will provide the expected quality of experience. The necessary convergence between video processing and computer vision also implies a shift from error-based performance metrics commonly used in computer vision, to visual quality metrics of the type used in video processing.