Fusing multiple sensors information into mixed reality-based user interface for robot teleoperation

Mixed Reality commonly refers to the merging of real and virtual worlds to produce new visualization environments where physical and digital objects co-exist and interact in real time. Mixed Reality can also be used for fusing sensor data into the existing user interface to efficiently improve situation awareness, to facilitate the understanding of surrounding environment, and to predict the future status. The work presented in this paper fuses and then represents the real video and complementary information into one single Mixed Reality interface. A simulation platform to test the Mixed Reality interface for teleoperation is also discussed in this paper.