From ROS to unity: Leveraging robot and virtual environment middleware for immersive teleoperation

Virtual reality systems are often proposed as an appropriate technology for the development of teleoperational interfaces for autonomous and semi-autonomous systems. In the past such systems have typically been developed as “one off” experimental systems in part due to a lack of common software systems for both robot software development and virtual environment infrastructure. More recently, common frameworks have begun to emerge for both robot control (e.g., ROS) and virtual environment display and interaction (e.g., Unity). Here we consider the task of developing systems that integrate these two environments. A yaml-based communications protocol over web sockets is used to glue the two software environments together. This allows each system to be controlled using standard software toolkits independently while providing a flexible interface between these two infrastructures.