Multimodal interaction techniques for situational awareness and command of robotic combat entities

In the next generation of Army combat vehicles, the future combat system (FCS), lightly-armored robotic vehicles are commanded by human commanders operating from within moving command vehicles, and well back from the forward edge of the battle area. Two key questions to be answered for the successful realization of FCS are how can the commander maintain situational awareness and what methods best allow the commander to interact with these robotic forces to achieve the mission goals. In our research, our approach is to immerse the commander in a virtual battlespace, with capability to directly interact with visual representations of the robotic entities. We developed a multimodal (visual, speech, and gesture) interaction method and specific techniques that allow the commander to navigate the virtual battlespace and command robotic entities while seated in a vehicle and without use of a keyboard.