Camera based cross devices manipulating with augmented reality

Heterogeneous devices with large displays have become pervasive in people lives. While conveying public information, most of them are lack of support for human interaction. This paper proposes a cross devices interaction system for users to conveniently manipulate the content on public displays through personal mobile phones with camera. Users can acquire, insert, and act on elements of the displayed content with simple operations on mobile phones. Cross device elements targeting is done with an efficient visual based object matching and tracking algorithm when users capture the displayed contents as a live video. Augmented information for the elements is shown on the mobile phone screen. The content in the large displays can then be operated by clicking and dragging, etc., on the mobile phone. The proposed system can be applied to interactive advertisements, multi-user games, and conference discussions. An evaluation consisting of a series of experiments demonstrated the usability and efficiency of the proposed system.