AI Goggles: Real-time Description and Retrieval in the Real World with Online Learning

In this paper, we present the AI Goggles system, which can instantly describe objects and scenes in the real world and retrieve visual memories about them using keywords input by the users. This is a stand-alone wearable system working on a tiny mobile computer. Also, the system can quickly learn unknown objects and scenes by teaching and learn to label and retrieve them on site, without loss of recognition ability for previously learnt ones. As the core algorithm of the system, we propose and implement a new method of multi labeling and retrieval of unconstrained real-world images. Our method outperforms the current state-of-the-art method, in terms of both accuracy and computation speed on the standard benchmark dataset. This is a major contribution to development of visual and memory assistive man-machine user interface.