Mobile visual search using image and text features

We present a mobile visual search system that utilizes both text and low bit-rate image features. Using a cameraphone, a user can snap a picture of a document image and search for the document in online databases. From the query image, the title text is detected and recognized and image features are extracted and compressed, as well. Both types of information are sent from the cameraphone client to a server. The server uses the recognized title to retrieve candidate documents from online databases. Then, image features are used to select the correct document(s). We show that by using a novel geometric verification method that incorporates both text and image feature information, we can reduce the missed positives up to 50%. The proposed method can also speed up the geometric process, enabling a larger set of verified titles, resulting in a superior performance compared to previous schemes.