The universal media book: tracking and augmenting moving surfaces with projected information

We explore the integration of projected imagery with a physical book that acts as a tangible interface to multimedia data. Using a camera and projector pair, a tracking framework is presented wherein the 3D position of planar pages are monitored as they are turned back and forth by a user, and data is correctly warped and projected onto each page at interactive rates. The book pages are blank, so traditional approaches to tracking physical features on the display surface do not apply. Instead, in each frame, feature points are independently extracted from the camera and projector images, and matched in order to recover the geometry of the pages in motion. The book can be loaded with multimedia content, including images, videos, and volumetric datasets (in which case a page can be removed from the book and used to navigate through a virtual 3D volume).