Sensor-Display Registration for 3D Physical User Interaction Using a Flat-Panel Display

We present a method to register a camera-type sensor and a flat panel display to each other to build a 3D physical user interface. In the system, the sensor can not observe the display directly, which makes the registration non-trivial. We develop an auxiliary device containing a camera which can see the display and a plane pattern which can be seen by the sensor to make a transformation loop between sensors and the display. By using the transformation loop, the relative pose between the sensor and the display can be linearly estimated without any user interaction. Experiments show that the proposed method is accurate enough to build a 3D physical user interface using a commercial display and a low-cost sensor.