Augmented reality live-action compositing

This report describes a system that performs live-action compositing of physical and virtual objects to a panoramic background image in real-time at interactive rates. A static camera is directed towards a 40 cm³ miniature stage, whose backdrop has been colored in chromatic green. Users can add virtual objects and manipulate their parameters within the scene by using a proxy device that consists of a small rod attached to a fiducial marker. Our system runs on commodity hardware such as a notebook equipped with a firewire video camera. The necessary chroma-keying and adaptive difference-matting algorithms have been implemented on a GPU using fragment shading.