Engineering a realistic real-time conducting system for the audio/video rendering of a real orchestra

This paper describes the first multimedia system that allows users to conduct a realistic electronic orchestra. Users control tempo, dynamics, and instrument emphasis of the orchestra through natural conducting gestures with an infrared baton. Using gesture recognition and tempo adjustment algorithms, the system plays back an audio and video recording of an actual orchestra that follows the user´s conducting in real time. A major achievement of this system is that it is able to play back the audio and video recording of the orchestra at variable speed while avoiding pitch changes or other artifacts in the playback. These speed changes can be executed interactively in real time while the recording is being played back. The system has been deployed as an exhibit that has become a major attraction of a large Vienna-based music exhibition center.