3D Auditory Scene Visualizer with Face Tracking: Design and Implementation for Auditory Awareness Compensation

This paper presents the design and implementation of 3D Auditory Scene Visualizer based on the visual information seeking mantra, ``overview first, zoom and filter, then details on demand´´. The machine audition system called HARK captures 3D sounds with a microphone array.The natural language processing called SalienceGraph visualizes topic transition by using discourse salience. The 3D visualizer implemented in Java 3D displays topic transition and each sound stream as a beam originating from the microphones (overview mode), shows temporal snapshots with/without specifying focusing areas (zoom-and-filter mode), and shows detailed information about a particular sound stream (details-on-demand mode). This three-mode visualization will give the user auditory awareness enhanced by HARK and SalienceGraph. In addition, a face-tracking system automatically determines the user´s intention by tracking the user´s face. The resulting system will enable users to manage and browse auditory scene files effectively, so it should acceleration and support the information explosion to compensate the lack of auditory awareness.