Contextual Anatomic Mimesis Hybrid In-Situ Visualization Method for Improving Multi-Sensory Depth Perception in Medical Augmented Reality

The need to improve medical diagnosis and reduce invasive surgery is dependent upon seeing into a living human system. The use of diverse types of medical imaging and endoscopic instruments has provided significant breakthroughs, but not without limiting the surgeon´s natural, intuitive and direct 3D perception into the human body. This paper presents a method for the use of augmented reality (AR) for the convergence of improved perception of 3D medical imaging data (mimesis) in context to the patient´s own anatomy (in-situ) incorporating the physician´s intuitive multi- sensory interaction and integrating direct manipulation with endoscopic instruments. Transparency of the video images recorded by the color cameras of a video see-through, stereoscopic head- mounted-display (HMD) is adjusted according to the position and line of sight of the observer, the shape of the patient´s skin and the location of the instrument. The modified video image of the real scene is then blended with the previously rendered virtual anatomy. The effectiveness has been demonstrated in a series of experiments at the Chirurgische Klinik in Munich, Germany with cadaver and in-vivo studies. The results can be applied for designing medical AR training and educational applications.