A haptic augmented reality surgeon console for a laparoscopic surgery robot system

Robot surgery needs measures for safety and dexterous control of the surgical instrument for wider application, in spite of evident clinical efficacy in diverse surgery areas. Integration of the advanced human machine interface technologies including haptic rendering and augmented reality in surgeon console for robot-assisted laparoscopic surgery to provide enhanced safety and easier system control has been tried in this study. The surgeon console is composed of various hardware and software modules for endoscope video signal capture, image/vision signal processing, 3D deformable model handling, haptic and graphic rendering, and interface to displays and haptic devices. Intra-operative endoscopic video signal is processed to extract information for the tracking of “Object-Of-Interest (OOI)”s such as anatomic structure that needs cautious handling, bleeding site and the relative position of the surgical instruments, and displayed with overlaid image of 3D-reconstructed preoperative medical imaging data. Parts of the extracted or user-defined OOIs can be transformed into a deformable 3D model and interactively manipulated by the surgeon during the operation for intuitive information utilization. The haptic rendering provides virtual force field experience for the surgeon to have safer handling of the surgical instruments and dexterous execution of surgical task. The surgeon console framework has been implemented on a PC integrated in a laparoscopic surgery robot system under development. The system showed successfully the feasibility of the concept and further development for enhanced usability and graphical contents quality is underway.