Towards multi-modal image-guided tumour identification in robot-assisted partial nephrectomy

Tumour identification is a critical step in robot-assisted partial nephrectomy (RAPN) during which the surgeon determines the tumour localization and resection margins. To help the surgeon in achieving this step, our research work aims at leveraging both pre- and intra-operative imaging modalities (CT, MRI, laparoscopic US, stereo endoscopic video) to provide an augmented reality view of kidney-tumour boundaries with uncertainty-encoded information. We present herein the progress of this research work including segmentation of preoperative scans, biomechanical simulation of deformations, stereo surface reconstruction from stereo endoscopic camera, pre-operative to intra-operative data registration, and augmented reality visualization.