Accurate Multiscale Selective Fusion of CT and Video Images for Real-Time Endoscopic Camera 3D Tracking in Robotic Surgery

Robotic surgery requires endoscope 3D tracking to navigate the endoscope in the body. This paper proposes an accurate multiscale selective fusion framework to register 2D endoscopic video images to 3D pre-operative CT data for endoscope 3D tracking. Current video-based 3D tracking depends on the performance of the 2D-3D fusion procedure that suffers from inaccurate similarity and image uncertainties. To boost video-based 3D tracking, we develop multiscale selective similarity characterization to enhance the 2D-3D fusion procedure. Such fusion not only uses image pyramids in multiple scales to represent endoscopic images but also selects specific structure information from these multiscale images to compute the similarity. We validated our method on clinical data. Our method can reduce the current tracking error from 8.9 to 5.4 mm without using any external trackers, while it provides surgeons with robust real-time surgical 3D tracking.