Spatio-temporal layers based intra-operative stereo depth estimation network via hierarchical prediction and progressive training

Background and Objective: Safety of robotic surgery can be enhanced through augmented vision or artificial constraints to the robotl motion, and intra-operative depth estimation is the cornerstone of these applications because it provides precise position information of surgical scenes in 3D space. High-quality depth estimation of endoscopic scenes has been a valuable issue, and the development of deep learning provides more possibility and potential to address this issue.Methods: In this paper, a deep learning-based approach is proposed to recover 3D information of intra-operative scenes. To this aim, a fully 3D encoder-decoder network integrating spatio-temporal layers is designed, and it adopts hierarchical prediction and progressive learning to enhance prediction accuracy and shorten training time.Results: Our network gets the depth estimation accuracy of MAE 2.55 +/- 1.51 (mm) and RMSE 5.23 +/- 1.40 (mm) using 8 surgical videos with a resolution of 1280x1024, which performs better compared with six other state-of-the-art methods that were trained on the same data.Conclusions: Our network can implement a promising depth estimation performance in intra-operative scenes using stereo images, allowing the integration in robot-assisted surgery to enhance safety.

Automated summary

In this paper, a deep learning-based approach is proposed to recover 3D information of intra-operative scenes, which can enhance the safety of robot-assisted surgery by implementing depth estimation using stereo images.