Domain gap in adapting self-supervised depth estimation methods for stereo-endoscopy

Introduction

The task of depth estimation is a commonly encountered problem in computer vision. Beyond prevalent applications in the field of autonomous driving and robotic navigation, depth estimation also finds use for endoscopy in a surgical setup. Particularly in the case of minimally invasive surgeries, depth estimation shows promise in assisting navigation, quantification of visual information, and facilitating an improved understanding of the surface anatomy. 2D endoscopes, which are more commonly used in a surgical scenario, compromise depth perception. On the other hand, 3D endoscopes provide on-screen depth perception through stereo-imaging, from which the data can be captured intra-operatively and further analysed. Reconstructive mitral valve surgery is one such application, that is used to treat mitral valve insufficiency, a valvular disease of the heart [2]. Modern mitral valve repair is commonly done in a minimally invasive setup. In this case, intra-operative depth estimation could be extremely valuable to support decision making based on quantification [3] of endoscopic data.

Depth estimation has been tackled through various approaches in the literature. Sparse depth estimation methods focus on identifying matching feature points, or matching image patches [4]. In the case of dense depth estimation approaches, large datasets can be leveraged along with convolutional neural networks to predict depth for unseen examples. The datasets comprise of depth information acquired by depth sensors such as infrared or LiDAR cameras [5] as the ground truth to supervise the learning. In the case where the ground truth information is not available, the supervision comes from motion or binocular parallax, in other words additional information from the temporal or spatial domain [6]. In mitral valve surgery, the acquisition of ground truth depth information is unfeasible due to logistical and safety considerations. Furthermore, the off-the-shelf depth sensors typically operate in a range much wider than suitable for surgery. Besides, endoscopic scenes in the case of mitral valve repair are prone to specularities, reflection and occlusion artefacts. Occlusions occur due to tissue or instruments partially obstructing the endoscopic field of view, and may persist for a major part of the surgery. On the other hand, artefacts like specularities and reflection occur sporadically depending on the lighting conditions of the scene. The paper examines how existing self-supervised depth estimation approaches address this domain gap in endoscopy, in particular for mitral valve repair.

Related work

Dense depth estimation approaches aim to predict a depth value for each pixel of the input image. The input can either be a single frame or a sequence of frames from a monocular or stereo-camera. Learning based approaches compare patches of stereo-images to estimate correspondences and in turn the disparity, which can be used to compute the depth [7]. In endoscopy, [8] used Hybrid Recursive Matching to estimate the depth and model intra-operative surface deformation. More recently, the introduction of large datasets such as [5] enabled supervised training of convolutional neural networks for depth estimation. Some such approaches include comparing image patches to estimate the disparities using a Siamese architecture [9], and using an encoder-decoder architecture for end-to-end learning of depth from ground truth information [10]. While ground truth information is not easy to obtain in endoscopy, [11] demonstrated the use of a synthetic dataset created from CT images to train a network for depth estimation.

Self-supervised learning approaches

The approaches pertaining to monocular depth estimation without ground truth data, typically rely on a sequence of frames as a form of self-supervision. A survey of such monocular depth estimation techniques is provided by [6]. In the case of depth estimation using multi-view data, the shift information of the pixels between the views can be used to estimate the depth. [12] used a Structure-from-Motion (SfM) approach to predict depth from a monocular endoscopic video sequence. The approach uses sparse depth information as a form of supervision to generate a dense depth map.

Godard et al. [13] provided a self-supervised network that imposed left-right consistency from a stereo-pair of images during training time, to predict depth from monocular images during test time. Monodepth [1], the approach considered in this paper, further improved upon [13] by using a minimum reprojection loss, pixel auto-masking and a multi-scale encoder-decoder architecture. The network is capable of learning from both temporal (sequence of frames) and spatial (stereo-image pair) information without requiring the use of ground truth data. Therefore, this approach is chosen as a relevant candidate for depth prediction of stereo-endoscopic images from mitral valve repair. In this paper, three variants of the Monodepth [1] model (Figure 1) are trained on a combination of the KITTI [5] and Mitral Valve datasets.

Figure 1:

An overview of the different models used in [1].

Methods

Results

Figure 2 shows sample predictions on the Mitral Valve dataset of three different network variants trained on the KITTI [5] dataset. An overview of the three different models used, namely mono, stereo and mixed, is provided in Figure 1. Similarly, Figure 3 shows the three variants of the model, first trained on the KITTI [5] dataset and further finetuned on the Mitral Valve dataset. Since the model is trained to converge for a baseline of 0.1, the outputs obtained from the stereo training are scaled by a factor of 0.391, to adjust for the baseline of the stereo-endoscope.

Figure 2:

Depth prediction results on the stereo-endoscopic images from reconstructive mitral valve surgery. The network is trained on the KITTI dataset [5]. A min-max normalisation is performed on the depth map for visual representation and a reverse magma colormap is applied. Lighter areas represent regions closer to the camera and darker regions represent distant regions. B, C and D represent mono, stereo and mixed models respectively.

Figure 3:

Depth prediction results of the network trained on the stereo-endoscopic images from mitral valve repair surgery. A min-max normalisation is performed on the depth map for visual representation and a reverse magma colormap is applied. Lighter areas represent regions closer to the camera and darker regions represent distant regions. B, C and D represent mono, stereo and mixed models respectively.

The depth map predictions on the Mitral Valve Dataset are further evaluated on the test set to provide a quantitative comparison. The metrics of Mean Relative Error (MRE) and the Root Mean Square Error (RMSE) are computed for each image in the test set and then averaged over the test set. Table 1 shows the results of this evaluation and presents the comparison of the results obtained from the different experiments. Table 1 indicates that the mono model trained on the KITTI [5] dataset is the best performing model, and there is a decrease in performance when trained on the Mitral Valve dataset.

Discussion

From Figure 2, it can be seen that while predictions on the Mitral Valve dataset results in a smooth depth map, the predicted depth map does not completely align with the objects in the scene. Further training on the Mitral Valve dataset introduces texture copy artefacts in stereo and mixed training, as can be noticed from Figure 3. Additionally, while training on both datasets, the network finds it harder to adapt to the new endoscopic dataset, due to differences in the nature of the scene and the calibration parameters. In the case of stereo-endoscopy, especially in mitral valve repair, irregular lighting conditions give rise to specularities that are not constant between the views of a stereo-camera. This is in contrast to the constant light source assumption made in autonomous driving. Moreover, in autonomous driving and in other endoscopic applications such as colonoscopy, a moving camera enables the use of temporal supervision for depth estimation. The setup in mitral valve repair involves a static camera with partially moving objects in the scene.

In conclusion, it is a non-trivial task to transfer the self-supervised depth estimation methods as such from the field of autonomous driving to the stereo-endoscopic domain. It is indeed known that endoscopic scenes are more challenging than outdoor driving scenes. However, the paper performs experiments with Monodepth [1] to compare quantitatively the domain gap and describe the challenges involved in the transfer of these methods.