Keywords

1 Introduction

CT colonography (CTC) is currently performed as one of methods to find colonic polyps from CT images. If colonic polyps or early-stage cancers are found in a CTC, a colonoscopic examination or polypectomy is performed to endoscopically remove them. During a colonoscopic examination, a physician controls the colonoscope based on its camera view. However, its viewing field is unclear because the camera is often covered by fluid or the colonic wall. Furthermore, the colon changes shape significantly during colonoscope insertion. Physicians require great experience and skill to estimate how the colonoscope is traveling inside the colon. Inexperienced physicians overlook polyps or such complications as colon perforation. A colonoscope navigation system is needed that leads a physician to the polyp position. To develop a colonoscope navigation system, a colonoscope tracking method must be developed.

Endoscope tracking methods have been proposed by several research groups [1,2,3,4,5,6,7,8,9,10]. For bronchoscope tracking, image- and sensor-based methods exist. Image-based methods estimate the camera positions and movements based on 2D/3D image registrations. Registrations between temporally continuous bronchoscopic images [1] or between real and virtualized bronchoscopic images [2,3,4] are used for tracking. Sensor-based tracking methods use small position and direction sensors attached to a bronchoscope [5, 6]. For colonoscope tracking, image- and sensor-based methods also exist. The image-based method [7] has difficulty continuing to track when unclear colonoscopic views are obtained. Electromagnetic (EM) sensors are used to obtain colonoscope shapes [8, 9]. Unfortunately, they cannot guide physicians to polyp positions because they cannot map the colonoscope shape to a colon in a CT volume, which may contain polyp detection results. A colonoscope tracking method that uses CT volume and EM sensors was reported [10]. It obtains two curved lines that representing the colon and colonoscope shapes to estimate the colonoscope position on a CT volume coordinate system. This method enables real-time tracking regardless of the colonoscopic image quality. However, this method does not consider the colon deformations caused by colonoscope insertions. Large tracking errors were observed at the transverse and sigmoid colons, which are significantly deformed by a colonoscope insertion. To improve the tracking accuracy, we need to develop a method that estimates the colon shape during colonoscope insertions.

We propose a method that estimates the colon shape with the deformations caused by colonoscope insertion. The shape of the colonoscope, which is inserted into the colon, affects the colon’s deformation. We propose a shape estimation network (SEN) to model the relationships between the colon and colonoscope shapes by a deep learning framework. After training, SEN estimates the colon shape from the colonoscope shape. SEN has a long short-term memory (LSTM) layer [11], which is a recurrent neural network (RNN), to perform estimations based on temporal transitions. To make maximum use of the colonoscope shape information, we developed a relative feature of the shape. Relative, positional, and directional features are given to SEN for the estimations. We performed a phantom study to confirm the performance of the proposed method.

The followings are the contributions of this paper: (1) it propose a new deep learning framework that models the relationships between the organ shape and the forces that cause organ deformations and (2) it introduce a new relative feature that represents 3D shape information as a 2D matrix shape. The feature can be processed by convolutional neural networks (CNNs) to extract features.

2 Colon Shape Estimation Method

2.1 Overview

We estimate the colon shape from the colonoscope shape. These shapes are the temporal information that was observed during the colonoscope insertions. The estimation is performed using a SEN with CNN and LSTM layers. We extract the relative features of the colonoscope shape using CNN layers and combine them with other features that are processed by a LSTM layer. LSTM performs regression based on the temporal transition of the feature values.

2.2 Colon and Colonoscope Shape Representation

We used a point-set representation to describe the colonoscope and colon shapes. Both are represented as sets of points aligned along the colonoscope and colon centerlines. The colonoscope shape of time \(t \ (t=1, \ldots , T)\) is a set of points and directions \(\mathbf{X}^{(t)} = \{ \mathbf{p}_{n}^{(t)}, \mathbf{d}_{n}^{(t)}; \ n=1, \ldots , N \}\) related to the colonoscope (Fig. 1(a)). \(\mathbf{p}_{n}^{(t)}\) is a point aligned along the colonoscope. \(\mathbf{d}_{n}^{(t)}\) is a tangent direction of the colonoscope fiber at \(\mathbf{p}_{n}^{(t)}\). T is the total number of time frames and N is the total number of the points in the colonoscope shape. The colon shape is a set of points \(\mathbf{Y}^{(t)} = \{ \mathbf{y}_{m}^{(t)}; \ m=1, \ldots , M \}\) aligned along a colon centerline (Fig. 1(b)). M is the total number of points in the colon shape.

Fig. 1.
figure 1

(a) Green curved line represents colonoscope fiber shape. (b) shows \(\mathbf{y}_{m}^{(t)}\) on colon centerline. (c) shows positional relation feature. Distances between two points are stored in 2D-matrix \(\mathrm{P}^{(t)}\). (d) shows directional relation feature. Inner products of two directions are stored in 2D-matrix \(\mathrm{D}^{(t)}\).

Full size image

2.3 Colonoscope Shape Features

From \(\mathbf{X}^{(t)}\), we calculate the features that related to the colonoscope shape.

Structure Features. Structure feature \(\mathbf{A}^{(t)}\) includes \(\mathbf{p}_{n}^{(t)}\), \(\mathbf{d}_{n}^{(t)}\), and the insertion length of colonoscope \(l^{(t)}\), calculated as follows. We applied the Hermite spline interpolation [12] to generate a curved line that is connected to \(\mathbf{p}_{n}^{(t)}\). The curved line’s length is used as the insertion length of colonoscope \(l^{(t)}\). A structure feature is a set of these values \(\mathbf{A}^{(t)} = \{ \mathbf{p}^{(t)}_{n}, \mathbf{d}^{(t)}_{n}, l^{(t)} \}\).

Relative Features. Relative features include the positional relations between pairs of \(\mathbf{p}_{n}^{(t)}\) and the directional relations between pairs of \(\mathbf{d}_{n}^{(t)}\). Positional relation feature \(\mathrm{P}^{(t)}\) is a \(N \times N\) matrix with the distances between \(\mathbf{p}_{i}^{(t)}\) and \(\mathbf{p}_{j}^{(t)} \ (i, j=1, \ldots , N)\) as (i, j) elements (Fig. 1(c)). Directional relation feature \(\mathrm{D}^{(t)}\) is a \(N \times N\) matrix with inner products \(\mathbf{d}_{i}^{(t)} \cdot \mathbf{d}_{j}^{(t)}\) as (i, j) elements (Fig. 1(d)). \(\mathrm{P}^{(t)}\) and \(\mathrm{D}^{(t)}\) contain positional and directional relationship information.

2.4 Shape Estimation Network

We designed SEN with colonoscope shape feature input paths and the output of colon shape parameters (Fig. 2). Among the shape features, the relative features are processed by convolutional layers, which analyze the positional and directional relationship of the points on the colonoscope shape. The features extracted by the convolutional layers are combined with the structure features and given to a LSTM layer, which considers the temporal transition of all the features. To perform estimation utilizing temporal information, the SEN input is the features in a past time period \(t=t_{c}-\tau , \ldots , t_{c}-1\) until current time \(t_{c}\). The LSTM layer’s output is processed by fully connected layers. The final layer outputs estimated colon shape \(\hat{\mathbf{Y}}^{(t_{c})}\).

Fig. 2.
figure 2

Structure of shape estimation network (SEN). Input is colonoscope shape features in past time period \(t=t_{c}-\tau , \ldots , t_{c}-1\). Output is estimated colon shape of current time \(t_{c}\). Numbers written after @ are kernel or unit numbers.

Full size image

3 Experimental Setup

We confirmed the colon shape estimation performance of our method in phantom-based experiments. Our method needs pairs of \(\mathbf{X}^{(t)}\) and \(\mathbf{Y}^{(t)}\) at every time step for training SEN. \(\mathbf{X}^{(t)}\) and \(\mathbf{Y}^{(t)}\) were measured using an EM and distance sensors. We used a colon phantom (colonoscopy training model type I-B, Koken, Tokyo, Japan), a CT volume of the phantom, a colonoscope (CF-Q260AI, Olympus, Tokyo, Japan), an EM sensor (Aurora 5/6DOF Shape Tool Type 1, NDI, Ontario, Canada), and a distance image sensor (Kinect v2, Microsoft, WA, USA).

In colonoscopic examinations, physicians observe and treat the colon while retracting the colonoscope after its insertion up to the cecum. We assume the colonoscope tip is inserted up to the cecum when the colonoscope tracking starts. The proposed colon shape estimation method is also used during colonoscope tracking. The colonoscope was moved from the cecum to the anus.

3.1 Colonoscope Shape Measurement

The EM sensor is strap-shaped with six sensors at its tip and points along its strap-shaped body. Each sensor gives the 3D position and the 3D/2D direction along the colonoscope by inserting the sensor into the colonoscope working channel. The measured data are a set of points and directions \(\mathbf{X}^{(t)} = \{ \mathbf{p}_{n}^{(t)}, \mathbf{d}_{n}^{(t)}; \ n=1, \dots , 6 \}\) at time t. They are used as the colonoscope shape.

3.2 Colon Shape Measurement

We used a 3D printer to make 12 position markers to detect the surface position of the colon phantom, which has an easy-to-detect color and shape. The blue marker gives good color contrast to the orange colon phantom. The marker has a spherical shape, which enables detection from all directions. The position markers are attached to the surface of the colon phantom.

The distance image sensor is mounted to measure the surface shape of the colon phantom (Fig. 3). We obtained both distance and color images from the sensor. We applied an automated marker position extraction process to these images to obtain 12 three-dimensional points of the markers. The measured points of the markers were aligned along the colon centerline and numbered. The colon centerline was extracted from the CT volume of the colon phantom. The numbered markers are described as \(\mathbf{Y}^{(t)} = \{ \mathbf{y}_{m}^{(t)}; \ m=1, \ldots , 12 \}\) at time t. \(\mathbf{y}_{1}^{(t)}\) and \(\mathbf{y}_{12}^{(t)}\) respectively correspond to markers near the cecum and the anus. \(\mathbf{Y}^{(t)}\) is the colon shape.

3.3 Shape Estimation Network Training

We simultaneously recorded both \(\mathbf{X}^{(t)}\) and \(\mathbf{Y}^{(t)}\) during colonoscope insertions to the phantom. The measurements were performed using the experimental setup shown in Fig. 3. The shapes were recorded six times per second. Inaccurate measurement results caused by the mis-detection were manually corrected.

\(\mathbf{X}^{(t)}\) and \(\mathbf{Y}^{(t)}\) belong to the EM and distance image sensor coordinate systems. We registered them in the CT coordinate system using the iterative closest point (ICP) algorithm [13] and manual registrations. Registered \(\mathbf{X}^{(t)}\) and \(\mathbf{Y}^{(t)}\) in the CT coordinate system were used to train the SEN under these conditions: \(\tau = 20\) past frames used by the LSTM layer, 50% dropout of fully connected layers, 50 mini batch size, and 480 training epochs.

Fig. 3.
figure 3

Colonoscope and colon shapes measurement setup: colon phantom and sensors are mounted, as shown on the left.

Full size image

3.4 Colon Shape Estimation

We measured the colonoscope shapes in a past time period, \(\mathbf{X}^{(t_{c}-\tau )}, \ldots , \mathbf{X}^{(t_{c}-1)}\), during a colonoscope insertion. These colonoscope shapes were registered to the CT coordinate system using the ICP algorithm and input to the SEN to obtain estimated colon shape \(\hat{\mathbf{Y}}^{(t_{c})}\) of current time \(t_{c}\).

3.5 Evaluation Metric

We use the mean distance (MD) (mm) between \(\mathbf{Y}^{(t)}\) and \(\hat{\mathbf{Y}}^{(t)}\) as an evaluation metric. The MD of one colonoscope insertion is described as

$$\begin{aligned} E = \frac{1}{12(T-\tau )} \sum ^{T}_{t=\tau +1} \sum ^{12}_{m=1} |\hat{\mathbf{y}}^{(t)}_{m} - \mathbf{y}^{(t)}_{m}|. \end{aligned}$$
(1)

This metric indicates how an estimated colon shape is close to a ground truth.

4 Experimental Results

We evaluated the following three colon shape estimation methods: (1) our proposed method, (2) the proposed method without a relative feature, and (3) the previous method [14]. The method [14] estimates the colon shape from the colonoscope shape using regression forests and without temporal information. We recorded colonoscope and colon shapes during seven colonoscope insertions and recorded 1,179 shapes. An engineering researcher operated the colonoscope. Shapes of six colonoscope insertions were used as training data, and the remaining colonoscope insertion was used as testing data. We performed a leave-one-colonoscope-insertion-out cross validation in our evaluation. The following are the MDs of the methods: (1) 12.39, (2) 12.61, and (3) 21.41 (mm). The proposed method performed estimation with less error than the previous method (comparing (1) and (3)). Also, using the relative feature reduced the errors (comparing (1) and (2)). For methods (1) and (3), we compared distances between the ground truth (measured) and estimated colon shapes in each frame in Fig. 4(a). Estimation results of the proposed method were close to the ground truth in most of the frames. Estimated colon shapes are shown in Figs. 4(b) and (c). The shape obtained from the proposed method was similar to the ground truth.

Fig. 4.
figure 4

(a) is distances between ground truth and estimated colon shapes in each frame for proposed and previous [14] methods. Proposed method starts estimation after \(\tau = 20\) frames given. (b) and (c) show colonoscope shapes (points on green curved lines), estimated colon shapes (blue numbered points), and surface shapes of colon phantom (small dots). (b) and (c) are results of proposed and previous [14] methods.

Full size image

5 Discussion

The proposed SEN accurately and stably estimated the colon shape during colonoscope insertion. SEN utilizes not only sensor information but also relative information and temporal transition for its estimation. These features contributed to the improvement of the estimation accuracy. Estimation results can be used to improve the colonoscope tracking accuracy [10]. The results of the proposed methods are important to achieve practical colonoscope tracking methods and will also contribute to the assistance of endoscopic procedures.

Our experimental result showed one application of the proposed method, which can be used as a soft organ shape estimation method of the forces that affect organ deformation. For example, bronchus shape estimation during a bronchoscopic insertion and estimation of the organ deformation were caused by contact with surgical tools. The proposed method models the relationships between the forces and organ deformations caused by the forces. This modeling framework is applicable for many computer-assisted intervention topics. The proposed method has a potential to work on phantom and real colons, even on colonoscope operations made by different operators.

The proposed SEN can be applied to estimate human colon shapes. To do this, we need pairs of colon and colonoscope shapes during colonoscope insertions into human colons. Taking X-ray images of the abdominal region is one candidate to observe these shapes, which we believe that we can extract from such X-ray images. Once SEN is trained using human data, it estimates the colon shape. This will enable colonoscope navigation during polypectomy.

This paper proposed a colon shape estimation method using an RNN technique. SEN models the relationships between the colonoscope and colon shapes during colonoscope insertions. SEN input includes the structure and relative features of colonoscope shapes. SEN was trained to output a colon shape from these features. We applied the proposed method to estimate colon phantom shapes. The proposed method achieved more accurate and stable estimation results than the previous method. Future work includes applications to a colonoscope tracking method and estimations of the human colon shape.