Immersive Virtual Colonography Viewer for Colon Growths Diagnosis: Design and Think-Aloud Study

Abstract

Desktop-based virtual colonoscopy is a proven and accurate process for identifying colon abnormalities. However, it is time-consuming. Faster, immersive interfaces for virtual colonoscopy are still incipient and need to be better understood. This article introduces a novel design that leverages VR paradigm components to enhance the efficiency and effectiveness of immersive analysis. Our approach contributes a novel tool highlighting unseen areas within the colon via eye-tracking, a flexible navigation approach, and a distinct interface for displaying scans blended with the reconstructed colon surface. The path to evaluating and validating such a tool for clinical settings is arduous. This article contributes a formative evaluation using think-aloud sessions with radiology experts and students. Questions related to colon coverage, diagnostic accuracy, and time to complete are analyzed with different user profiles. Although not aimed at quantitatively measuring performance, the experiment provides lessons learned to guide other researchers in the field.

1. Introduction

Colorectal cancer (CRC) ranks as the third most common cause of cancer-related deaths globally and in the United States. In 2023 alone, it is projected that around 153,020 individuals will receive a CRC diagnosis, and approximately 52,550 individuals will sadly lose their lives to this disease [1]. In this context, the prevention and treatment of colorectal cancer are strongly related to the early detection of adenomas, polyps, and CRC in its early stages, with traditional or total colonoscopy (TC) being the gold standard for their identification and removal [2,3]. However, it can be invasive, uncomfortable, and expensive.

In recent years, significant medical imaging technology advancements have revolutionised how various medical conditions are diagnosed and treated. One such innovation is computed tomography (CT) colonography, also called virtual colonoscopy. However, unlike other CT-based exams, the detection of colon polyps and diverticula cannot be easily accomplished solely by examining sliced image volumes.

The 3D reconstruction of the colonic mucosa is presented on a desktop display for radiologists, offering a view as if stationed along the median of the colon. In conjunction with a broad visual field, this facilitates a tunnel-like experience (refer to Figure 1, left), enhancing the visibility of neoplasms within the colon and enabling precise measurement by specialists.

Over the past two decades, integrating virtual reality into scientific and data visualization has yielded remarkable advancements [4,5]. Virtual reality’s capabilities, including immersive presence, natural surveying, seamless navigation, and direct interaction, contribute significantly to the efficacy and concentration of visualization tasks, particularly when dealing with multidimensional datasets. Furthermore, thanks to economies of scale, VR headsets with eye-tracking have become more affordable than the specialized medical desktop monitors they aim to replace [6].

In this scenario, immersive virtual colonography (IVC) is an innovative VR approach to simulate colonoscopies. The procedure is similar to a conventional virtual colonography. Still, the radiologist wears a headset and can navigate inside a 3D reconstruction of the patient’s colon and rectum, using natural movements to look around and identify polyps. IVC has the potential to be a less invasive, more cost-effective, and more accessible screening method for colorectal cancer. Despite the seemingly simple task of adapting a desktop viewer for an immersive environment, only a handful of studies explore immersive interfaces for colon diagnostic visualization. Moreover, there are no comprehensive guidelines for developing these visualizations and their associated interactive functionalities. Some of the challenges still hindering the adoption of IVC are the lack of orientation cues, the absence of tools to verify coverage of the whole lumen, and the poor integration of CT data in the process for accurate diagnosis.

Addressing these challenges and adding to the expanding research on IVC, we discuss the design of an immersive viewer to detect colon growths. This viewer employs cutting-edge technology to generate a detailed and interactive three-dimensional model of the colon, enabling physicians to detect and scrutinize growths more thoroughly. By doing so, it holds the potential to enhance the precision and efficiency of diagnosing colon growths, providing a more patient-friendly and minimally invasive option compared to conventional colonoscopy methods and existing desktop-based solutions.

This paper details the outcomes of our work on a novel VR application. This technique immerses users, allowing them to navigate through a virtual representation of the entire colon to identify clinically significant observations. Notably, as a novel development, our VR approach integrates eye-tracking technology and the ability to view CT scans using VR metaphors. Our ultimate goal is to make the radiologist’s task less time-consuming and at least as accurate so that the procedure can be made more widely accessible to patients. We also present the results from applying a user-centered assessment protocol to investigate the usability of these technologies in the IVC context and explore radiologists’ attitudes toward their use. In sessions with specialists, we also evaluate the effectiveness of these technologies in identifying and diagnosing colon abnormalities.

In the remainder of the text, we review related work, present our prototype’s design and construction, describe the experimental evaluation, and synthesize the results into a lessons-learned list.

2. Background and Related Work

In the initial part of this section, we briefly introduce the types of colon anomalies that a diagnostic procedure is expected to find. Then, as specific immersive virtual colonoscopy literature is rather scarce, we include other relevant related elements in the second part of this literature review section. We explore topics like 3D navigation and eye-tracking techniques and build on top of desktop-based virtual colonography.

2.1. Colon Anomalies

Colon examinations, whether conducted via endoscopy, virtual reality, or immersive technology, share a common goal: to detect and classify abnormalities within the colon. Such anomalies can occur in anyone and are classified into five distinct categories:

i. Adenomatous (tubular adenoma)—Predominantly benign and the most frequently encountered colon polyp variant. There is potential for these to morph into malignant forms over several years, which preventative regular screenings can avert.

ii. Serrated— serrated polyps may or may not evolve into cancerous formations depending on their dimensions and placement. Large, typically flat polyps located in the proximal colon are considered to be in the precancerous stage. Their specific traits make them difficult to detect.

iii. Hyperplastic—Termed hyperplastic when a serrated polyp is diminutive and situated in the distal colon. These are seldom associated with malignancy.

iv. Inflammatory—Most frequently arising in individuals with inflammatory bowel disease, also called pseudopolyps, stemming from persistent colon inflammation. Inflammatory polyps are non-malignant and generally do not progress to cancerous forms.

v. Villous Adenoma (tubulovillous adenoma)—Typically sessile and bearing a heightened likelihood of malignancy, these polyps can be removed via colonoscopy if small. However, larger ones might necessitate surgical intervention for complete excision.

Detecting all polyp varieties, with adenomas particularly prone to cancerous transformation, is imperative. However, flat adenomas present detection challenges. Additionally, serrated polyps significantly contribute to colorectal cancer risk, as they may be too diminutive for detection, particularly following inadequate preparatory measures [7].

2.2. Virtual Colonography

Total colonoscopy (TC) remains the gold standard for detecting colorectal cancer despite its invasiveness and associated drawbacks, including discomfort, preparation requirements, and risk of perforation due to the necessity of inserting a scope through the rectum.

Virtual colonography (VC), developed as an alternative to mitigate some of TC’s downsides [8], utilizes CT scans and desktop software to enable radiologists to traverse a digitally reconstructed colon in both directions—from the rectum to the cecum and back—to ensure no abnormalities are overlooked in the colon’s folds [9]. Although VC addresses the issue of invasiveness, it introduces its own set of limitations. The navigation is limited to a singular perspective (refer to Figure 2), increasing the uncertainty about polyp types and, consequently, leading to more referrals for conventional colonoscopies and increased costs [10,11]. While VC can identify most colonic anomalies, it requires significantly more time than TC. Furthermore, VC focuses solely on the colon’s interior, potentially missing extracolonic lesions and necessitating additional tests, escalating expenses and the likelihood of false positives.

2.3. Immersive Virtual Colonography

Following the establishment of virtual colonography (VC) as an effective tool in clinical settings [12], the research community shifted focus towards developing innovative methods for data exploration and analysis [13,14,15]. Subsequently, IVC was introduced [16]. In IVC, a three-dimensional model of the colon is generated from CT scans akin to VC. However, the 3D representation of the colon is navigated using a virtual reality setup (Figure 3), enabling radiologists to examine both the interior and exterior of the colon through intuitive movements, enhancing volume data analysis with depth perception and motion parallax, while retaining VC’s capabilities [17]. IVC offers distinct benefits over traditional VC, including improved efficiency in search tasks, as evidenced by 3D visualization studies [18,19]. The freedom from fixed paths and views opens up numerous possibilities, including direct 3D measurements. Despite these advancements, radiologists prioritize practical VR interactions over novelty. The primary challenge in designing an IVC VR interface lies in balancing efficiency with high fidelity [20], aiming to reduce examination time without sacrificing quality. This paper will later discuss our approach to designing efficient interactions. We begin by examining how previous designs have tackled navigation and interface challenges.

2.3.1. Navigation

One of IVC’s earliest and most commonly used navigation techniques is fly-through. This technique was borrowed from VC [21]. It involves linearly navigating through the colon, from the anus to the cecum, by moving the virtual camera along the center line of the colon.

Fly-through is simple and easy to use, but it may miss small lesions or polyps not directly visible from the center line [22]. On the other hand, IVC allows for more natural navigation with physical walking and looking around. However, free natural navigation has a higher degree of disorientation [23]. To assist with spatial awareness, the world-in-miniature (WIM) technique [24] has been proposed, where the colon is represented in a scaled-down version that can fit into a VR headset [16].

Another challenge of IVC navigation is to provide smooth and accurate travel to reduce cybersickness. To do so, several speed control techniques have been proposed, including fixed, adaptive, and path-based methods [25].

Besides fly-through navigation, one popular approach is the “point-and-teleport” method [26], where users point to a location in the virtual environment and teleport to that location. This method is simple and easy to use but can be immersion-breaking, as it requires users to constantly shift their attention from the virtual environment to the user interface.

2.3.2. Annotation

Regarding the classification of polyps, measurements are necessary. Since it is such an important tool in diagnosis, many studies have been performed and assessed [27,28]. The few existing IVC viewers in the literature use ray casting to indicate points on the colon surface for annotation and measurement. While this is natural and follows the mouse pointer paradigm, which is widely dominant, pointing in 3D is harder and less precise. This is another challenge for IVC.

2.3.3. CT Slice Referencing

CT scan viewing techniques are crucial for IVC, as they allow physicians to examine the density of the colon tissue, which is part of discriminating polyps from other features. Several CT scan viewing techniques have been proposed, such as multiplanar reformation (MPR) and maximum intensity projection (MIP) [29,30]. MPR allows the user to view the colon from different angles, while MIP creates a 3D visualization of the colon by projecting the maximum density values along each ray. MPR is implemented in VC as three views in the canonical orientation of the location being examined (see Figure 2). This is not suitable for immersive interaction and remains a challenge.

2.4. Coverage

The surgeon or radiologist must inspect the full length of the colon on either TC, VC, or IVC. Coverage in TC is only measured by the endoscope attaining the cecum region and the duration of the inspection. The midline is often used in VC to determine if every location was visited. In IVC, the current standard uses head tracking to predict eye gaze direction, which allows for estimation of the areas covered by the eye gaze. None of these approaches informs where the expert looked and attended to. This section provides an overview of the existing literature on eye tracking for diagnosis, highlighting its applications, methodologies, and findings.

Eye tracking has emerged as a valuable tool in medical diagnosis. It provides insights into visual attention, cognitive processes, and decision-making during the diagnostic process. By tracking eye movements and gaze patterns, researchers and clinicians can better understand how healthcare professionals interpret medical images, identify abnormalities, and make diagnostic decisions.

For instance, studies have shown that radiologists focus more on regions of interest and anomalies in the images, indicating the importance of localized visual attention in diagnosis [31,32]. Eye tracking has also shown promise in diagnostic training by providing insights into the visual strategies experts and novices employ. It has been employed to assess the effectiveness of training interventions by monitoring changes in gaze behavior and diagnostic performance over time [31,33].

In addition to its role in understanding diagnosis processes, eye tracking has been explored for developing decision support systems in medical contexts. These decision support systems have the potential to enhance diagnostic accuracy, reduce cognitive biases, and improve patient outcomes [31,33].

3. Prototype Development

This section presents the design and implementation of our IVC system. We used Unity, which offers a cutting-edge approach to simulating the colonoscopy procedure in a realistic and interactive virtual environment. We presented a preliminary version of the system design at the IEEE Virtual Reality XR for Healthcare and Wellbeing Workshop [34].

3.1. User Interface Overview

The radiologist starts with a view from outside of the colon, as in Figure 4b. They can rotate the colon for external inspection, point to the colon to teleport to the pointed position, or simply switch to the inside view (teleport to the rectum). Once inside, they can move using the “fly-through over the center line” mechanism, which allows doctors to navigate seamlessly through the colon by following its central path. The controller joystick changes the speed and direction of the fly-through. When charged to the maximum, the joystick triggers short jump locomotion, i.e., some frames are skipped while moving. This is intended to control cybersickness while moving faster.

In addition to navigation, the interface incorporates advanced measurement capabilities (see Figure 5). Users can utilize measurement tools to assess various aspects of the colon, such as the diameter of polyps or the length of potential abnormalities. These measurements are recorded and stored, enabling radiologists to review and compare previous measurements during subsequent sessions. This feature proves particularly valuable for monitoring the progression of conditions or tracking changes over time.

For flexibility and convenience, the interface includes a teleportation function. This functionality allows users to move to specific locations within the colon instantly. This provides quick access to areas of interest and enables them to revisit specific regions for further examination. The teleportation feature promotes efficient exploration and facilitates focused analysis during the virtual colonography experience. They can teleport by pointing at the full-scale colon or the world-in-miniature colon attached to the left hand.

Furthermore, the UI visually represents CT scans at the pointed positions within the colon. By interacting with the virtual environment, users can trigger the display of corresponding CT scans, offering a synchronized view of the tissue behind the reconstructed colon wall at specific locations, as seen in Figure 6. A plane orthogonal to the user view is initially generated from the stack and can be manipulated.

Another noteworthy feature of the virtual colonography project is the inclusion of coverage visualization using a heatmap, shown in Figure 7. The heatmap utilizes a color gradient (red for the areas not seen and blue for the most seen areas) to highlight areas of the colon that have not been thoroughly examined or remain unseen during the procedure. This feature assists users in identifying potentially overlooked regions, prompting them to redirect their attention to those areas and ensuring comprehensive coverage during the virtual colonography examination. This feature can be displayed on the WIM or the full-scale model of the colon.

Regarding the controllers, the design decision was to place all the controls that require pointing on the right-hand controller (dominant hand for most cases) and the remaining controls on the left-hand controller (see Figure 4a).

3.2. Implementation

The first step is to build a detailed 3D model of the colon from CT images. DICOM (Digital Imaging and Communications in Medicine) is the standard format for storing and transmitting medical images, including CT scans. Starting from a DICOM file, a suitable geometric representation of the colon must include the appropriate shape, size, and internal structures, such as the lumen, folds, and flexures. Notice that authentic colors cannot be extracted with this modality, and thus any graphics rendering features that appear are synthetic and for visualization purposes only.

Creating an accurate 3D model of the colon requires expertise in medical imaging and 3D modeling techniques to ensure it faithfully represents the geometry of the actual organ. As colonoscopy CT images are taken with colon insufflation, and dye contrast is used in preparation for the scan, this process can be done automatically using existing free software [35]. We applied a pipeline based on previous work [36,37,38]. In summary, we used segmentation by global thresholding followed by semi-automatic active contours using ITK-SNAP (http://www.itksnap.org/ (accessed on 25 March 2024)) or 3D Slicer (https://www.slicer.org/ (accessed on 25 March 2024)). To cope with the hardware limitations of current head-mounted displays (HMDs), a mesh simplification might be required, which will decrease the number of vertices of the object.

To make the fly-through over the center line functionality possible, the project requires a set of points representing the colon midline. These points should be derived from medical data or anatomical references to define the path users traverse during the virtual colonography accurately. Several methods exist to estimate the midline. We employed a mesh shrinking technique [37]. Precise positioning and connectivity of these points are crucial to ensuring a smooth and realistic fly-through experience and cybersickness-free navigation.

For the sake of reproducibility, we have included some accessory information here. We used Unity 2021.1.22f1 to implement the project. 3D Slicer was used to export points in the midline. For mesh simplification, we used web software provided by MyMiniFactory (https://myminifactory.github.io/Fast-Quadric-Mesh-Simplification (accessed on 25 March 2024)), but other solutions exist. The HMD used is the Pico Neo 3 Pro Eye.

3.3. Details of the Interface Design

In this section, we further describe details of the controls and advanced features of the designed interface.

3.3.1. Controls and User Interface

Regarding the controls, as already mentioned, all commands that require pointing, such as measure, teleport, slice view, and rotation, are placed on the right hand. The remaining controls trigger actions and toggle options, such as: go inside/outside, toggle mini-map, show a list of measures, move forward/backwards, and rotate objects. These are on the left hand. The radiologists can check the controls at any moment by looking at them. They have labels attached to them at all times and an up arrow to help with orientation (see Figure 4a).

Beside the colon are two small panels in the outside view (see Figure 4b). One is informative and displays the progress of the exam, showing the current duration and percentage of the colon traveled. The other is a research feature to export eye-gaze information and colormap data to files.

3.3.2. Eye Tracking Feature

This feature exploits eye-tracking capabilities embedded in HMDs to compute the portion of the colon the clinician is looking at. Eye-tracking data encompass two distinct movements: rapid saccades that can shift the pupil across the visual field in as little as 30 ms and fixations, where the gaze dwells in a specific area for extended periods exceeding 100 ms. We exploit fixations as indicators of sustained attention. Notably, foveal fixations enable high-resolution color vision, while peripheral attention provides only contours and low-resolution monochrome images. Our technique employs fixations to identify areas of the colon that have captured the clinician’s focused attention. After reading the eye gaze information, ray casts are thrown at regular locations within a small circle around the eye gaze vector. Ray cast hits on triangles of the colon mesh increment the time counters at the triangle vertices. Every vertex count starts at zero.

We then apply colors to each vertex, corresponding to a mapping of the accumulated times, to create a simple colormap with five colors. Zero corresponds to red, and blue corresponds to 1 s. Green, yellow, and orange colors are attributed to intermediary values by linear interpolation. Times above 1 s remain saturated with blue.

The feature may also work when an HMD is used without eye tracking. In that case, we use a ray from the camera origin through the point at the middle of the viewport.

3.3.3. Slice View

To implement the slice view tool (Figure 6), we used an external library “UnityVolumeRendering” [39]. It loads the DICOM file into a volume (3D matrix) and can output an image for any arbitrary slice plane. We then place the slice at the position the user points to, orienting the plane normally parallel to the tangent of the midline curve at the viewer location. The slice is presented on the same scale as the colon model and is registered with the model. The user can displace the plane along the line using the controller joystick. They can also rotate the plane around a pivot (pointed location) using a hand gesture to the side of the intended rotation. There is one slice at a time, but it can be freely manipulated to focus on any structure of interest for diagnosis. This interface represents a differential feature and a contribution of this system.

4. Evaluation

Several prototypes of our immersive colonography viewer were shown to experts and non-experts for informal feedback during development. Many redesigns occurred in this iterative process. This paper presents an advanced formative evaluation case study with our current prototype.

4.1. Method

Our research team carried out user test sessions to evaluate the effectiveness, efficiency, and overall user experience of the interface, employing the think-aloud protocol [40] throughout these sessions. By having participants articulate their thoughts in real time as they engaged with the primary task, this approach facilitated the collection of firsthand reports about their interaction experiences. This method is a staple in interaction design, enabling usability evaluators to identify and understand fundamental issues.

Although the think-aloud method can pose challenges for participants, it yields vital qualitative insights. Additionally, our protocol captured objective metrics during the interaction, including task completion time, accuracy of radiological observations, and the extent of colon examination.

We applied our protocol to two users with distinct profiles. The first, EXPERT, is a seasoned radiologist conducting virtual colonoscopies daily. The second, STUDENT, is a medical school student versed in anatomy and general pathology. This approach not only offered qualitative insights into the user experience but also enabled us to examine the impact on various user profiles and discern potential navigation patterns. The study simulates a realistic clinical scenario. The participant’s task is to analyze the virtual colon and identify abnormalities. This task comprises several subtasks: navigation, visualization, selection, manipulation, and decision-making.

For the analysis, we collected eye-gaze information using eye-tracking technology. These data are used to calculate the time spent looking at any region of the colon surface and explore visual attention patterns. During the examination, a color map indicating the seen and unseen regions is available and exported for analysis. Such information can be crucial in identifying potential limitations or strengths of the immersive virtual colonoscopy system.

4.2. Data

The colon model was reconstructed from “The Cancer Imaging Archive (TCIA)” (https://www.cancerimagingarchive.net/ (accessed on 25 March 2024)), subject ID: CTC-3105759107, using open-access tools [37]. The 3D model and center line are first reconstructed from the images, then simplified and sanitized.

The model is generated using 3D Slicer software through a sequence of steps:

• Importing DICOM files;
• Semi-automatic segmentation;
• 3D reconstruction using marching cubes;
• Refinement: smoothing, filling holes or gaps;
• Exporting the model to an OBJ file;
• Generating the center line with the SlicerVMTK extension.

After exporting the model, a decimation process must still be applied to reduce its vertices.

4.3. Protocol

The session comprises four steps: (1) a presentation of the system interface and explanation of the controls; (2) an interactive tutorial where the participant is immersed and follows instructions to try each functionality; (3) the participant completes the task while thinking aloud; and (4) a debriefing interview to collect more information about any difficulties and missing items, especially in comparison with desktop VC.

5. Results

Both subjects completed the task without experiencing any symptoms of cybersickness.

5.1. Performance

The results are summarized in

Table 1. Performance and coverage results.

	Area Covered	Total Time	Anomalies Reported	Ground Truth	True Positives	False Positives	Paths
EXPERT	89.0%	00:22:15	8	4	4	4	2
STUDENT	67.9%	00:12:02	6		3	3	1

. The EXPERT completed the analysis in 22 min and 15 s, performing two passes from rectum to cecum and cecum to rectum. This approach was employed to prevent overlooking features due to the wide-angle projection commonly used in these systems. The EXPERT successfully identified and measured eight anomalies within the colon. Among these, four were true positives, representing actual abnormalities, while the remaining four were false positives, indicating non-abnormal tissue. During the thinking-aloud process, the EXPERT mentioned their usual workflow of initially selecting potential candidates and subsequently analyzing those locations on the CT scan to determine the type of material based on the density scale. However, our tool only displays the CT scan without allowing for density windowing, which prevented the EXPERT from ruling out the false positives.

Although less experienced than the radiologist, the STUDENT demonstrated agility with the interface, completing the examination in 12 min, just over half the time spent by the EXPERT. Arguably, the leading cause for this shorter time is that the STUDENT made only one pass: rectum to cecum. The STUDENT mentioned during the analysis that a second pass was unnecessary because of the possibility of looking around in 3D. Notably, they managed to identify and measure six anomalies within the colon, three of them being actual abnormalities (true positive).

5.2. Coverage

Both participants explored the entire colon midline; however, their approaches differed. In Figure 8, areas painted in red remain unseen, while those painted in blue were seen for at least one second.

Both participants traversed the entire colon along the mid-line but employed different viewing approaches. Figure 8 presents a heatmap indicating coverage, where red denotes unseen areas and blue represents areas observed for at least one second. The resulting maps of the STUDENT are illustrated in Figure 8a,c, exhibiting a prevalence of red regions. Conversely, the EXPERT’s maps, shown in Figure 8b,d, display a more widespread presence of blue. These findings indicate that the EXPERT dedicated more time and attention to more areas than the STUDENT. By summing the non-red areas, we can calculate the percentage of the colon covered by each participant, as presented in Table 1, under the column labeled Area Covered.

5.3. Usability and Experience

The initial observation of both participants was the abundance of buttons on the controller, which led to difficulty recalling the various interaction possibilities. They were subsequently encouraged to examine the hologram labels on the buttons carefully. Despite the steep learning curve, both participants quickly adapted to the system. However, it was notable that the EXPERT sought assistance more frequently than the STUDENT. This behavior difference could be attributed to the EXPERT’s greater age and experience, which instilled a sense of caution and responsibility.

The STUDENT tried VR for the first time and enjoyed the experience. He commented that the interface was natural, easy to learn, and familiar. He also stated that he did not use the slice view tool much, but suggested that the slice could be in a panel on one of the hands. He mentioned that the colon in miniature was helpful to assist in the diagnosis, as well as the heatmap feature. The STUDENT used the mini-map by the end of the exam to check for the unseen areas of the colon. He argued that VR has more freedom of movement to look around and that a “2-way trip” is unnecessary, contrary to TC and desktop VC. He pointed out that visual measures and marks from the outside of the colon assist in navigation and orientation in the virtual environment.

Part of the EXPERT’s initial difficulty with the controller is explained by an observation she made that the controller has a virtual representation but not the hand, making it difficult to know where a finger is located in reference to the buttons. The EXPERT did not enjoy the slice view. She misses the possibility of controlling the density window, a feature we still need to implement. Moreover, she reported a need to see the camera’s current location on the three canonical slice orientations. She pointed out that density windowing is necessary because other findings may confound the diagnosis in addition to polyps and diverticula. These features include lipomas, mixed oral contrast dye bulks, and fecal matter. In the list below, we present an extract of the thinking-aloud output:

• “How do I delete a wrong measurement?”
• “Congratulate the patient for the excellent preparation; this colon is really free of residue, not like we see everyday”.
• “This looks like a diverticulum at 7 o’clock from my position”.
• “There are jumps sometimes, but I don’t know how much I skipped”. (This was when she moved the joystick to 100%, and the navigation switched to short-jumps mode. She went back to check nothing was missed.)
• “When the colon curves down, I feel like falling on a pit”.
• “Another diverticulum at 12 o’clock”.
• “Now here there are many [semilunar] folds”.
• “Jumped again. This cannot be”.
• “I can’t confirm what it is without checking the CT”.
• “Some irregular folds, I needed to investigate. This zone here, I’m not sure it’s a polyp”.
• “May I see the colors?”
• “Takes too long to become blue. We look much faster than that”.
• “The splenic flexure… I know because it is very angled”.
• “When there are folds, I need the CT”.
• “I need to find the valve”. (Arrived at the cecum, 12 min.)
• “How do I turn to travel in the other direction?”
• “Starting the transverse colon, it is smoother”.
• “There are small points that might be polyps”.
• “I hadn’t measured here. Can I do it now?”
• “Measuring is a little hard [low stability with the free hand], the second point quickly appears”.
• “We usually don’t take so long in the real exam”.
• “Already in the descending colon, cheer up!”
• “Arriving at the sigmoid to see that sympathetic polyp again”.
• “It is by the medial sigmoid. One can see it clearly”.

This feedback in itself is a significant contribution to the paper. We have made some additional observations. First, the EXPERT is cautious about noticing everything. She was not comfortable with the short jumps for navigation. This forced her to move slower than she wished for a considerable extent of the path. As she made the return trip, she wanted the path trodden on the inbound trip to be painted differently than the path trodden on the way back. Coloring has a role in how she assembles a mental model of the examined colon that needs further investigation. She insisted on starting the experiment seated. We used a swivel chair without wheels to allow for turning. She is used to doing the desktop VC seated. However, she decided to stand up, as turning around 180° for the return trip was necessary. After that, she stated, “I see better now”. Standing up changed to a higher perspective within the colon so the head can be kept at a low pitch with a larger field of view. In lower heights, when seated, a user needs to pitch up the head to obtain the same field. Considering the weight of the HMD, a lower pitch is more easily sustained. The EXPERT also explained that, besides analyzing density, the CT slices are helpful in indicating the user’s location within the colon. The fact that she missed this feature suggests that she was somewhat disoriented.

5.4. Lessons Learned

This paper presents several requirements and observations from the user-centered design process we conducted to develop a usable immersive viewer for virtual colonoscopy. This process helped us acquire and build knowledge about how VR can help the analysis. The think-aloud evaluation additionally helped us learn the following lessons.

3D Selection:

Selection by pointing is less accurate in 3D than in 2D mouse-based UIs. Actions that require selecting precise points, such as measuring polyp sizes, may require posterior adjustments. A revised design should address this by providing a second level of precise position control for previously inserted markings.

Orientation:

Colon shapes vary considerably, making position and orientation awareness challenging during navigation. Restricting global displacements to a midline helps with cybersickness and provides an important sense of direction. However, our design still leaves the roll component uncontrolled, i.e., the ground reference changes along the curve smoothly but unpredictably. Additional orientation cues and/or constraints must provide constant feedback on a fixed body reference.

Navigation:

Using the joystick for global 1D motion and natural movement for local adjustments works well. However, the speed control needs special attention. The analog joystick provides a means to adjust speed between zero and a predefined maximum quickly. There is a compromise between the max speed allowed and cybersickness effects, so we kept the max speed low but offered short jumps when the joystick is kept to the max. The discontinuities caused by jumps provided a poor experience and need additional research to offer proper hints to adjust the length of the jump.

UI Patterns:

There are no widely adopted UI patterns for basic controls such as “delete”. Thus, users still need to learn the styles adopted in the design. A recommendation is to use labels for possible or probable actions when pointing to objects.

Diagnosis:

Radiologists do not just look at isolated colors and shapes in images. They use a combination of factors to decide whether a colonic relief feature has clinical relevance. The number and location of these features change the odds for each, so the system must provide unambiguous positional awareness. Modifying the density window, at least for preset windows (fat, soft tissue, contrast dye), is necessary.

6. Conclusions

The current state-of-the-art in the design of immersive virtual colonoscopy is not properly developed. Only a few research groups have tackled the problem. Therefore, much is still to be learned. This paper presented our latest results in designing a viable VR interface for the diagnostic procedure. We further explored some previously addressed elements, such as the navigation and measurement tools. However, we also innovated with eye tracking to estimate the coverage and developed an in situ CT slice viewer. Our findings suggest that these technologies enhance IVC’s effectiveness by providing valuable insights into the user’s decision-making process and improving the accuracy of lesion detection and diagnosis.

Eye-tracking technology in IVC can improve the procedure’s efficiency by enabling radiologists to navigate the colon more efficiently and quickly identify abnormalities. The integration of CT scan viewing features into IVC provides high-resolution images of the colon that can aid in identifying and diagnosing abnormalities, further enhancing the procedure’s accuracy.

We believe including new features is not the most important contribution. Our experimental evaluation with specialists provided valuable feedback to encourage more researchers to approach the problem.

Few radiologists have desktop VC expertise, so obtaining information and experimental data is challenging. The main reason for this shortage is that the diagnostic procedure takes longer for the professional than other CT analyses without paying proportionally, reducing professional interest in it. Significantly diminishing the time to complete the task seems to be a key factor in increasing the attractiveness of this technique. We demonstrated that we are not too far off in making immersive VC more efficient than desktop VC. The time to complete, between 12 and 22 min, is similar to the average of 20 min reported for a desktop VC [41]. The effectiveness of finding abnormalities was also similar in our IVC experiment.

The scope of this paper, however, is limited. We suggest that formal quantitative experiments be performed to generalize these findings after following the lessons learned. Future experiments should also determine the eventual differences between VC and IVC, in terms coverage, analysis pattern, and time to complete, on the accuracy of the outcome. Indeed, quantitative statistics are essential for objectively assessing efficiency, accuracy, and user satisfaction changes. Enhancing eye-tracking technology with more sophisticated machine-learning algorithms to assist radiologists in detecting lesions is also a research direction. Even though we have documented an exploratory design, we are optimistic that further development of immersive virtual colonography will continue to refine diagnostic processes and improve clinical outcomes through enhanced efficiency and precision.