Showing 1–19 of 19 results for author: Mansi, T

SE(3)-Hyena Operator for Scalable Equivariant Learning

Authors: Artem Moskalev, Mangal Prakash, Rui Liao, Tommaso Mansi

Abstract: Modeling global geometric context while maintaining equivariance is crucial for accurate predictions in many fields such as biology, chemistry, or vision. Yet, this is challenging due to the computational demands of processing high-dimensional data at scale. Existing approaches such as equivariant self-attention or distance-based message passing, suffer from quadratic complexity with respect to se… ▽ More Modeling global geometric context while maintaining equivariance is crucial for accurate predictions in many fields such as biology, chemistry, or vision. Yet, this is challenging due to the computational demands of processing high-dimensional data at scale. Existing approaches such as equivariant self-attention or distance-based message passing, suffer from quadratic complexity with respect to sequence length, while localized methods sacrifice global information. Inspired by the recent success of state-space and long-convolutional models, in this work, we introduce SE(3)-Hyena operator, an equivariant long-convolutional model based on the Hyena operator. The SE(3)-Hyena captures global geometric context at sub-quadratic complexity while maintaining equivariance to rotations and translations. Evaluated on equivariant associative recall and n-body modeling, SE(3)-Hyena matches or outperforms equivariant self-attention while requiring significantly less memory and computational resources for long sequences. Our model processes the geometric context of 20k tokens x3.5 times faster than the equivariant transformer and allows x175 longer a context within the same memory budget. △ Less

Submitted 1 July, 2024; originally announced July 2024.

Cardiac ultrasound simulation for autonomous ultrasound navigation

Authors: Abdoul Aziz Amadou, Laura Peralta, Paul Dryburgh, Paul Klein, Kaloian Petkov, Richard James Housden, Vivek Singh, Rui Liao, Young-Ho Kim, Florin Christian Ghesu, Tommaso Mansi, Ronak Rajani, Alistair Young, Kawal Rhode

Abstract: Ultrasound is well-established as an imaging modality for diagnostic and interventional purposes. However, the image quality varies with operator skills as acquiring and interpreting ultrasound images requires extensive training due to the imaging artefacts, the range of acquisition parameters and the variability of patient anatomies. Automating the image acquisition task could improve acquisition… ▽ More Ultrasound is well-established as an imaging modality for diagnostic and interventional purposes. However, the image quality varies with operator skills as acquiring and interpreting ultrasound images requires extensive training due to the imaging artefacts, the range of acquisition parameters and the variability of patient anatomies. Automating the image acquisition task could improve acquisition reproducibility and quality but training such an algorithm requires large amounts of navigation data, not saved in routine examinations. Thus, we propose a method to generate large amounts of ultrasound images from other modalities and from arbitrary positions, such that this pipeline can later be used by learning algorithms for navigation. We present a novel simulation pipeline which uses segmentations from other modalities, an optimized volumetric data representation and GPU-accelerated Monte Carlo path tracing to generate view-dependent and patient-specific ultrasound images. We extensively validate the correctness of our pipeline with a phantom experiment, where structures' sizes, contrast and speckle noise properties are assessed. Furthermore, we demonstrate its usability to train neural networks for navigation in an echocardiography view classification experiment by generating synthetic images from more than 1000 patients. Networks pre-trained with our simulations achieve significantly superior performance in settings where large real datasets are not available, especially for under-represented classes. The proposed approach allows for fast and accurate patient-specific ultrasound image generation, and its usability for training networks for navigation-related tasks is demonstrated. △ Less

Submitted 9 February, 2024; originally announced February 2024.

Comments: 24 pages, 10 figures, 5 tables

ACM Class: I.6.0; I.5.4; J.3

A multi-organ point cloud registration algorithm for abdominal CT registration

Authors: Samuel Joutard, Thomas Pheiffer, Chloe Audigier, Patrick Wohlfahrt, Reuben Dorent, Sebastien Piat, Tom Vercauteren, Marc Modat, Tommaso Mansi

Abstract: Registering CT images of the chest is a crucial step for several tasks such as disease progression tracking or surgical planning. It is also a challenging step because of the heterogeneous content of the human abdomen which implies complex deformations. In this work, we focus on accurately registering a subset of organs of interest. We register organ surface point clouds, as may typically be extra… ▽ More Registering CT images of the chest is a crucial step for several tasks such as disease progression tracking or surgical planning. It is also a challenging step because of the heterogeneous content of the human abdomen which implies complex deformations. In this work, we focus on accurately registering a subset of organs of interest. We register organ surface point clouds, as may typically be extracted from an automatic segmentation pipeline, by expanding the Bayesian Coherent Point Drift algorithm (BCPD). We introduce MO-BCPD, a multi-organ version of the BCPD algorithm which explicitly models three important aspects of this task: organ individual elastic properties, inter-organ motion coherence and segmentation inaccuracy. This model also provides an interpolation framework to estimate the deformation of the entire volume. We demonstrate the efficiency of our method by registering different patients from the LITS challenge dataset. The target registration error on anatomical landmarks is almost twice as small for MO-BCPD compared to standard BCPD while imposing the same constraints on individual organs deformation. △ Less

Submitted 15 March, 2022; originally announced March 2022.

Comments: Accepted at WBIR 2022

Automated Catheter Tip Repositioning for Intra-cardiac Echocardiography

Authors: Young-Ho Kim, Jarrod Collins, Zhongyu Li, Ponraj Chinnadurai, Ankur Kapoor, C. Huie Lin, Tommaso Mansi

Abstract: Purpose: Intra-Cardiac Echocardiography (ICE) is a powerful imaging modality for guiding cardiac electrophysiology and structural heart interventions. ICE provides real-time observation of anatomy and devices, while enabling direct monitoring of potential complications. In single operator settings, the physician needs to switch back-and-forth between the ICE catheter and therapy device, making con… ▽ More Purpose: Intra-Cardiac Echocardiography (ICE) is a powerful imaging modality for guiding cardiac electrophysiology and structural heart interventions. ICE provides real-time observation of anatomy and devices, while enabling direct monitoring of potential complications. In single operator settings, the physician needs to switch back-and-forth between the ICE catheter and therapy device, making continuous ICE support impossible. Two operators setup are therefore sometimes implemented, with the challenge of increase room occupation and cost. Two operator setups are sometimes implemented, but increase procedural costs and room occupation. Methods: ICE catheter robotic control system is developed with automated catheter tip repositioning (i.e. view recovery) method, which can reproduce important views previously navigated to and saved by the user. The performance of the proposed method is demonstrated and evaluated in a combination of heart phantom and animal experiments. Results: Automated ICE view recovery achieved catheter tip position accuracy of 2.09 +/-0.90 mm and catheter image orientation accuracy of 3.93 +/- 2.07 degree in animal studies, and 0.67 +/- 0.79 mm and 0.37 +/- 0.19 degree in heart phantom studies, respectively. Our proposed method is also successfully used during transeptal puncture in animals without complications, showing the possibility for fluoro-less transeptal puncture with ICE catheter robot. Conclusion: Robotic ICE imaging has the potential to provide precise and reproducible anatomical views, which can reduce overall execution time, labor burden of procedures, and x-ray usage for a range of cardiac procedures. Keywords: Automated View Recovery, Path Planning, Intra-cardiac echocardiography (ICE), Catheter, Tendon-driven manipulator, Cardiac Imaging △ Less

Submitted 21 January, 2022; originally announced January 2022.

Comments: arXiv admin note: substantial text overlap with arXiv:2009.05859

A Wide-area, Low-latency, and Power-efficient 6-DoF Pose Tracking System for Rigid Objects

Authors: Young-Ho Kim, Ankur Kapoor, Tommaso Mansi, Ali Kamen

Abstract: Position sensitive detectors (PSDs) offer possibility to track single active marker's two (or three) degrees of freedom (DoF) position with a high accuracy, while having a fast response time with high update frequency and low latency, all using a very simple signal processing circuit. However they are not particularly suitable for 6-DoF object pose tracking system due to lack of orientation measur… ▽ More Position sensitive detectors (PSDs) offer possibility to track single active marker's two (or three) degrees of freedom (DoF) position with a high accuracy, while having a fast response time with high update frequency and low latency, all using a very simple signal processing circuit. However they are not particularly suitable for 6-DoF object pose tracking system due to lack of orientation measurement, limited tracking range, and sensitivity to environmental variation. We propose a novel 6-DoF pose tracking system for a rigid object tracking requiring a single active marker. The proposed system uses a stereo-based PSD pair and multiple Inertial Measurement Units (IMUs). This is done based on a practical approach to identify and control the power of Infrared-Light Emitting Diode (IR-LED) active markers, with an aim to increase the tracking work space and reduce the power consumption. Our proposed tracking system is validated with three different work space sizes and for static and dynamic positional accuracy using robotic arm manipulator with three different dynamic motion patterns. The results show that the static position root-mean-square (RMS) error is 0.6mm. The dynamic position RMS error is 0.7-0.9mm. The orientation RMS error is between 0.04 and 0.9 degree at varied dynamic motion. Overall, our proposed tracking system is capable of tracking a rigid object pose with sub-millimeter accuracy at the mid range of the work space and sub-degree accuracy for all work space under a lab setting. △ Less

Submitted 10 January, 2022; v1 submitted 15 September, 2021; originally announced September 2021.

Shape-adaptive Hysteresis Compensation for Tendon-driven Continuum Manipulators

Authors: Young-Ho Kim, Tommaso Mansi

Abstract: Tendon-driven continuum manipulators (TDCM) are commonly used in minimally invasive surgical systems due to their long, thin, flexible structure that is compliant in narrow or tortuous environments. There exist many researches for precise tip control of the articulating section. However, these models do not account for the proximal shaft shape of TDCM, affecting the tip controls in practical setti… ▽ More Tendon-driven continuum manipulators (TDCM) are commonly used in minimally invasive surgical systems due to their long, thin, flexible structure that is compliant in narrow or tortuous environments. There exist many researches for precise tip control of the articulating section. However, these models do not account for the proximal shaft shape of TDCM, affecting the tip controls in practical settings. In this paper, we propose a gradient-based shift detection method based on motor current that can easily find the offset of task space models (i.e., hysteresis). We analyze our proposed methods with multiple Intra-cardiac Echocardiography catheters, which are typical commercial example of TDCM. Our results show that the errors from varied proximal shape are considerably reduced, and the accuracy of the tip manipulation is improved when changing external environmental structures. △ Less

Submitted 14 September, 2021; originally announced September 2021.

Non-linear Hysteresis Compensation of a Tendon-sheath-driven Robotic Manipulator using Motor Current

Authors: Dong-Ho Lee, Young-Ho Kim, Jarrod Collins, Ankur Kapoor, Dong-Soo Kwon, Tommaso Mansi

Abstract: Tendon-sheath-driven manipulators (TSM) are widely used in minimally invasive surgical systems due to their long, thin shape, flexibility, and compliance making them easily steerable in narrow or tortuous environments. Many commercial TSM-based medical devices have non-linear phenomena resulting from their composition such as backlash hysteresis and dead zone, which lead to a considerable challeng… ▽ More Tendon-sheath-driven manipulators (TSM) are widely used in minimally invasive surgical systems due to their long, thin shape, flexibility, and compliance making them easily steerable in narrow or tortuous environments. Many commercial TSM-based medical devices have non-linear phenomena resulting from their composition such as backlash hysteresis and dead zone, which lead to a considerable challenge for achieving precise control of the end effector pose. However, many recent works in the literature do not consider the combined effects and compensation of these phenomena, and less focus on practical ways to identify model parameters in realistic conditions. This paper proposes a simplified piecewise linear model to construct both backlash hysteresis and dead zone compensators together. Further, a practical method is introduced to identify model parameters using motor current from a robotic controller for the TSM. Our proposed methods are validated with multiple Intra-cardiac Echocardiography (ICE) catheters, which are typical commercial example of TSM, by periodic and non-periodic motions. Our results show that the errors from backlash hysteresis and dead zone are considerably reduced and therefore the accuracy of robotic control is improved when applying the presented methods. △ Less

Submitted 29 January, 2021; v1 submitted 3 November, 2020; originally announced November 2020.

Learning a Generative Motion Model from Image Sequences based on a Latent Motion Matrix

Authors: Julian Krebs, Hervé Delingette, Nicholas Ayache, Tommaso Mansi

Abstract: We propose to learn a probabilistic motion model from a sequence of images for spatio-temporal registration. Our model encodes motion in a low-dimensional probabilistic space - the motion matrix - which enables various motion analysis tasks such as simulation and interpolation of realistic motion patterns allowing for faster data acquisition and data augmentation. More precisely, the motion matrix… ▽ More We propose to learn a probabilistic motion model from a sequence of images for spatio-temporal registration. Our model encodes motion in a low-dimensional probabilistic space - the motion matrix - which enables various motion analysis tasks such as simulation and interpolation of realistic motion patterns allowing for faster data acquisition and data augmentation. More precisely, the motion matrix allows to transport the recovered motion from one subject to another simulating for example a pathological motion in a healthy subject without the need for inter-subject registration. The method is based on a conditional latent variable model that is trained using amortized variational inference. This unsupervised generative model follows a novel multivariate Gaussian process prior and is applied within a temporal convolutional network which leads to a diffeomorphic motion model. Temporal consistency and generalizability is further improved by applying a temporal dropout training scheme. Applied to cardiac cine-MRI sequences, we show improved registration accuracy and spatio-temporally smoother deformations compared to three state-of-the-art registration algorithms. Besides, we demonstrate the model's applicability for motion analysis, simulation and super-resolution by an improved motion reconstruction from sequences with missing frames compared to linear and cubic interpolation. △ Less

Submitted 31 January, 2021; v1 submitted 3 November, 2020; originally announced November 2020.

Comments: accepted at IEEE TMI

Graph convolutional regression of cardiac depolarization from sparse endocardial maps

Authors: Felix Meister, Tiziano Passerini, Chloé Audigier, Èric Lluch, Viorel Mihalef, Hiroshi Ashikaga, Andreas Maier, Henry Halperin, Tommaso Mansi

Abstract: Electroanatomic mapping as routinely acquired in ablation therapy of ventricular tachycardia is the gold standard method to identify the arrhythmogenic substrate. To reduce the acquisition time and still provide maps with high spatial resolution, we propose a novel deep learning method based on graph convolutional neural networks to estimate the depolarization time in the myocardium, given sparse… ▽ More Electroanatomic mapping as routinely acquired in ablation therapy of ventricular tachycardia is the gold standard method to identify the arrhythmogenic substrate. To reduce the acquisition time and still provide maps with high spatial resolution, we propose a novel deep learning method based on graph convolutional neural networks to estimate the depolarization time in the myocardium, given sparse catheter data on the left ventricular endocardium, ECG, and magnetic resonance images. The training set consists of data produced by a computational model of cardiac electrophysiology on a large cohort of synthetically generated geometries of ischemic hearts. The predicted depolarization pattern has good agreement with activation times computed by the cardiac electrophysiology model in a validation set of five swine heart geometries with complex scar and border zone morphologies. The mean absolute error hereby measures 8 ms on the entire myocardium when providing 50\% of the endocardial ground truth in over 500 computed depolarization patterns. Furthermore, when considering a complete animal data set with high density electroanatomic mapping data as reference, the neural network can accurately reproduce the endocardial depolarization pattern, even when a small percentage of measurements are provided as input features (mean absolute error of 7 ms with 50\% of input samples). The results show that the proposed method, trained on synthetically generated data, may generalize to real data. △ Less

Submitted 28 September, 2020; originally announced September 2020.

Comments: Accepted at the MICCAI 2020 Workshop Statistical Atlases and Computational Modeling of the Heart (STACOM)

Towards Automatic Manipulation of Intra-cardiac Echocardiography Catheter

Authors: Young-Ho Kim, Jarrod Collins, Zhongyu Li, Ponraj Chinnadurai, Ankur Kapoor, C. Huie Lin, Tommaso Mansi

Abstract: Intra-cardiac Echocardiography (ICE) is a powerful imaging modality for guiding electrophysiology and structural heart interventions. ICE provides real-time observation of anatomy, catheters, and emergent complications. However, this increased reliance on intraprocedural imaging creates a high cognitive demand on physicians who can often serve as interventionalist and imager. We present a robotic… ▽ More Intra-cardiac Echocardiography (ICE) is a powerful imaging modality for guiding electrophysiology and structural heart interventions. ICE provides real-time observation of anatomy, catheters, and emergent complications. However, this increased reliance on intraprocedural imaging creates a high cognitive demand on physicians who can often serve as interventionalist and imager. We present a robotic manipulator for ICE catheters to assist physicians with imaging and serve as a platform for developing processes for procedural automation. Herein, we introduce two application modules towards these goals: (1) a view recovery process that allows physicians to save views during intervention and automatically return with the push of a button and (2) a data-driven approach to compensate kinematic model errors that result from non-linear behaviors in catheter bending, providing more precise control of the catheter tip. View recovery is validated by repeated catheter positioning in cardiac phantom and animal experiments with position- and image-based analysis. We present a simplified calibration approach for error compensation and verify with complex rotation of the catheter in benchtop and phantom experiments under varying realistic curvature conditions. Results support that a robotic manipulator for ICE can provide an efficient and reproducible tool, potentially reducing execution time and promoting greater utilization of ICE imaging. △ Less

Submitted 29 January, 2021; v1 submitted 12 September, 2020; originally announced September 2020.

Probabilistic Motion Modeling from Medical Image Sequences: Application to Cardiac Cine-MRI

Authors: Julian Krebs, Tommaso Mansi, Nicholas Ayache, Hervé Delingette

Abstract: We propose to learn a probabilistic motion model from a sequence of images. Besides spatio-temporal registration, our method offers to predict motion from a limited number of frames, useful for temporal super-resolution. The model is based on a probabilistic latent space and a novel temporal dropout training scheme. This enables simulation and interpolation of realistic motion patterns given only… ▽ More We propose to learn a probabilistic motion model from a sequence of images. Besides spatio-temporal registration, our method offers to predict motion from a limited number of frames, useful for temporal super-resolution. The model is based on a probabilistic latent space and a novel temporal dropout training scheme. This enables simulation and interpolation of realistic motion patterns given only one or any subset of frames of a sequence. The encoded motion also allows to be transported from one subject to another without the need of inter-subject registration. An unsupervised generative deformation model is applied within a temporal convolutional network which leads to a diffeomorphic motion model, encoded as a low-dimensional motion matrix. Applied to cardiac cine-MRI sequences, we show improved registration accuracy and spatio-temporally smoother deformations compared to three state-of-the-art registration algorithms. Besides, we demonstrate the model's applicability to motion transport by simulating a pathology in a healthy case. Furthermore, we show an improved motion reconstruction from incomplete sequences compared to linear and cubic interpolation. △ Less

Submitted 23 September, 2019; v1 submitted 31 July, 2019; originally announced July 2019.

Comments: Probabilistic Motion Model, Motion Tracking, Temporal Super-Resolution, Diffeomorphic Registration, Temporal Variational Autoencoder (Final version)

Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations

Authors: Chen Qin, Bibo Shi, Rui Liao, Tommaso Mansi, Daniel Rueckert, Ali Kamen

Abstract: We propose a fully unsupervised multi-modal deformable image registration method (UMDIR), which does not require any ground truth deformation fields or any aligned multi-modal image pairs during training. Multi-modal registration is a key problem in many medical image analysis applications. It is very challenging due to complicated and unknown relationships between different modalities. In this pa… ▽ More We propose a fully unsupervised multi-modal deformable image registration method (UMDIR), which does not require any ground truth deformation fields or any aligned multi-modal image pairs during training. Multi-modal registration is a key problem in many medical image analysis applications. It is very challenging due to complicated and unknown relationships between different modalities. In this paper, we propose an unsupervised learning approach to reduce the multi-modal registration problem to a mono-modal one through image disentangling. In particular, we decompose images of both modalities into a common latent shape space and separate latent appearance spaces via an unsupervised multi-modal image-to-image translation approach. The proposed registration approach is then built on the factorized latent shape code, with the assumption that the intrinsic shape deformation existing in original image domain is preserved in this latent space. Specifically, two metrics have been proposed for training the proposed network: a latent similarity metric defined in the common shape space and a learningbased image similarity metric based on an adversarial loss. We examined different variations of our proposed approach and compared them with conventional state-of-the-art multi-modal registration methods. Results show that our proposed methods achieve competitive performance against other methods at substantially reduced computation time. △ Less

Submitted 21 March, 2019; originally announced March 2019.

Comments: Accepted as an oral presentation in IPMI 2019

Learning a Probabilistic Model for Diffeomorphic Registration

Authors: Julian Krebs, Hervé Delingette, Boris Mailhé, Nicholas Ayache, Tommaso Mansi

Abstract: We propose to learn a low-dimensional probabilistic deformation model from data which can be used for registration and the analysis of deformations. The latent variable model maps similar deformations close to each other in an encoding space. It enables to compare deformations, generate normal or pathological deformations for any new image or to transport deformations from one image pair to any ot… ▽ More We propose to learn a low-dimensional probabilistic deformation model from data which can be used for registration and the analysis of deformations. The latent variable model maps similar deformations close to each other in an encoding space. It enables to compare deformations, generate normal or pathological deformations for any new image or to transport deformations from one image pair to any other image. Our unsupervised method is based on variational inference. In particular, we use a conditional variational autoencoder (CVAE) network and constrain transformations to be symmetric and diffeomorphic by applying a differentiable exponentiation layer with a symmetric loss function. We also present a formulation that includes spatial regularization such as diffusion-based filters. Additionally, our framework provides multi-scale velocity field estimations. We evaluated our method on 3-D intra-subject registration using 334 cardiac cine-MRIs. On this dataset, our method showed state-of-the-art performance with a mean DICE score of 81.2% and a mean Hausdorff distance of 7.3mm using 32 latent dimensions compared to three state-of-the-art methods while also demonstrating more regular deformation fields. The average time per registration was 0.32s. Besides, we visualized the learned latent space and show that the encoded deformations can be used to transport deformations and to cluster diseases with a classification accuracy of 83% after applying a linear projection. △ Less

Submitted 17 March, 2019; v1 submitted 18 December, 2018; originally announced December 2018.

Comments: Accepted at (c) IEEE TMI and featured on https://ieee-tmi.org/

Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation

Authors: Yue Zhang, Shun Miao, Tommaso Mansi, Rui Liao

Abstract: Automatic parsing of anatomical objects in X-ray images is critical to many clinical applications in particular towards image-guided invention and workflow automation. Existing deep network models require a large amount of labeled data. However, obtaining accurate pixel-wise labeling in X-ray images relies heavily on skilled clinicians due to the large overlaps of anatomy and the complex texture p… ▽ More Automatic parsing of anatomical objects in X-ray images is critical to many clinical applications in particular towards image-guided invention and workflow automation. Existing deep network models require a large amount of labeled data. However, obtaining accurate pixel-wise labeling in X-ray images relies heavily on skilled clinicians due to the large overlaps of anatomy and the complex texture patterns. On the other hand, organs in 3D CT scans preserve clearer structures as well as sharper boundaries and thus can be easily delineated. In this paper, we propose a novel model framework for learning automatic X-ray image parsing from labeled CT scans. Specifically, a Dense Image-to-Image network (DI2I) for multi-organ segmentation is first trained on X-ray like Digitally Reconstructed Radiographs (DRRs) rendered from 3D CT volumes. Then we introduce a Task Driven Generative Adversarial Network (TD-GAN) architecture to achieve simultaneous style transfer and parsing for unseen real X-ray images. TD-GAN consists of a modified cycle-GAN substructure for pixel-to-pixel translation between DRRs and X-ray images and an added module leveraging the pre-trained DI2I to enforce segmentation consistency. The TD-GAN framework is general and can be easily adapted to other learning tasks. In the numerical experiments, we validate the proposed model on 815 DRRs and 153 topograms. While the vanilla DI2I without any adaptation fails completely on segmenting the topograms, the proposed model does not require any topogram labels and is able to provide a promising average dice of 85% which achieves the same level accuracy of supervised training (88%). △ Less

Submitted 11 June, 2018; originally announced June 2018.

Unsupervised Probabilistic Deformation Modeling for Robust Diffeomorphic Registration

Authors: Julian Krebs, Tommaso Mansi, Boris Mailhé, Nicholas Ayache, Hervé Delingette

Abstract: We propose a deformable registration algorithm based on unsupervised learning of a low-dimensional probabilistic parameterization of deformations. We model registration in a probabilistic and generative fashion, by applying a conditional variational autoencoder (CVAE) network. This model enables to also generate normal or pathological deformations of any new image based on the probabilistic latent… ▽ More We propose a deformable registration algorithm based on unsupervised learning of a low-dimensional probabilistic parameterization of deformations. We model registration in a probabilistic and generative fashion, by applying a conditional variational autoencoder (CVAE) network. This model enables to also generate normal or pathological deformations of any new image based on the probabilistic latent space. Most recent learning-based registration algorithms use supervised labels or deformation models, that miss important properties such as diffeomorphism and sufficiently regular deformation fields. In this work, we constrain transformations to be diffeomorphic by using a differentiable exponentiation layer with a symmetric loss function. We evaluated our method on 330 cardiac MR sequences and demonstrate robust intra-subject registration results comparable to two state-of-the-art methods but with more regular deformation fields compared to a recent learning-based algorithm. Our method reached a mean DICE score of 78.3% and a mean Hausdorff distance of 7.9mm. In two preliminary experiments, we illustrate the model's abilities to transport pathological deformations to healthy subjects and to cluster five diseases in the unsupervised deformation encoding space with a classification performance of 70%. △ Less

Submitted 20 July, 2018; v1 submitted 19 April, 2018; originally announced April 2018.

Dilated FCN for Multi-Agent 2D/3D Medical Image Registration

Authors: Shun Miao, Sebastien Piat, Peter Fischer, Ahmet Tuysuzoglu, Philip Mewes, Tommaso Mansi, Rui Liao

Abstract: 2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to per… ▽ More 2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding confidence levels. The contributions of this paper are threefold. First, we formulate 2D/3D registration as a MDP with observations, actions, and rewards properly defined with respect to X-ray imaging systems. Second, to handle various artifacts in 2D X-ray images, multiple local agents are employed efficiently via FCN-based structures, and an auto attention mechanism is proposed to favor the proposals from regions with more reliable visual cues. Third, a dilated FCN-based training mechanism is proposed to significantly reduce the Degree of Freedom in the simulation of registration environment, and drastically improve training efficiency by an order of magnitude compared to standard CNN-based training method. We demonstrate that the proposed method achieves high robustness on both spine cone beam Computed Tomography data with a low signal-to-noise ratio and data from minimally invasive spine surgery where severe image artifacts and occlusions are presented due to metal screws and guide wires, outperforming other state-of-the-art methods (single agent-based and optimization-based) by a large margin. △ Less

Submitted 21 November, 2017; originally announced December 2017.

Comments: AAAI 2018

An Artificial Agent for Robust Image Registration

Authors: Rui Liao, Shun Miao, Pierre de Tournemire, Sasa Grbic, Ali Kamen, Tommaso Mansi, Dorin Comaniciu

Abstract: 3-D image registration, which involves aligning two or more images, is a critical step in a variety of medical applications from diagnosis to therapy. Image registration is commonly performed by optimizing an image matching metric as a cost function. However, this task is challenging due to the non-convex nature of the matching metric over the plausible registration parameter space and insufficien… ▽ More 3-D image registration, which involves aligning two or more images, is a critical step in a variety of medical applications from diagnosis to therapy. Image registration is commonly performed by optimizing an image matching metric as a cost function. However, this task is challenging due to the non-convex nature of the matching metric over the plausible registration parameter space and insufficient approaches for a robust optimization. As a result, current approaches are often customized to a specific problem and sensitive to image quality and artifacts. In this paper, we propose a completely different approach to image registration, inspired by how experts perform the task. We first cast the image registration problem as a "strategy learning" process, where the goal is to find the best sequence of motion actions (e.g. up, down, etc.) that yields image alignment. Within this approach, an artificial agent is learned, modeled using deep convolutional neural networks, with 3D raw image data as the input, and the next optimal action as the output. To cope with the dimensionality of the problem, we propose a greedy supervised approach for an end-to-end training, coupled with attention-driven hierarchical strategy. The resulting registration approach inherently encodes both a data-driven matching metric and an optimal registration strategy (policy). We demonstrate, on two 3-D/3-D medical image registration examples with drastically different nature of challenges, that the artificial agent outperforms several state-of-art registration methods by a large margin in terms of both accuracy and robustness. △ Less

Submitted 30 November, 2016; originally announced November 2016.

Comments: To appear in AAAI Conference 2017

Shaping the Future through Innovations: From Medical Imaging to Precision Medicine

Authors: Dorin Comaniciu, Klaus Engel, Bogdan Georgescu, Tommaso Mansi

Abstract: Medical images constitute a source of information essential for disease diagnosis, treatment and follow-up. In addition, due to its patient-specific nature, imaging information represents a critical component required for advancing precision medicine into clinical practice. This manuscript describes recently developed technologies for better handling of image information: photorealistic visualizat… ▽ More Medical images constitute a source of information essential for disease diagnosis, treatment and follow-up. In addition, due to its patient-specific nature, imaging information represents a critical component required for advancing precision medicine into clinical practice. This manuscript describes recently developed technologies for better handling of image information: photorealistic visualization of medical images with Cinematic Rendering, artificial agents for in-depth image understanding, support for minimally invasive procedures, and patient-specific computational models with enhanced predictive power. Throughout the manuscript we will analyze the capabilities of such technologies and extrapolate on their potential impact to advance the quality of medical care, while reducing its cost. △ Less

Submitted 8 June, 2016; v1 submitted 1 May, 2016; originally announced May 2016.

Comments: Submitted to Medical Image Analysis, Elsevier, 20th Anniversary Special Issue

A Self-Taught Artificial Agent for Multi-Physics Computational Model Personalization

Authors: Dominik Neumann, Tommaso Mansi, Lucian Itu, Bogdan Georgescu, Elham Kayvanpour, Farbod Sedaghat-Hamedani, Ali Amr, Jan Haas, Hugo Katus, Benjamin Meder, Stefan Steidl, Joachim Hornegger, Dorin Comaniciu

Abstract: Personalization is the process of fitting a model to patient data, a critical step towards application of multi-physics computational models in clinical practice. Designing robust personalization algorithms is often a tedious, time-consuming, model- and data-specific process. We propose to use artificial intelligence concepts to learn this task, inspired by how human experts manually perform it. T… ▽ More Personalization is the process of fitting a model to patient data, a critical step towards application of multi-physics computational models in clinical practice. Designing robust personalization algorithms is often a tedious, time-consuming, model- and data-specific process. We propose to use artificial intelligence concepts to learn this task, inspired by how human experts manually perform it. The problem is reformulated in terms of reinforcement learning. In an off-line phase, Vito, our self-taught artificial agent, learns a representative decision process model through exploration of the computational model: it learns how the model behaves under change of parameters. The agent then automatically learns an optimal strategy for on-line personalization. The algorithm is model-independent; applying it to a new model requires only adjusting few hyper-parameters of the agent and defining the observations to match. The full knowledge of the model itself is not required. Vito was tested in a synthetic scenario, showing that it could learn how to optimize cost functions generically. Then Vito was applied to the inverse problem of cardiac electrophysiology and the personalization of a whole-body circulation model. The obtained results suggested that Vito could achieve equivalent, if not better goodness of fit than standard methods, while being more robust (up to 11% higher success rates) and with faster (up to seven times) convergence rate. Our artificial intelligence approach could thus make personalization algorithms generalizable and self-adaptable to any patient and any model. △ Less

Submitted 1 May, 2016; originally announced May 2016.

Comments: Submitted to Medical Image Analysis, Elsevier