Abstract
Reconstructing 2D freehand Ultrasound (US) frames into 3D space without using a tracker has recently seen advances with deep learning. Predicting good frame-to-frame rigid transformations is often accepted as the learning objective, especially when the ground-truth labels from spatial tracking devices are inherently rigid transformations. Motivated by a) the observed nonrigid deformation due to soft tissue motion during scanning, and b) the highly sensitive prediction of rigid transformation, this study investigates the methods and their benefits in predicting nonrigid transformations for reconstructing 3D US. We propose a novel co-optimisation algorithm for simultaneously estimating rigid transformations among US frames, supervised by ground-truth from a tracker, and a nonrigid deformation, optimised by a regularised registration network. We show that these two objectives can be either optimised using meta-learning or combined by weighting. A fast scattered data interpolation is also developed for enabling frequent reconstruction and registration of non-parallel US frames, during training. With a new data set containing over 357,000 frames in 720 scans, acquired from 60 subjects, the experiments demonstrate that, due to an expanded thus easier-to-optimise solution space, the generalisation is improved with the added deformation estimation, with respect to the rigid ground-truth. The global pixel reconstruction error (assessing accumulative prediction) is lowered from 18.48 to 16.51 mm, compared with baseline rigid-transformation-predicting methods. Using manually identified landmarks, the proposed co-optimisation also shows potentials in compensating nonrigid tissue motion at inference, which is not measurable by tracker-provided ground-truth. The code and data used in this paper are made publicly available at https://github.com/QiLi111/NR-Rec-FUS.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The interpolation process has an average speed of less than 1 ms over the dataset.
- 2.
This study was performed in accordance with the ethical standards in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Approval was granted by the Ethics Committee of local institution (UCL Department of Medical Physics and Biomedical Engineering) on \(20^{th}\) Jan. 2023 [24055/001].
References
Amidror, I.: Scattered data interpolation methods for electronic imaging systems: a survey. Journal of electronic imaging 11(2), 157–176 (2002)
Balakrishnan, G., Zhao, A., et al.: Voxelmorph: a learning framework for deformable medical image registration. IEEE transactions on medical imaging 38(8), 1788–1800 (2019)
Bartier, P.M., Keller, C.P.: Multivariate interpolation to incorporate thematic surface data using inverse distance weighting (idw). Computers & Geosciences 22(7), 795–799 (1996)
Chen, J.F., Fowlkes, J.B., et al.: Determination of scan-plane motion using speckle decorrelation: Theoretical considerations and initial test. International Journal of Imaging Systems and Technology 8(1), 38–44 (1997)
Ebner, M., Chouhan, M., et al.: Point-spread-function-aware slice-to-volume registration: application to upper abdominal mri super-resolution. In: Reconstruction, Segmentation, and Analysis of Medical Images: First International Workshops, RAMBO 2016 and HVSMR 2016. pp. 3–13. Springer (2017)
Guo, H., Chao, H., et al.: Ultrasound volume reconstruction from freehand scans without tracking. IEEE Transactions on Biomedical Engineering 70(3), 970–979 (2022)
Hu, Y., Gibson, E., et al.: Freehand ultrasound image simulation with spatially-conditioned generative adversarial networks. In: Molecular imaging, reconstruction and analysis of moving body organs, and stroke imaging and treatment, pp. 105–115. Springer (2017)
Hu, Y., Modat, M., et al.: Weakly-supervised convolutional neural networks for multimodal image registration. Medical image analysis 49, 1–13 (2018)
Huang, H., Cui, C., et al.: Grid interpolation algorithm based on nearest neighbor fast search. Earth Science Informatics 5, 181–187 (2012)
Jaderberg, M., Simonyan, K., Zisserman, A., kavukcuoglu, k.: Spatial transformer networks. In: Advances in Neural Information Processing Systems. vol. 28. Curran Associates, Inc. (2015)
Lang, A., Mousavi, P., et al.: Multi-modal registration of speckle-tracked freehand 3d ultrasound to ct in the lumbar spine. Medical image analysis 16(3), 675–686 (2012)
Lasso, A., Heffter, T., et al.: Plus: open-source toolkit for ultrasound-guided intervention systems. IEEE transactions on biomedical engineering 61(10), 2527–2537 (2014)
Leblanc, T., Lalys, F., et al.: Stretched reconstruction based on 2d freehand ultrasound for peripheral artery imaging. International Journal of Computer Assisted Radiology and Surgery 17(7), 1281–1288 (2022)
Lee, S., Wolberg, G., et al.: Scattered data interpolation with multilevel b-splines. IEEE transactions on visualization and computer graphics 3(3), 228–244 (1997)
Li, Q., Shen, Z., et al.: Privileged anatomical and protocol discrimination in trackerless 3d ultrasound reconstruction. In: International Workshop on Advances in Simplifying Medical Ultrasound. pp. 142–151. Springer (2023)
Li, Q., Shen, Z., et al.: Trackerless freehand ultrasound with sequence modelling and auxiliary transformation over past and future frames. In: International Symposium on Biomedical Imaging. pp. 1–5. IEEE (2023)
Li, Q., Shen, Z., et al.: Long-term dependency for 3d reconstruction of freehand ultrasound without external tracker. IEEE Transactions on Biomedical Engineering 71(3), 1033–1042 (2024)
Lin, B., Feiyang, Y., Zhang, Y.: A closer look at loss weighting in multi-task learning (2021)
Lindseth, F., Kaspersen, J.H., et al.: Multimodal image fusion in ultrasound-based neuronavigation: improving overview and interpretation by integrating preoperative mri with intraoperative 3d ultrasound. Computer Aided Surgery 8(2), 49–69 (2003)
Liu, H., Simonyan, K., et al.: Darts: Differentiable architecture search. In: International Conference on Learning Representations (2018)
Luo, M., Yang, X., et al.: Self context and shape prior for sensorless freehand 3d ultrasound reconstruction. In: Medical Image Computing and Computer Assisted Intervention. pp. 201–210. Springer (2021)
Luo, M., Yang, X., et al.: Deep motion network for freehand 3d ultrasound reconstruction. In: Medical Image Computing and Computer-Assisted Intervention. pp. 290–299. Springer (2022)
Luo, M., Yang, X., et al.: Multi-imu with online self-consistency for freehand 3d ultrasound reconstruction. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 342–351. Springer (2023)
Luo, M., Yang, X., et al.: Recon: Online learning for sensorless freehand 3d ultrasound reconstruction. Medical Image Analysis 87, 102810 (2023)
Mikaeili, M., Bilge, H.Ş.: Trajectory estimation of ultrasound images based on convolutional neural network. Biomedical Signal Processing and Control 78, 103965 (2022)
Miura, K., Ito, K., et al.: Localizing 2d ultrasound probe from ultrasound image sequences using deep learning for volume reconstruction. In: International Workshop on Advances in Simplifying Medical Ultrasound. pp. 97–105. Springer (2020)
Miura, K., Ito, K., et al.: Probe localization from ultrasound image sequences using deep learning for volume reconstruction. In: International Forum on Medical Imaging in Asia. vol. 11792, pp. 133–138. SPIE (2021)
Ning, G., Liang, H., et al.: Spatial position estimation method for 3d ultrasound reconstruction based on hybrid transfomers. In: International Symposium on Biomedical Imaging (ISBI). pp. 1–5. IEEE (2022)
Prevost, R., Salehi, M., et al.: 3d freehand ultrasound without external tracking using deep learning. Medical image analysis 48, 187–202 (2018)
Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. pp. 6105–6114. PMLR (2019)
Vercauteren, T., Perchant, A., et al.: Robust mosaicing with correction of motion distortions and tissue deformations for in vivo fibered microscopy. Medical image analysis 10(5), 673–692 (2006)
Wein, W., Lupetti, M., et al.: Three-dimensional thyroid assessment from untracked 2d ultrasound clips. In: Medical Image Computing and Computer-Assisted Intervention. pp. 514–523. Springer (2020)
Xie, Y., Liao, H., et al.: Image-based 3d ultrasound reconstruction with optical flow via pyramid warping network. In: IEEE Engineering in Medicine & Biology Society (EMBC). pp. 3539–3542. IEEE (2021)
Xingfang, Y., Yumei, H., Feng, G.: A simple camera calibration method based on sub-pixel corner extraction of the chessboard image. In: 2010 IEEE International Conference on Intelligent Computing and Intelligent Systems. vol. 3, pp. 688–692. IEEE (2010)
Acknowledgments
This work was supported by the EPSRC [EP/T029404/1], a Royal Academy of Engineering/Medtronic Research Chair [RCSRF1819\(\backslash \)7\(\backslash \)734] (TV), Wellcome/EPSRC Centre for Interventional and Surgical Sciences [203145Z/16/Z], and the International Alliance for Cancer Early Detection, an alliance between Cancer Research UK [C28070/A30912; C73666/A31378], Canary Center at Stanford University, the University of Cambridge, OHSU Knight Cancer Institute, University College London and the University of Manchester. TV is co-founder and shareholder of Hypervision Surgical. Qi Li was supported by the University College London Overseas and Graduate Research Scholarships. For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, Q. et al. (2024). Nonrigid Reconstruction of Freehand Ultrasound Without a Tracker. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15004. Springer, Cham. https://doi.org/10.1007/978-3-031-72083-3_64
Download citation
DOI: https://doi.org/10.1007/978-3-031-72083-3_64
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72082-6
Online ISBN: 978-3-031-72083-3
eBook Packages: Computer ScienceComputer Science (R0)
