research-article

Intelligent interaction and uncertainty visualization for efficient drusen and retinal layer segmentation in Optical Coherence Tomography

Authors:

Shekoufeh Gorgi Zadeh,

Maximilian W.M. Wintergerst,

Thomas SchultzAuthors Info & Claims

Volume 83, Issue C

Pages 51 - 61

https://doi.org/10.1016/j.cag.2019.07.001

Published: 01 October 2019 Publication History

Highlights

•

Developed semi-automated tools for correcting retinal layer and drusen segmentation.

•

Constrained shortest path, local smoothing, semi-automated drusen extraction.

•

Automated proposal of improved segmentations based on 3D context.

•

Derived and validated uncertainty measures for CNN-based retinal layer segmentation.

•

Intelligent interaction tools reduce time to 53% (layers)/73% (drusen).

Graphical abstract

Display Omitted

Abstract

Convolutional neural networks (CNNs) represent the state of the art for fully automated medical image segmentation. However, few works have combined CNNs with interactive user feedback in order to verify and, where necessary, correct their results. We present an interactive visual system that achieves this for the specific use case of segmenting drusen, which serve as a biomarker of age related macular degeneration, from Optical Coherence Tomography. Our main idea is to exploit the probabilistic nature of CNN-based segmentation. First, we derive two uncertainty measures from it. We demonstrate that they indicate cases in which automated segmentation is likely to have failed, and that visualizing them makes manual verification and correction more efficient. Second, based on the probabilistic information, we design intelligent tools for segmentation correction, which automatically propose the most likely alternative segmentation in agreement with user-specified constraints. In a small user study, uncertainty visualization and intelligent interaction reduced the time required to correct retinal layer segmentation by around 53% and, for drusen segmentation, even by 73%. In the future, we plan to use our system not only for efficient segmentation correction, but also for rapid creation of larger training sets.

References

[1]

Guo Y., Liu Y., T. Georgiou, M.S. Lew, A review of semantic segmentation using deep neural networks, Int J Multimed Inf Retr 7 (2) (2018) 87–93,.

[2]

G. Litjens, Kooi T., B.E. Bejnordi, A.A.A. Setio, F. Ciompi, M. Ghafoorian, et al., A survey on deep learning in medical image analysis, Med Image Anal 42 (2017) 60–88,.

[3]

M.W. Wintergerst, T. Schultz, J. Birtel, A.K. Schuster, N. Pfeiffer, S. Schmitz-Valckenberg, et al., Algorithms for the automated analysis of age-related macular degeneration biomarkers on optical coherence tomography: a systematic review, Transl Vis Sci Technol 6 (4) (2017) 10,.

[4]

S. Gorgi Zadeh, M. Wintergerst, T. Schultz, Uncertainty-guided semi-automated editing of CNN-based retinal layer segmentations in optical coherence tomography, Proceedings of the visual computing for biology and medicine, 2018, pp. 107–115.

[5]

S.D. Olabarriaga, A.W.M. Smeulders, Interaction in the segmentation of medical images: a survey, Med Image Anal 5 (2) (2001) 127–142,.

[6]

D. Haehn, S. Knowles-Barley, M. Roberts, J. Beyer, N. Kasthuri, J.W. Lichtman, et al., Design and evaluation of interactive proofreading tools for connectomics, IEEE Trans Vis Comput Graph 20 (12) (2014) 2466–2475,.

[7]

W. Valenzuela, S.J. Ferguson, D. Ignasiak, G. Diserens, L. Häni, R. Wiest, et al., FISICO: Fast Image SegmentatIon COrrection, PLoS One 11 (5) (2016) e0156035,.

[8]

F. Heckel, J.H. Moltz, C. Tietjen, H.K. Hahn, Sketch-based editing tools for tumour segmentation in 3d medical images, Comput Graph Forum 32 (8) (2013) 144–157,.

[9]

A. Neumann, C. Lorenz, Statistical shape model based segmentation of medical images, Comput Med Imag Graph 22 (2) (1998) 133–143.

[10]

P. Carpineto, M. Ciancaglini, A. Aharrh-Gnama, D. Cirone, L. Mastropasqua, Custom measurement of retinal nerve fiber layer thickness using stratus oct in normal eyes, Eur J Ophthalmol 15 (3) (2005) 360–366.

[11]

U.E. Wolf-Schnurrbusch, L. Ceklic, C.K. Brinkmann, M.E. Iliev, M. Frey, S.P. Rothenbuehler, et al., Macular thickness measurements in healthy eyes using six different optical coherence tomography instruments, Invest Ophthalmol Vis Sci 50 (7) (2009) 3432–3437.

[12]

M. Mayer, Automated glaucoma detection with optical coherence tomography, Friedrich-Alexander-Universität Erlangen-Nürnberg, 2018.

[13]

R. Cárdenes, R. de Luis García, M.B. Cuadra, A multidimensional segmentation evaluation for medical image data, Comput Methods Programs Biomed 96 (2) (2009) 108–124,.

Digital Library

[14]

T. von Landesberger, G.L. Andrienko, N.V. Andrienko, S. Bremm, M. Kirschner, S. Wesarg, et al., Opening up the “black box” of medical image segmentation with statistical shape models, Vis Comput 29 (9) (2013) 893–905,.

Digital Library

[15]

T. von Landesberger, D. Basgier, M. Becker, Comparative local quality assessment of 3D medical image segmentations with focus on statistical shape model-based algorithms, IEEE Trans Vis Comput Graph 22 (12) (2016) 2537–2549,.

Digital Library

[16]

A. Geurts, G. Sakas, A. Kuijper, M. Becker, T. von Landesberger, Visual comparison of 3D medical image segmentation algorithms based on statistical shape models, Proceedings of the international conference on digital human modeling (DHM), LNCS, 9185, Springer, 2015, pp. 336–344,.

[17]

R.G. Raidou, F.J.J. Marcelis, M. Breeuwer, M.E. Gröller, A. Vilanova, H.M.M. van de Wetering, Visual analytics for the exploration and assessment of segmentation errors, Proceedings of the Eurographics workshop on visual computing for biology and medicine, VCBM 16, 2016, pp. 193–202,.

[18]

J.-S. Prassni, T. Ropinski, K. Hinrichs, Uncertainty-aware guided volume segmentation, IEEE Trans Vis Comput Graph 16 (6) (2010) 1358–1365.

[19]

B. Summa, J. Tierny, V. Pascucci, Visualizing the uncertainty of graph-based 2d segmentation with min-path stability, Comput Graph Forum 36 (3) (2017) 133–143,.

Digital Library

[20]

A. Saad, G. Hamarneh, T. Möller, Exploration and visualization of segmentation uncertainty using shape and appearance prior information, IEEE Trans Vis Comput Graph 16 (6) (2010) 1366–1375,.

Digital Library

[21]

Tzeng F.-Y., Ma K.-L., Opening the black box – data driven visualization of neural networks, in: Silva C., Gröller E., Rushmeier H. (Eds.), Proceedings of the IEEE visualization, 2005, pp. 383–390.

[22]

D. Erhan, Y. Bengio, A. Courville, P. Vincent, Visualizing higher-layer features of a deep network, Tech. Rep., University of Montreal, 2009.

[23]

M.D. Zeiler, R. Fergus, Visualizing and understanding convolutional networks, Proceedings of the European conference on computer vision (ECCV), LNCS, 8689, Springer, 2014, pp. 818–833.

[24]

A. Mahendran, A. Vedaldi, Understanding deep image representations by inverting them, Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2015, pp. 5188–5196.

[25]

J. Yosinski, J. Clune, A.M. Nguyen, T.J. Fuchs, H. Lipson, Understanding neural networks through deep visualization, CoRR (2015).

[26]

P.E. Rauber, S.G. Fadel, A.X. Falcao, A.C. Telea, Visualizing the hidden activity of artificial neural networks, IEEE Trans Vis Comput Graphics 23 (1) (2017) 101–110.

[27]

K. Wongsuphasawat, D. Smilkov, J. Wexler, J. Wilson, D. Mane, D. Fritz, et al., Visualizing dataflow graphs of deep learning models in tensorflow, IEEE Trans Vis Comput Graph 24 (1) (2018) 1–12.

[28]

M. Kahng, P.Y. Andrews, A. Kalro, D.H. Chau, ActiVis: Visual exploration of industry-scale deep neural network models, IEEE Trans Vis Comput Graph 24 (1) (2018) 88–97.

[29]

H. Strobelt, S. Gehrmann, H. Pfister, A.M. Rush, LSTMVis: a tool for visual analysis of hidden state dynamics in recurrent neural networks, IEEE Trans Vis Comput Graph 24 (1) (2018) 667–676.

[30]

Liu M., Shi J., Cao K., Zhu J., Liu S., Analyzing the training processes of deep generative models, IEEE Trans Vis Comput Graph 24 (1) (2018) 77–87.

[31]

Liu M., Shi J., Li Z., Li C., Zhu J., Liu S., Towards better analysis of deep convolutional neural networks, IEEE Trans Vis Comput Graph 23 (1) (2017) 91–100.

Digital Library

[32]

N. Pezzotti, T. Höllt, J.V. Gemert, B.P. Lelieveldt, E. Eisemann, A. Vilanova, DeepEyes: progressive visual analytics for designing deep neural networks, IEEE Trans Vis Comput Graph 24 (1) (2018) 98–108.

[33]

I. Roesch, T. Günther, Visualization of neural network predictions for weather forecasting, in: Hullin M., Klein R., Schultz T., Yao A. (Eds.), Proceedings of the vision, modeling, visualization (VMV), The Eurographics Association, 2017, pp. 61–68.

[34]

K.C. Potter, S. Gerber, E.W. Anderson, Visualization of uncertainty without a mean, IEEE Comput Graph Appl 33 (1) (2013) 75–79,.

Digital Library

[35]

A. Al-Taie, H.K. Hahn, L. Linsen, Uncertainty estimation and visualization in probabilistic segmentation, Comput Graph 39 (2014) 48–59,.

Digital Library

[36]

R.D. Jager, W.F. Mieler, J.W. Miller, Age-related macular degeneration, New Engl J Med 358 (24) (2008) 2606–2617.

[37]

M.D. Abràmoff, M.K. Garvin, M. Sonka, Retinal imaging and image analysis, IEEE Rev Biomed Eng 3 (2010) 169–208.

[38]

S. Gorgi Zadeh, M.W. Wintergerst, V. Wiens, S. Thiele, F.G. Holz, R.P. Finger, et al., CNNs enable accurate and fast segmentation of drusen in optical coherence tomography, Proceedings of the deep learning in medical image analysis and multimodal learning for clinical decision support, Springer, 2017, pp. 65–73.

[39]

S. Apostolopoulos, S.D. Zanet, C. Ciller, S. Wolf, R. Sznitman, Pathological OCT retinal layer segmentation using branch residual u-shape networks, Proceedings of the medical image computing and computer assisted intervention (MICCAI), Part III, 2017, pp. 294–301,.

Digital Library

[40]

L. Fang, Cunefare D., Wang C., R.H. Guymer, Li S., S. Farsiu, Automatic segmentation of nine retinal layer boundaries in OCT images of non-exudative AMD patients using deep learning and graph search, Biomed Opt Express 8 (5) (2017) 2732,.

[41]

J. Hamwood, D. Alonso-Caneiro, S.A. Read, S.J. Vincent, M.J. Collins, Effect of patch size and network architecture on a convolutional neural network approach for automatic segmentation of OCT retinal layers, Biomed Opt Express 9 (7) (2018) 3049,.

[42]

Chen Q., Leng T., Zheng L., L. Kutzscher, Ma J., L. de Sisternes, et al., Automated drusen segmentation and quantification in SD-OCT images, Med Image Anal 17 (8) (2013) 1058–1072.

[43]

O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, Proceedings of the international conference on medical image computing and computer-assisted intervention (MICCAI), Springer, 2015, pp. 234–241.

[44]

S. Skiena, Dijkstra’s algorithm, Proceedings of the implementing discrete mathematics: combinatorics and graph theory with mathematica, Addison-Wesley, Reading, MA, 1990, pp. 225–227.

[45]

M.G. Kendall, The treatment of ties in ranking problems, Biometrika 33 (3) (1945) 239–251.

[46]

G. Papandreou, Chen L.-C., K.P. Murphy, A.L. Yuille, Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation, Proceedings of the ieee international conference on computer vision (ICCV), IEEE, 2015, pp. 1742–1750,.

Digital Library

Cited By

Goyanes ELeyva SHerrero Pde Moura JNovo JOrtega M(2024)Fully-automatic end-to-end approaches for 3D drusen segmentation in Optical Coherence Tomography imagesProcedia Computer Science10.1016/j.procs.2024.09.529246:C(1100-1109)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.procs.2024.09.529

Index Terms

Intelligent interaction and uncertainty visualization for efficient drusen and retinal layer segmentation in Optical Coherence Tomography

Index terms have been assigned to the content through auto-classification.

Recommendations

Adaptive Segmentation Method for Evaluating of Choroidal Thickness on Optical Coherence Tomography
ICDSP '18: Proceedings of the 2nd International Conference on Digital Signal Processing

Optical coherence tomography (OCT) is a medical imaging technique that has been used for many years, and it can capture the three-dimensional image from retina in a stable, quick and noninvasive way. In many clinical diagnoses from OCT imaging, ...
Minimising retinal vessel artefacts in optical coherence tomography images

Optical coherence tomography (OCT) is commonly used to investigate the layers of the retina including retinal nerve fiber layer (RNFL) and retinal pigment epithelium (RPE). OCT images are altered by vessels on the retinal surface producing artefacts. We ...
A benchmark study of automated intra-retinal cyst segmentation algorithms using optical coherence tomography B-scans

The different automated OCT intra-retinal cyst segmentation methods are reviewed and comparatively analysed for quantitative and qualitative benchmarking purposes.A modular approach to standardize the existing cyst segmentation methods is presented for ...

Comments

Information & Contributors

Information

Published In

cover image Computers and Graphics

Computers and Graphics Volume 83, Issue C

Oct 2019

122 pages

ISSN:0097-8493

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 October 2019

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Goyanes ELeyva SHerrero Pde Moura JNovo JOrtega M(2024)Fully-automatic end-to-end approaches for 3D drusen segmentation in Optical Coherence Tomography imagesProcedia Computer Science10.1016/j.procs.2024.09.529246:C(1100-1109)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.procs.2024.09.529

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents