research-article

NeuroConstruct: 3D Reconstruction and Visualization of Neurites in Optical Microscopy Brain Images

Authors:

Parmida Ghahremani,

Saeed Boorboor,

Pooya Mirhosseini,

Chetan Gudisagar,

Arie E. KaufmanAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 28, Issue 12

Pages 4951 - 4965

https://doi.org/10.1109/TVCG.2021.3109460

Published: 01 December 2022 Publication History

Abstract

We introduce NeuroConstruct, a novel end-to-end application for the segmentation, registration, and visualization of brain volumes imaged using wide-field microscopy. NeuroConstruct offers a Segmentation Toolbox with various annotation helper functions that aid experts to effectively and precisely annotate micrometer resolution neurites. It also offers an automatic neurites segmentation using convolutional neuronal networks (CNN) trained by the Toolbox annotations and somas segmentation using thresholding. To visualize neurites in a given volume, NeuroConstruct offers a hybrid rendering by combining iso-surface rendering of high-confidence classified neurites, along with real-time rendering of raw volume using a 2D transfer function for voxel classification score versus voxel intensity value. For a complete reconstruction of the 3D neurites, we introduce a Registration Toolbox that provides automatic coarse-to-fine alignment of serially sectioned samples. The quantitative and qualitative analysis show that NeuroConstruct outperforms the state-of-the-art in all design aspects. NeuroConstruct was developed as a collaboration between computer scientists and neuroscientists, with an application to the study of cholinergic neurons, which are severely affected in Alzheimer's disease.

References

[1]

T. H. Ferreira-Vieira, I. M. Guimaraes, F. R. Silva, and F. M. Ribeiro, “Alzheimer's disease: Targeting the cholinergic system,” Curr. Neuropharmacol., vol. 14, pp. 101–115, 2016.

[2]

S. Boorboor, S. Jadhav, M. Ananth, D. Talmage, L. Role, and A. Kaufman, “Visualization of neuronal structures in wide-field microscopy brain images,” IEEE Trans. Vis. Comput. Graph., vol. 25, no. 1, pp. 1018–1028, Aug. 2018.

[3]

E. C. Ballinger, M. Ananth, D. A. Talmage, and L. W. Role, “Basal forebrain cholinergic circuits and signaling in cognition and cognitive decline,” Neuron, vol. 91, pp. 1199–1218, 2016.

[4]

J. R. Glaser and E. M. Glaser, “Neuron imaging with neurolucida— PC-based system for image combining microscopy,” Comput. Med. Imag. Graph., vol. 14, pp. 307–317, 1990.

[5]

H. Peng, Z. Ruan, F. Long, J. H. Simpson, and E. W. Myers, “V3d enables real-time 3D visualization and quantitative analysis of large-scale biological image data sets,” Nat. Biotechnol., vol. 28, no. 4, pp. 348–353, 2010.

[6]

C. Magliaro, A. L. Callara, N. Vanello, and A. Ahluwalia, “A manual segmentation tool for three-dimensional neuron datasets,” Front. Neuroinform., vol. 11, 2017, Art. no.

[7]

H. Ikeno, A. Kumaraswamy, K. Kai, T. Wachtler, and H. Ai, “A segmentation scheme for complex neuronal arbors and application to vibration sensitive neurons in the honeybee brain,” Front. Neuroinform., vol. 12, 2018, Art. no.

[8]

H. Peng, F. Long, and G. Myers, “Automatic 3D neuron tracing using all-path pruning,” Bioinformatics, vol. 27, pp. 239–247, 2011.

Digital Library

[9]

M. H. Longair, D. A. Baker, and J. D. Armstrong, “Simple neurite tracer: Open source software for reconstruction, visualization and analysis of neuronal processes,” Bioinformatics, vol. 27, pp. 2453–2454, 2011.

Digital Library

[10]

T. Quanet al., “NeuroGPS-tree: Automatic reconstruction of large-scale neuronal populations with dense neurites,” Nat. Methods, vol. 13, pp. 51–54, 2016.

[11]

Y. Wang, A. Narayanaswamy, C.-L. Tsai, and B. Roysam, “A broadly applicable 3D neuron tracing method based on open-curve snake,” Neuroinformatics, vol. 9, pp. 193–217, 2011.

[12]

S. Liet al., “Optimization of traced neuron skeleton using lasso-based model,” Front. Neuroanatomy., vol. 13, 2019, Art. no.

[13]

S. Basu and D. Racoceanu, “Reconstructing neuronal morphology from microscopy stacks using fast marching,” in Proc. IEEE Int. Conf. Image Process., 2014, pp. 3597–3601.

[14]

J. Yang, M. Hao, X. Liu, Z. Wan, N. Zhong, and H. Peng, “FMST: An automatic neuron tracing method based on fast marching and minimum spanning tree,” Neuroinformatics, vol. 17, pp. 185–196, 2019.

[15]

H. Xiao and H. Peng, “APP2: Automatic tracing of 3D neuron morphology based on hierarchical pruning of a gray-weighted image distance-tree,” Bioinformatics, vol. 29, pp. 1448–1454, 2013.

Digital Library

[16]

H. Chen, H. Xiao, T. Liu, and H. Peng, “SmartTracing: Self-learning-based neuron reconstruction,” Brain Inform., vol. 2, pp. 135–144, 2015.

[17]

R. Li, T. Zeng, H. Peng, and S. Ji, “Deep learning segmentation of optical microscopy images improves 3D neuron reconstruction,” IEEE Trans. Med. Imag., vol. 36, no. 7, pp. 1533–1541, Jul. 2017.

[18]

Z. Zhou, H.-C. Kuo, H. Peng, and F. Long, “DeepNeuron: An open deep learning toolbox for neuron tracing,” Brain Inform., vol. 5, pp. 1–9, 2018.

[19]

A. Fakhry, T. Zeng, and S. Ji, “Residual deconvolutional networks for brain electron microscopy image segmentation,” IEEE Trans. Med. Imag., vol. 36, no. 2, pp. 447–456, Feb. 2016.

[20]

M. Januszewskiet al., “High-precision automated reconstruction of neurons with flood-filling networks,” Nat. Methods, vol. 15, pp. 605–610, 2018.

[21]

K. Luther and H. S. Seung, “Learning metric graphs for neuron segmentation in electron microscopy images,” in Proc IEEE Int. Symp. Biomed. Imag., 2019, pp. 244–248.

[22]

D. Haehnet al., “Design and evaluation of interactive proofreading tools for connectomics,” IEEE Trans. Vis. Comput. Graph., vol. 20, no. 12, pp. 2466–2475, Dec. 2014.

[23]

P. Bajcsy, S.-C. Lee, A. Lin, and R. Folberg, “Three-dimensional volume reconstruction of extracellular matrix proteins in Uveal Melanoma from fluorescent confocal laser scanning microscope images,” J. Microsc., vol. 221, pp. 30–45, 2006.

[24]

T. Juet al., “3D volume reconstruction of a mouse brain from histological sections using warp filtering,” J. Neurosci. Methods, vol. 156, pp. 84–100, 2006.

[25]

H. Liang, N. Dabrowska, J. Kapur, and D. S. Weller, “Whole brain reconstruction from multilayered sections of a mouse model of status epilepticus,” in Proc. Asilomar Conf. Signals, Syst., Comput., 2017, pp. 1260–1263.

[26]

K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3D point sets,” IEEE Trans. Pattern Anal. Mach. Intell., vol. PAMI-9, no. 5, pp. 698–700, Sep. 1987.

Digital Library

[27]

J. Luck, C. Little, and W. Hoff, “Registration of range data using a hybrid simulated annealing and iterative closest point algorithm,” in Proc. IEEE Int. Conf. Robot. Automat., 2000, pp. 3739–3744.

[28]

C.-L. Tsaiet al., “Robust, globally consistent and fully automatic multi-image registration and montage synthesis for 3D multi-channel images,” J. Microsc., vol. 243, pp. 154–171, 2011.

[29]

M. Yigitsoy and N. Navab, “Structure propagation for image registration,” IEEE Trans. Med. Imag., vol. 32, no. 9, pp. 1657–1670, Sep. 2013.

[30]

S.-C. Lee and P. Bajcsy, “Trajectory fusion for 3D volume reconstruction,” Comput. Vis. Image Understanding, vol. 110, pp. 19–31, 2008.

Digital Library

[31]

V. J. Dercksen, H.-C. Hege, and M. Oberlaender, “The filament editor: An interactive software environment for visualization, proof-editing and analysis of 3D neuron morphology,” Neuroinformatics, vol. 12, pp. 325–339, 2014.

[32]

H. Chen, D. M. Iascone, N. M. da Costa, E. S. Lein, T. Liu, and H. Peng, “Fast assembling of neuron fragments in serial 3D sections,” Brain Inform., vol. 4, 2017, Art. no.

[33]

H. Pfisteret al., “Visualization in connectomics,” in Scientific Visualization, Berlin, Germany: Springer, 2014, pp. 221–245.

[34]

K. Mosaligantiet al., “Reconstruction of cellular biological structures from optical microscopy data,” IEEE Trans. Vis. Comput. Graph., vol. 14, no. 4, pp. 863–876, Jul./Aug. 2008.

Digital Library

[35]

M. Nakaoet al., “Visualizing in vivo brain neural structures using volume rendered feature spaces,” Comput. Biol. Med., vol. 53, pp. 85–93, 2014.

Digital Library

[36]

Y. Wan, H. Otsuna, C.-B. Chien, and C. Hansen, “FluoRender: An application of 2D image space methods for 3D and 4D confocal microscopy data visualization in neurobiology research,” in Proc. IEEE Pacific Vis. Symp., 2012, pp. 201–208.

[37]

J. Beyer, A. Al-Awami, N. Kasthuri, J. W. Lichtman, H. Pfister, and M. Hadwiger, “ConnectomeExplorer: Query-guided visual analysis of large volumetric neuroscience data,” IEEE Trans. Vis. Comput. Graph., vol. 19, no. 12, pp. 2868–2877, Dec. 2013.

Digital Library

[38]

M. Hadwiger, J. Beyer, W. Jeong, and H. Pfister, “Interactive volume exploration of petascale microscopy data streams using a visualization-driven virtual memory approach,” IEEE Trans. Vis. Comput. Graph., vol. 18,no. 12, pp. 2285–2294, Dec. 2012.

Digital Library

[39]

D. Haehnet al., “Scalable interactive visualization for connectomics,” Informatics, vol. 4, 2017, Art. no.

[40]

WebGL-based viewer for volumetric data. Accessed: Oct.12, 2020. [Online]. Available: https://github.com/google/neuroglancer

[41]

O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in Proc. Int. Conf. Med. Image Comput. Comput.-Assist. Interv., 2015, pp. 234–241.

[42]

Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested u-net architecture for medical image segmentation,” in Proc. Deep Learn. Med. Image Anal. Multimodal Learn. Clin. Decis. Support, 2018, pp. 3–11.

[43]

X. Qin, Z. Zhang, C. Huang, M. Dehghan, O. R. Zaiane, and M. Jagersand, “U2-Net: Going deeper with nested U-structure for salient object detection,” Pattern Recognit., vol. 106, 2020, Art. no.

[44]

P.-S. Liaoet al., “A fast algorithm for multilevel thresholding,” J. Inf. Sci. Eng., vol. 17, no. 5, pp. 713–727, 2001.

[45]

B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proc. 7th Int. Joint Conf. Artif. Intell., vol. 2, pp. 674––679, 1981.

[46]

M. Robertset al., “Neural process reconstruction from sparse user scribbles,” in Proc. Int. Conf. Med. Image Comput. Comput.-Assist. Interv., 2011, pp. 621–628.

[47]

T.-C. Lee, R. L. Kashyap, and C.-N. Chu, “Building skeleton models via 3D medial surface axis thinning algorithms,” Graphical Models Image Process., vol. 56, pp. 462–478, 1994.

Digital Library

[48]

U. Bagci and L. Bai, “Automatic best reference slice selection for smooth volume reconstruction of a mouse brain from histological images,” IEEE Trans. Med. Imag., vol. 29, no. 9, pp. 1688–1696, Sep. 2010.

[49]

D. Mattes, D. R. Haynor, H. Vesselle, T. K. Lewellen, and W. Eubank, “PET-CT image registration in the chest using free-form deformations,” IEEE Trans. Med. Imag., vol. 22, no. 1, pp. 120–128, Jan. 2003.

[50]

H. W. Kuhn, “The Hungarian method for the assignment problem,” Nav. Res. Logist. Quart., vol. 2, no. 1/2, pp. 83–97, 1955.

[51]

W. J. Schroeder, B. Lorensen, and K. Martin, The Visualization Toolkit: An Object-oriented Approach to 3D Graphics. New York, NY, USA: Kitware, 2004.

[52]

L. Ibanez, W. Schroeder, L. Ng, and J. Cates, The ITK Software Guide, 1st ed. New York, NY, USA: Kitware, 2003.

[53]

W. S. Rasband, “ImageJ: Image processing and analysis in java,” Astrophysics Source Code Library, vol. 1, pp. 06013, Jun. 2012.

[54]

S. Liu, D. Zhang, S. Liu, D. Feng, H. Peng, and W. Cai, “Rivulet: 3D neuron morphology tracing with iterative back-tracking,” Neuroinformatics, vol. 14, pp. 387–401, 2016.

[55]

Z. Zhou, X. Liu, B. Long, and H. Peng, “TReMAP: Automatic 3D neuron reconstruction based on tracing, reverse mapping and assembling of 2D projections,” Neuroinformatics, vol. 14, pp. 41–50, 2016.

[56]

H. Huanget al., “UNET 3+: A full-scale connected UNet for medical image segmentation,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2020, pp. 1055–1059.

Cited By

Fujiwara TKucher KWang JMartins RKerren AYnnerman A(2025)Adversarial attacks on machine learning-aided visualizationsJournal of Visualization10.1007/s12650-024-01029-228:1(133-151)Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s12650-024-01029-2
Zhao MHuang XJiang JMou LYan DMa L(2024)Accurate Registration of Cross-Modality Geometry via Consistent ClusteringIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324716930:7(4055-4067)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3247169
Afzal SGhani SHittawe MRashid SKnio OHadwiger MHoteit I(2023)Visualization and Visual Analytics Approaches for Image and Video Datasets: A SurveyACM Transactions on Interactive Intelligent Systems10.1145/357693513:1(1-41)Online publication date: 9-Mar-2023
https://dl.acm.org/doi/10.1145/3576935
Show More Cited By

Index Terms

NeuroConstruct: 3D Reconstruction and Visualization of Neurites in Optical Microscopy Brain Images
1. Applied computing
  1. Life and medical sciences
    1. Computational biology
    2. Genetics
2. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

3D Nucleus Instance Segmentation for Whole-Brain Microscopy Images
Information Processing in Medical Imaging
Abstract
Tissue clearing and light-sheet microscopy technologies offer new opportunities to quantify the three-dimensional (3D) neural structure at a cellular or even sub-cellular resolution. Although many efforts have been made to recognize nuclei in 3D ...
3D Volume Reconstruction and Biometric Analysis of Fetal Brain from MR Images
Computational Intelligence Methods for Bioinformatics and Biostatistics

Magnetic resonance imaging (MRI) is becoming increasingly popular as a second-level technique, performed after ultrasonography (US) scanning, for detecting morphologic brain abnormalities. For this reason, several medical researchers in the past few ...
Template-driven segmentation of confocal microscopy images

High quality 3D visualization of anatomic structures is necessary for many applications. The anatomic structures first need to be segmented. A variety of segmentation algorithms have been developed for this purpose. For confocal microscopy images, the ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 28, Issue 12

Dec. 2022

1222 pages

ISSN:1077-2626

Issue’s Table of Contents

1077-2626 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 December 2022

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fujiwara TKucher KWang JMartins RKerren AYnnerman A(2025)Adversarial attacks on machine learning-aided visualizationsJournal of Visualization10.1007/s12650-024-01029-228:1(133-151)Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s12650-024-01029-2
Zhao MHuang XJiang JMou LYan DMa L(2024)Accurate Registration of Cross-Modality Geometry via Consistent ClusteringIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324716930:7(4055-4067)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3247169
Afzal SGhani SHittawe MRashid SKnio OHadwiger MHoteit I(2023)Visualization and Visual Analytics Approaches for Image and Video Datasets: A SurveyACM Transactions on Interactive Intelligent Systems10.1145/357693513:1(1-41)Online publication date: 9-Mar-2023
https://dl.acm.org/doi/10.1145/3576935
Hu PBoorboor SMarino JKaufman A(2023)Geometry-Aware Planar Embedding of Treelike StructuresIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.315387129:7(3182-3194)Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1109/TVCG.2022.3153871
Hou YRen ZHou QTao YJiang YChen W(2023)InstantTrace: fast parallel neuron tracing on GPUsThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-02969-w39:8(3783-3796)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1007/s00371-023-02969-w

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents