Real-Time Resolution Enhancement of Confocal Laser Scanning Microscopy via Deep Learning

Abstract

Confocal laser scanning microscopy is one of the most widely used tools for high-resolution imaging of biological cells. However, the imaging resolution of conventional confocal technology is limited by diffraction, and more complex optical principles and expensive optical-mechanical structures are usually required to improve the resolution. This study proposed a deep residual neural network algorithm that can effectively improve the imaging resolution of the confocal microscopy in real time. The reliability and real-time performance of the algorithm were verified through imaging experiments on different biological structures, and an imaging resolution of less than 120 nm was achieved in a more cost-effective manner. This study contributes to the real-time improvement of the imaging resolution of confocal microscopy and expands the application scenarios of confocal microscopy in biological imaging.

1. Introduction

Confocal microscopy plays an important role in numerous scientific research areas. The resolution and other performances of traditional confocal microscopy are slightly better than those of wide-field fluorescence microscopy. The pinhole, the core component of confocal microscopy, can provide the system with a high signal-to-noise ratio and optical sectioning ability by rejecting most of the out-of-focus light. However, the imaging resolution limit of conventional confocal microscopy, which is approximately 180 nm, remains limited by diffraction and cannot satisfy the requirements for observing finer biological structures.

Recently, a series of methods have been proposed to improve the imaging performance of confocal microscopy. Heintzmann. et al. proposed confocal subtraction imaging, which can improve the resolution of confocal microscopes by capturing two pinhole images of different sizes and performing a weighted subtraction [1]. Di Franco. et al. used a lifetime tuning algorithm to process a series of pinhole images of different sizes and reduce the confocal defocus background [2]. However, subtraction imaging cannot significantly improve the imaging resolution of the confocal microscopy. Kuang. et al. proposed fluorescence difference microscopy (FED) and achieved a spatial resolution smaller than λ/4 by subtracting the intensity of solid and hollow spots with coefficients [3,4]. Spinning disk confocal microscopy uses a pinhole array to scan samples, increasing the imaging speed by hundreds of times. It can achieve a resolution of approximately 120 nm when combined with a back-end deconvolution algorithm [5,6,7]. Airyscan uses a 32-channel GaAsP detector array to simultaneously improve the signal-to-noise ratio and resolution to approximately 120 nm [8,9]. Most of these methods increase the resolution to 120 nm which is beyond the diffraction limit by adding the complex and excessively expensive optical components.

In contrast, from an algorithmic perspective, improving the imaging performance of confocal microscopy is both efficient and more cost-effective. Nicolas. et al. proposed a more robust Richardson–Lucy algorithm with total variation regularization to improve the image quality of the 3D confocal microscopy [10]. Dupé. et al. used a fast proximal backward forward split iterative deconvolution algorithm to improve image quality at low signal-to-noise ratios [11]. He. et al. employed prominent total variation regularization to enhance the sparsity of confocal images and reduce artifacts [12]. In addition, combining adaptive optics with microscope systems can improve the quality of focused beams, reduce aberrations, and effectively increase imaging depth [13,14,15,16,17].

Presently, deep learning is playing an increasingly important role in improving the imaging performance of confocal microscopy. Martin. et al. combined confocal microscopy with a content-aware restoration network to achieve higher resolution at higher rates and lower light intensities [18]. Li. et al. proposed a back-projection generative adversarial network that can achieve a 64 times higher imaging speed of confocal microscopy [19]. Wang. et al. realized a resolution comparable to that of a pixel reassignment reconstruction algorithm using deep learning methods [20]. Fang. et al. obtained training pairs by downsampling and adding noise to high-resolution confocal images. They used information from adjacent frames to reduce flicker artifacts, thus improving the resolution and signal-to-noise ratio of confocal microscopy [21]. Huang. et al. proposed a dual-channel attention network that builds a training set of confocal and stimulated emission depletion (STED) images to learn the mapping relationship from low to high resolution, thereby improving the resolution of confocal microscopy [22]. Although STED has a higher imaging resolution, both structured illumination microscopy (SIM) and confocal microscopy are low-phototoxicity fluorescence imaging techniques, and their application scenarios are similar. In addition, the real-time performance of the output images of the above works still needs to be improved.

In this paper, we propose a deep learning algorithm based on an end-to-end deep residual neural network to improve the resolution of confocal microscopy and the real-time performance of the algorithm. Confocal microscopy and SIM were used on the same host as low- and high-resolution systems, respectively. Training pairs with pixel-level alignment in the same tissue area could be easily obtained in real-time, and a real-shot training set could be quickly constructed by adjusting the imaging system parameters. The trained neural network was experimentally verified to significantly improve the resolution of confocal microscopy to approximately 120 nm in real-time. We experimentally demonstrated the improved resolution of confocal microscopy using fluorescent beads and different biological samples. The real-time performance of the algorithm was verified in the fluorescence imaging of mitochondria in living cells.

2. Materials and Methods

We developed a deep learning algorithm that can realize instant super-resolution imaging using confocal microscopy. As shown in Figure 1, a deep learning dataset was constructed through experiments and simulations. We used confocal microscopy (NCF1000, Ningbo YongXin Optics Co., Ltd., Ningbo, China) and SIM [23] (NSR1000, Ningbo YongXin Optics Co., Ltd., Ningbo, China) on the same microscopy host. The voltage of the scanning galvanometer of the confocal microscope was changed, and its field of view was adjusted so that the fields of view of the confocal image and the SIM-reconstructed images were consistent. Low-resolution images (confocal) and high-resolution (SIM) images of the same sample were obtained directly to constitute a dataset. The 1:1 dataset can reduce the processing time of the prediction algorithm. The samples used in the experiment included 100 nm beads (Nanoparticles 4C flour 100 nm slide, Abberior Instruments, Gottingen, Germeny), mitochondria (labeled with 250 nM PK Mito Green in U20S cells, prepared by Zhang Yuhui’s group at Huazhong University of Science and Technology, Wuhan, China) and microtubes (GATTA-Cells 4C, labeled with Alexa Fluor 555, GATTAQUANT, Brunswick, Germany). The excitation and central emission wavelength of 100 nm fluorescent beads were 488 nm and 520 nm, respectively. The illumination wavelengths for mitochondria and microtubes are 488 nm and 561 nm, respectively. The experiment was conducted under a 100× objective lens (numerical aperture 1.49, Ningbo YongXin Optics Co., Ltd., Ningbo, China). In addition, the high-resolution images of clathrin-coated pit (CCP) were obtained from the SIM-reconstructed images of CCP in open-source BioSR [24], and the low-resolution images of CCP were obtained by using simulation algorithms to simulate the confocal imaging process. Confocal images and SIM super-resolution images were used as Low Resolution (LR) and High Resolution (HR) image pairs, which were divided into training and validation datasets in a ratio of 9:1. The training dataset was down-sampled multiple times using the Res U-Net [25] network to learn the mapping features between image pairs. The graph of the first prediction result for the LR data of the validation set was restored based on the current network. Then, the difference between the prediction results and the HR data of the validation set was calculated based on the loss function to change the weight factor. The samples in the training dataset were retrained until the loss function converged. The weight factors were saved, the network was exported when the loss function was minimal, and the training process was complete.

The network structure of the proposed Res U-Net, which includes modules such as convolution, downsampling and upsampling, is shown in Figure 2. First, the input LR image is convolved to obtain the LR feature map, which is then down-sampled. Four downsampling modules are responsible for extracting image features from the LR image. Each down-sampling process includes 3 × 3 convolution layers, a rectified linear unit planning function (ReLU), and a maximum pooling layer. The ReLU module can improve the nonlinearity of the neural network and alleviate the gradient disappearance problem. The spatial dimensions can be reduced by a maximum pooling layer with a 2 × 2 pool size. Next, the HR feature map is generated through four up-sampling modules. The up-sampling module includes 3 × 3 convolution layers, ReLU and a maximum pooling layer. The HR image can be output by convolving the HR feature map. The down- and up-sampling layers in each layer of the network can add a feature map directly from the input to the output through jump connections to retain the original input details and expand the network depth. We slightly reduced the number of network channels from 64 to 32 to improve the imaging speed. The loss function $\mathcal{L}$ uses a combination of the mean square error (MSE) loss [26] and structural similarity (SSIM) loss [27], as shown in Equation (1). $I_{predict}$ and $I_{HR}$ represent the image of algorithm prediction and high resolution, respectively. The MSE solves the mean square error between the HR and predicted images to control the accuracy of the network prediction graph, whereas the SSIM controls the perceptual quality of the network prediction model. The more detailed network structure can be found in Supplement S1.

$$\mathcal{L}\left(I_{predict},I_{HR}\right)=\mathrm{M}\mathrm{S}\mathrm{E}\left(I_{predict},I_{HR}\right)+\left[1-\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}\left(I_{predict},I_{HR}\right)\right]$$

(1)

The network was implemented using Python 3.9.18, CUDA 11.6, and Keras 2.10.0 [28,29,30]. The entire training and prediction processes were performed using an NVIDIA Quadro RTX4000 (8 GB). The experimental dataset included 58,319 pairs of training sets and 6247 pairs of validation sets (128 × 128 pixels). The CCP dataset from BioSR contained 5598 pairs of training sets and 516 pairs of validation sets (128 × 128 pixels). We selected the Adam Optimizer [31] with a learning rate of 0.0001, and the initial number of iterations was set to a maximum of 40,000. The hyperparameters β₁ and β₂ were 0.9 and 0.999, respectively. The training time for the entire dataset was approximately 9 h.

3. Results and Discussion

To verify the accuracy of the deep learning network, we tested 100 nm fluorescently labeled beads and biological samples (microtubes and mitochondria) using confocal microscopy (NCF1000). The experimental results for the beads are shown in Figure 3, where Figure 3a–c are the images from confocal microscopy, deep learning algorithm prediction, and SIM reconstruction, respectively. Figure 3d–f show enlarged images of the solid-line boxes on the right sides of Figure 3a–c. Figure 3g–i show enlarged images of the dotted-line boxes on the left sides of Figure 3a–c. The illumination wavelength is 488 nm. A comparison of the results of the three imaging modes shows that the two closely arranged 100 nm beads shown in Figure 3d,g could not be clearly distinguished via the confocal system. However, they could be clearly distinguished via the algorithm predictions shown in Figure 3e,h and the SIM-reconstructed images shown in Figure 3f,i. To further compare the resolutions of the three imaging modes, we make cross-sections (dashed lines in the figure, showing three 100 nm beads with different spacings) in the same area of Figure 3d–f. The normalized intensity distributions of the three curves are shown in Figure 3j. Confocal imaging (yellow curve) reveals only two peaks, and the two 100 nm beads on the left cannot be clearly distinguished. The deep learning algorithm prediction (blue curve) and SIM reconstruction (green curve) reveal three peaks. The two 100 nm beads on the left can be clearly distinguished based on the Rayleigh criterion. The full width at half maximum (FWHM) of the rightmost beads in the confocal curve was 240.75 nm, the FWHM of the rightmost beads in the prediction curve was 125.20 nm and the FWHM of the SIM reconstruction was 122.20 nm. Compared to confocal imaging, the resolution gains of the algorithm prediction and SIM reconstruction were 1.92 and 1.97 times, respectively. Figure 3k shows the normalized intensity distribution of the three cross-sections in Figure 3g–i. The three beads that could not be distinguished via confocal imaging, and were clearly distinguished in both algorithm prediction and SIM-reconstructed images. The FWHM of the beads on the right side in confocal imaging was 247.25 nm, and the FWHM of the beads on the right side in both the algorithm prediction and SIM was 126.50 nm. The resolution gain was 1.95 times. We measured the FWHM of 20 dispersed fluorescent beads in Figure 3a–c, and plotted the measurement results in Figure 3l. The average FWHM of confocal microscopy, algorithm prediction and SIM were 243.82 ± 13.03 nm, 125.24 ± 8.92 nm and 121.29 ± 7.15 nm, respectively. Moreover, the peak signal-to-noise ratio (PSNR) and the structural similarity index measure (SSIM) of the algorithm-predicted image were 29.56 and 0.89, respectively. The image pixel resolution in Figure 3a–c was 512 × 512, and the algorithm calculation time was approximately 0.09 s.

To further verify the reliability of the algorithm, we trained and experimentally verified the biological samples. Our training dataset involved microtubes, CCP, and mitochondria. The structural diversity of the test samples constituted a rigorous test of the adaptability and versatility of the proposed algorithm. Figure 4 compares the experimentally measured microtubes obtained using the three imaging methods. The deep learning algorithm significantly improved the resolution of the confocal image, particularly at the boundary. The contrast in the details between the predicted and confocal images is evident. The background between the filamentous structures in the four deep-learning algorithm-predicted images was significantly suppressed. The pixel resolution in Figure 4a–d was 1024 × 1024, and the algorithm calculation time was approximately 0.26 s. We compared the speed of our algorithm with those of existing point-scanning deep learning algorithms [31], as shown in

Table 1. Speed comparison of different algorithms.

Pixels	Ours (frame/s)	Reference [32] (frame/s)
512 × 512	≈11.11	≈2.43
1024 × 1024	≈3.85	≈1.45

. Our algorithm was 4.57 times faster than the reference at 512 × 512 pixels and 2.66 times faster at 1024 × 1024 pixels.

To further evaluate the image quality in Figure 4a–d, the following image quality evaluation functions were introduced: PSNR, RMSE and SSIM [33]. The image quality evaluation functions for the four images are show in Figure 5. The average PSNR was approximately 23.36, the average SSIM was approximately 0.86, and the average RMSE was approximately 0.146.

To generalize the algorithm, we added the CCP to the open-source dataset BioSR for simulation testing. Confocal images were generated using the wide-field image of the CCP in the open-source dataset instead of the SIM-reconstructed image as the original image to increase the prediction ability of the network in low-SNR confocal imaging. Numerical simulation was conducted on confocal imaging with a pinhole size of 1 Airy unit. The training and validation sets were formed together with the SIM-reconstructed images. Figure 6a shows confocal simulation, algorithm prediction, and SIM reconstruction images of the same sample. Compared with Figure 6b–d on the right, the algorithm-predicted and SIM-reconstructed images clearly distinguish the hollow structure of the CCP. Figure 6h shows the normalized intensity along the dotted lines in Figure 6e–f. The confocal imaging cannot distinguish the hollow structure of CCP. The deep learning algorithm could distinguish the spacing of the hollow structure to be approximately 113.75 nm, and the hollow spacing measured in the SIM-reconstructed image was approximately 145.00 nm. Although the algorithm could distinguish hollow results, the small number of samples led to a slight gap between the prediction results of the hollow structures and the SIM reconstruction results.

To further verify the real-time performance of the algorithm, we performed the algorithm prediction on the fluorescently labeled mitochondrial structure in living cells. The confocal microscopy and algorithm prediction results are shown in Figure 7a. By comparing Figure 7b,c, a mitochondrial ridge structure that could not be observed in the confocal image was observed in the algorithm-predicted image, and the resolution was significantly improved. We drew a series of time-lapse deep-learning algorithm-predicted images of the mitochondria in living cells with a pixel resolution of 512 × 512 and enlarged one of the areas, as shown in Figure 7d. The calculation time of a single algorithm-predicted image was approximately 0.09 s, and the confocal imaging speed was 4 fps, which fully satisfied the speed requirements of deep-learning real-time super-resolution prediction imaging of the confocal microscopy at 512 × 512 pixels. Our deep learning algorithm can provide real-time resolution enhancement for the confocal system. The figure shows that the morphologies of the two mitochondria changed continuously over time. We used two arrows to qualitatively depict the movement trends of the two mitochondrial structures in Figure 7d. The two arrows represent directions of movement of two types of cell structures, respectively. From the direction change in the arrow in Figure 7d and Supplementary Video S1, it can be seen that the two cell structures have undergone contraction and relaxation changes over time. The current computing speed fully meets the speed requirements of our confocal system. In the future, it is possible to simply replace better Graphics cards or adopt lighter networks to further improve the speed of algorithm calculations. The current network is suitable for predicting the aforementioned four types of structures. When different types of biological structures need to be predicted, the model requires additional training.

4. Conclusions

In this study, we proposed a deep learning algorithm based on Res U-Net that can effectively improve the imaging resolution of a confocal imaging system to approximately 120 nm in real time. We used confocal microscopy (NCF 1000) and SIM (NSR 1000) to build an experimental platform for quickly constructing the dataset. The confocal and SIM images constituted a low-resolution–high-resolution data pair. A supervised dataset could be quickly built by adjusting the field-of-view size of the confocal image and the pixel-by-pixel correspondence of the SIM-reconstructed image. The network was trained using the constructed experimental dataset and open-source data. Through simulation and experimental verification, the reliability of the deep learning algorithm was demonstrated on multiple samples: 100 nm beads, microtubes, and CCP. In the dynamic imaging of mitochondrial structures in living cells, the resolution of the confocal microscopy was improved in real-time to 512 × 512 pixels. In conclusion, our deep learning algorithm can help confocal systems achieve SIM-level imaging resolution in real time, avoid the construction of complex hardware equipment and facilitate organelle-level dynamic research.