Convolutional Neural Networks-Based Approach to Detect Neonatal Respiratory System Anomalies with Limited Thermal Image

In this section, firstly, the thermal imaging process and the disease classes in dataset are described. Then, the data augmentation methods used are explained and how the classification is done with the convolutional neural network model is expressed.

2.1 Used data and measurement setup

Thermal images were taken by the Variocam HD (InfraTec©) infrared camera, and recorded using a user interface named IRBIS, at the Selcuk University Medicine Faculty, Neonatal Intensive Care Unit. The camera resolution is 1.024 × 768 with a thermal sensitivity of about 0.01℃. A total of 34 neonates, 16 with pulmonary diseases, 10 with cardiovascular diseases, 8 with renal diseases with abdominal region anomaly were used. Images were obtained from neonates lying in supine position at a distance of 60 - 100 cm. Thermograms have been converted to raw RGB images by using portable computer and thermal camera application. The measurement setup is shown in Figure 1.

1.png

Figure 1. Thermogram recording process a) thermal camera, b) neonate, c) incubator, d) computer

In order to protect the temperature values of the infants, the image acquisition was set as one minute, and 100 thermal images were obtained from each newborn for approximately 60 seconds. Thus, 3400 thermal images were recorded from 34 newborn infants. In practice, 20 images from each neonate were randomly selected and used. Some of the thermal images used in the application are shown in Figure 2.

2.png

Figure 2. Image samples captured by thermal camera

2.2 Neonatal diseases and image dataset

In the study, neonates whose disease information was labelled by pediatricians were divided into 3 groups which respiratory system anomalies, cardiovascular and abdominal diseases. Then pulmonary diseases were examined in the first class and other diseases in the second class. Grouping process was done as shown in Table 1.

In practice, 320 images of 16 neonates with pulmonary abnormalities and 360 images of 18 neonates with heart diseases, abdominal and renal diseases were used. Of the 34 neonates, 11 were selected and reserved for testing, and 23 were used for training the model.

Table 1. Determination of dataset

2.3 Data augmentation

3.png

Figure 3. Data augmentation process

4.png

Figure 4. Augmented images (a) brightness enhancement, (b) contrast changing, (c) sharpening

Immunity in newborns begins to mature from the 8th week after birth. Infants under observation in NICU are vulnerable to environmental influences and diseases in their first weeks of life [21]. Newborns who are dependent on vital support and medical monitoring devices cannot be removed from the incubator where the temperature balance is maintained. Although thermal imaging is performed without leaving the incubator, the protective top cover of the incubator is opened during imaging and the thermal insulation is broken for a short time. Given this sensitive situation of neonatal intensive care units, creating a thermal image database of thousands of images CNN needs to achieve successful classification can pose medical drawbacks. Another reason that reveals the need to increase data is that it is difficult to find the sufficient sample cases from each of the many different diseases that can be encountered in newborns. Therefore, it becomes necessary to make an efficient classification study using a limited data set. Accordingly, in this study, data augmentation methods which brightness enhancement, contrast changing, and sharpening were used to obtain the amount of data required by deep learning. The data augmentation process and occurred training set size given in Figure 3.

220 of the 680 images in the original data set were reserved for testing the model and the remaining 460 images were increased to 1380 images with 3 different data augmentation technique. The following data-augmentation techniques were used in this study:

Brightness enhancement;

$B(i, j)=f(i, j)+(a)$               (1)  

Contrast changing;

$C(i, j)=f(i, j) *(b)$                   (2)

Sharpening;

$S(i, j)=f(i, j)+\tau * H(i, j)$               (3)

Here, $f(i, j)$ represents the actual pixel values, a represents the amount of light, and b represents the contrast coefficient. In this study, a and b values were chosen as 0.3 and 0.5, respectively, experimentally. Sharpness is expressed as the contrast between different colors. Conversely, sharpening images means increasing the contrast of the edges where different colors intersect. The term $\tau$ represents a tuning parameter that typically ranges from 0 to 2; high values of $\tau$ may create undesirable effects in the output image. Additionally, $H(i, j)$ used to implement a high-pass filter. Augmented images are shown in Figure 4.

With data augmentation, the total size of the dataset used in the application has reached 2060.

After removing 220 images from 11 infants allocated for testing, the remaining images were used to train the model.

2.4 CNN model

CNN is a deep learning model that produces high-performance results in mostly image-based implementations such as image classification, object recognition, noise cancellation, image segmentation etc. CNNs contain 2 basis layers which convolution and a fully connected network. There are two processes in the convolution layer: convolution and pooling. Feature extraction from the image with convolution process, feature selection is done by pooling process. The fully connected layer consists of two parts, flattening and network layers. The feature matrix obtained in the flattening layer is transformed into a feature vector.

3.png

Figure 5. CNN model

Table 2. Description of CNN architecture

4.png

Figure 6. Representation of the proposed system in the form of a flow chart

The fully connected network layer consists of two parts, flattening and network layers. The feature matrix obtained in the flattening layer is transformed into a feature vector. The classification of this feature vector obtained from the network layers is performed by softmax classifier. Softmax operate generates probability-based loss value by exploitation score values created by artificial neural network. As seen in Figure 5, proposed CNN architecture included three convolutional, three pooling, and three dense layers. We resized the dimensions of the input images from 512 x 512 to 64 x 64. The first convolutional layer consisted of 32 different 3 x 3 dimensional layers (activation function = relu), whereas the second and third convolutional layer contained 64 different 3 x 3 dimensional layers (activation function = relu). The dimensions of the three used pooling layers were 2 x 2 (pooling = max pooling). The first two dense layers consisted of 128, and 64 neurons, respectively (activation function = relu), and the third dense layer, called the output layer, included one neuron (activation function = sigmoid). The output values varied from 0 to 1.

The description of used CNN model is given in Table 2. The block representation of the system, which includes the steps of obtaining thermograms, data augmentation, training of the CNN algorithm, and evaluation of the results is in Figure 6.

In this study, CNN model and data enhancement methods were used to determine respiratory system anomalies in neonates. In the classification, neonates with various pulmonary abnormalities are in the 1st class, and neonates with other diseases (cardiovascular diseases, abdominal diseases) constitute the 2nd class. In the first part of the study, the classification was made with 20 images from each neonate and 680 images of 34 infants with 16 pulmonary diseases and 18 other diseases. In the second part, all the images of the infants and the images obtained with different augmentation techniques were used and the classification was made with 2060 images. The confusion matrices of the classification is given in Table 4 and Table 5. Table 6 shows the results of Accuracy, sensitivity, and specificity.

The evaluation of original limited dataset results obtained as 84% specificity, 85% sensitivity, 85% accuracy. When the number of the thermal images was increased from 680 to 2060, all evaluation metrics values of augmented dataset results were increased. For example, the test accuracy metric rised from 85% to 91% and sensitivity calculated as 100%.

The results showed that it is possible to get better results with data increase and the model used is suitable for the detection of neonatal diseases from thermal images. In order to propose a pre-diagnosis method that can detect respiratory system anomalies in newborns, a dataset containing different disease classes has been defined. For this reason, it was placed as the second / opposite class in the cardiovascular and abdominal diseases dataset, which is predicted to show thermal asymmetry between the neck and groin. This study may be a motivation for the classification tasks of cardiovascular diseases or abdominal anomalies in future studies. Promising results were obtained in reducing neonatal mortality by thermal imaging. Using more thermal data or increasing the number of data with different image enhancement techniques may have better results.

Table 4. Confusion matrix obtained by original dataset

Table 5. Confusion matrix obtained by augmented dataset

Table 6. Accuracy, sensitivity and specificity results