Detection of Diabetic Retinopathy using Convolutional Neural Networks for Feature Extraction and Classification (DRFEC)

Introduction

Diabetic Retinopathy (DR) is an eye disorder which is a consequence of Diabetes Mellitus (DM). DR causes inflammation and breach of retinal blood vessels for the formation of various irregular retinal lesions. It is frequently observed in patients having diabetes from a longer duration of time such as 10–15 years [47]. Statistics have stated that 80% of diabetic patients, suffering from protracted diabetes suffer from diverse phases of DR [28]. India has approximately 73 million individuals suffering from diabetes [77]. The early consultation of the patient with diabetes, with experts for DR assessment and evaluation has become necessary. The process of early diagnosis can reduce the development of DR and susceptibility to severe blindness [22]. If the disease remains undiagnosed, DR progresses in forms of various stages through formation of lesions such as Microaneurysms (MAs), Exudates (EXs), Hemorrhages (HEs), Cotton Wool Spots (CWSs), abnormal structure of the Optic Disc (OD), abnormal Foveal Avascular Zone (FAZ), Neovascularizations, Intra Retinal Microvascular Abnormalities (IRMAs), and many more [24–26, 52, 73, 76, 84]. The presence of these lesions can lead to vision problem and as the disease progresses to severe stages, it can lead to complete blindness [22, 73].

The experts of the domain, called as Ophthalmologist examines the human retina using a high-resolution digitized fundus camera for capturing fundus image. The fundus perceives various DR retinal lesions for image annotation and follow-up necessary treatments. The process of physical diagnosis is painstaking, and the dearth of unavailability of proper means for treatment makes early detection of DR, a challenging task. Thus, such a medical disorder necessitates advanced means for proper diagnosis and treatment. To overcome the challenges of labor-intensive DR detection, researchers have proposed intelligent expert systems, using Deep Learning (DL) for analysis and in-depth study of DR features from fundus images. Intelligent systems [4] are effective with respect to time, feature extraction, error recognition, and early diagnosis and treatment compared to traditional methods. These smart systems take fundus images as input, which are enhanced and analyzed for extraction of significant features, for classification of DR as moderate DR, moderate NPDR, mild DR, severe NPDR, mild Non-Proliferative DR (NPDR), early Proliferative DR (PDR), PDR, etc. [29].

Thus, intelligent systems using DL are proposed as an early and potentially scalable alternative for DR detection. Conventional ML models and data analysis approaches are shallow in nature, and have shown poor performance in learning and training complex non-linear features from larger datasets and hence, are unable to exhibit better analysis and interpretation [41]. Besides, DL based CNN models [7] have overshadowed ML models in performance, through inbuilt preprocessing, convolutional operations, better learning and generalization with deeper networks [88], less overfitting [88], data imbalance mitigation [88], optimization etc. Therefore, various state-of-the-art ML-based models [12, 49, 50] called DL models are proposed for deep feature extraction and image [6, 7] classification tasks. The inherent caliber of such models due to their hierarchical structures, enhances the learning of the model, to achieve state-of-the-art performance. In diagnosis of DR, different DL architectures proposed earlier, have exhibited different feature extraction and classification performances. However, in a single experimental set up, the effectiveness of assessing a comprehensive DL model is not exhaustive, and is unreliable. Therefore, this paper proposes a DL model entitled Diabetic Retinopathy Feature Extraction and Classification (DRFEC), an intelligent expert system using DL architectures, for an exhaustive comprehensive evaluation for detection of DR. The proposed approach is an intensive meta-analysis of 26 pretrained DL networks for identification of the optimal architecture suitable for a larger dataset such as Kaggle EyePACS DR detection with significant skewness, and thereby identify architectural patterns and corresponding loopholes in performance, to ease the process of traditional methods of detection. The model identifies suitable architecture-optimized goals to mitigate overfitting and poor generalization. The main contributions of the manuscript have been enlisted below:

1. The proposed model performs a comprehensive comparative assessment and evaluation to determine the behaviour of 26 DL models on the DR dataset

2. To conduct meta-analysis of traditional as well as state-of-the-art DL architectures on the highly skewed Kaggle DR detection dataset, with minimal resources and identify the best amongst them for future application and analysis.

3. The proposed model identifies high bias and high variance during DR image classification to identify the best DL classifier

This manuscript is sectionalized into various sections. Section 2 is an analysis of various related works and models proposed earlier, for identification of loopholes in early DR detection. Section 3 gives a detailed illustration of the proposed methodology, and establishes the significance of DL models for better feature extraction and classification. Section 4 illustrates the significance of the 26 DL models, based on the analysis and comparison of their performances, for determination of an optimal DL architecture(s) for DR detection. Section 5 concludes on a note identifying the suitable DL architecture(s) for DR detection.

Literature review

DR is a chronic health disease which requires early detection and treatment [48]. It is important to identify DR using an intelligent system for faster prediction since manual examination and detection of the disease are unreliable and highly prone to error. Therefore, various researchers and medical experts have adopted and approached for advanced feature extraction and image classification, for early DR detection. Thus, various works have been proposed in this respect using ML and DL techniques, and in which DL techniques have completely outperformed ML based models on grounds of processing large dataset, efficient computation, overfitting, generalization and better prediction. This section introduces some of the recognized works on image domain using DL techniques especially upon fundus images for early DR detection [32].

Wang et al. [85] have proposed a boosted CNN architecture using EfficientNet B3 for extraction of images features using integrated attention and feature fusion-based mechanisms, random center cropping upon Rectified Patch Camelyon (RPCam) datasets to predict and classify lymph node metastasis in breast cancer images. The model is compared with baseline models such as EfficientNet B3, ResNet50 and DenseNet121. Gurcan et al. [31] have proposed an automated DR classification system based on preprocessing, feature extraction, and classification steps using deep CNN and ML methods. The model has extracted features from a pre-trained InceptionV3 model using transfer learning. The model has compared various ML methods namely Bagged Decision Trees, XGBoost, Random Forest, Extra Trees, Support Vector Machines, Logistic Regression, and multilayer perceptron in which XGBoost performs better. The model has used Grid search and calibration for analysis in addition to comprehensive preprocessing and fine-tuning. The model has extracted generic descriptors from one of the initial layers of InceptionV3, without layer-wise tuning and has achieved competitive classification accuracy with ML methods. It is observed that the descriptors are obtained and extracted from the initial layers of CNN which have comparatively low-level information than the higher layers of a CNN. This may be inefficient in the process of DR detection and hinder the detection of actual features thereby affecting the robustness of the model. Sarki et al. [69] have proposed a methodical study which identifies and signifies various preprocessing operations such as Contrast-Limited Adaptive Histogram Equalization (CLAHE), morphological operations, image segmentation for blood vessel segmentation etc. for Diabetic Eye Disease (DED) detection. The model has used a mini-batch size of 32, cross-entropy loss function, Adam and RMSprop upon the datasets namely DRISHTI-GS, Messidor, Retinal Dataset and Messidor-2. The proposed automated classification framework has used image enhancement, image augmentation and segmentation, and classification using very small datasets having poor category DR samples. Mayyaa et al. [56] have proposed a methodical review to examine the diagnostic use of automated microaneurysm detection for DR, and has identified various strengths and weaknesses. The methodologies appraised in this article confronts challenges that needs to be addressed in designing an effective algorithm for early diagnosis. Hattiya et al. [33] have appraised AlexNet DL mechanism as an ideal CNN architecture for DR detection and compared diverse CNN architectures namely MobileNet, DenseNet201, InceptionV3, ResNet50, NASNetMobile and MNASNet, using 23,513 retina images.

Kamal et al. [44] have anticipated a transfer learning model for DR detection which mitigates imbalanced dataset. The model has fine-tuned Inception V3, VGG19, ResNet50, MobileNet, ResNet50V2, DenseNet121, MobileNetV2 and NASNetMobile using COVID-19 [5, 8] and Pneumonia datasets. Islama et al. [40] have reported a comprehensive systematic review of the performance of DL algorithms gauged precisely for computerized DR detection in fundus images. Lee et al. [53] have proposed a transfer learning NASNet-A (large) to extract bottleneck features from spectral-domain optical coherence tomography images and an ensemble training for prediction. Bodapati et al. [10] have anticipated a DR model using transfer learning, and feature extraction using VGG-16, Inception ResNetV2, Xception and NASNet to boost feature exemplification which are compared with handcrafted features, for DR detection. The model has compared feature fusion and pooling approaches and have identified averaging pooling simple fusion approach upon Deep Neural Networks (DNN) as effective in performance. It has extracted features of DR images collected from Kaggle APTOS 2019 contest dataset, using VGG16 and Xception. The authors blended these features to get the final feature representations, which are used to train DNN. Shah et al. [71] have proposed a DCNN model to distinguish referable DR using 1533 macula centered fundus images and MESSIDOR dataset. The model compares the evaluation process of ground truth data and machine learned data. The algorithm is a composition of three-version based training modules using 80,000, 96,500, and 112,489 unidentified fundus images, respectively. Pour et al. [62] have proposed EfficientNet based feature extraction and classification model using EfficientNet B5 for DR detection using MESSIDOR, MESSIDOR-2 and IDRiD datasets. The images are preprocessed using CLAHE, and the model has achieved an AUC of 0.945 on MESSIDOR, and AUC 0.932 on IDRiD.

Chetoui et al. [14] have proposed an EfficientNet based feature extraction and classification model using EfficientNet B7 and Global Average Pooling, for DR detection. The model has used Kaggle EyePACs and APTOS 2019 datasets, and have extracted features such as EXs, HEs and MAs using Gradient-weighted Class Activation Mapping (Grad-CAM). Tymchenko et al. [83] have proposed a DCNN encoder-based feature extraction for DR detection using pre-trained EfficientNet-B5, EfficientNet-B4, SE-ResNeXt50 and ensemble of 20 models. Chaturvedi et al. [13] have proposed a modified DenseNet121 network on APTOS 2019 dataset and uses 3662 fundus images, for DR detection. Samanta et al. [66] have anticipated a fine-tuned DenseNet121 model which uses 3050 images for training, and is tested using VGG16, InceptionV1, InceptionV3, InceptionV2, Xception, AlexNet, ResNet-50 and DenseNet for DR detection. Sarki et al. [68] have proposed an inclusive assessment of 13 pretrained CNNs, using MESSIDOR and Kaggle dataset, for detection of DR. Ji et al. [42] have proposed an augmented DNN model using Inception V3, DenseNet121 and ResNet50 for transfer learning and improve computational proficiency and classification, for DR detection. The model has proposed various subnetworks for each of the DNN and analyzed their performance on large OCT image datasets of 83,484 images, for detection of DR lesions. The proposed model achieves accuracy and stability for InceptionV3 and ResNet50, however in case of DenseNet121, no significant improvement could be seen due to its dense connection architecture.

Michele et al. [57] have proposed a fine-tuned pretrained feature extraction and classification model using MobileNetV2, dropout and a linear SVM classifier, for palmprint recognition. The model has used a PolyU palmprint dataset of 6000 images. Hui et al. [38] have anticipated a modified extreme inception-based U-Net segmentation module, to extract effective features using multitask learning and distance representation, from remote sensing images. The model has accomplished better performance with multitasking, on the datasets. Jiang et al. [43] have proposed an interpretable ensemble ResNets based feature mining and classification for DR detection using a DL model which constitutes Inception V3, ResNet152 and InceptionResNetV2, and Adaboost algorithm. The model has employed 28,244 training images, and achieved better performance using the integrated DL model than the individual models. Orlando et al. [59] have proposed an ensemble LeNet-CNN approach for the detection of MAs, HEs, red lesions for DR detection, using hand-crafted features, CNN features and a combination of both. The proposed model has achieved better results with the combination of CNN and handcrafted features. Suriyal et al. [78] have proposed a real-time DR model using MobileNet, on internet-deprived portable devices, for DR detection. The model has used 16,798 images.

Huang et al. [37] have proposed a compact novel network architecture called CondenseNet which combines dense connectivity with novel learned group convolutions, using CIFAR-10, ImageNet and CIFAR-100 datasets, for image classification. The novel CNN architecture has achieved cost-efficient performance in comparison to MobileNets, DenseNet-190 and ShuffleNets. Pogorelov et al. [61] have compared global features extracted from gastrointestinal tract images using transfer learning models such as ResNet50 and Inception V3, and proposed a modified CNN. The model has compared the predictions of ML and DL classifiers, and has achieved better performance using ResNet50 than Inception V3 for feature extraction. Gulshan et al. [30] have proposed a DL InceptionV3 for detection of DR and diabetic macular edema in fundus images. The proposed methodology has deployed the EyePACS-1 dataset and MESSIDOR-2 dataset. The model has used an ensemble of ten networks and their linear average is used for prediction, upon 128,175 images. Nneji et al. [58] have proposed a two-channel preprocessing weighted fusion deep learning network on fundus images which uses CLAHE fundus images and the contrast-enhanced canny edge detection (CECED) fundus images, from 2000 Kaggle images and MESSIDOR dataset for the detection of DR. The model uses VGG-16 and a modified Inception-V3 for feature extraction and merges the channel output using a weighted fusion approach. The model performs a comprehensive analysis on six different baseline models, reduces the kernel of the Inception module B to (4 × 4), and improperly manipulates the size of the layers and the symmetricity of the image. Dong et al. [23] have proposed a DL model using InceptionV3 and VGG-16 for the detection of DR which is compared with mainstream models such as GoogLeNet, AlexNet and ResNet50 using 2693 wide-field optical coherence tomography augmented images. The real-time dataset employed is expensive and time-consuming. The authors have claimed that wide-field optical coherence tomography images are better than optical coherence tomography images due to incorporation of a better field-of-view (FOV). The model is built upon conventional DL models with a very limited dataset, which is unreliable.

Padmanayana and Anoop [60] have proposed a CNN model for the detection of DR through comparison of the performance of various optimizers such as Adagrad, RMSProp with momentum and Adam, using the APTOS 2019 Kaggle dataset for training and 1000 images collected from a private institute for testing. It uses weighted CLAHE, Gaussian blur and Ben Grahams fraction maxpooling for preprocessing. Sivapriya et al. [75] have proposed a Recurrent Neural Network (RNN) to identify hard EXs for the detection of DR. The model uses limited data of 400 images from the MESSIDOR dataset and performs various preprocessing operations on them without any significant changes in the architecture of the RNN. It has compared the performance of the approach with some of the earlier works and reflects a conventional and unreliable behaviour. Bora et al. [11] have proposed an automated risk analysis system for the detection of development of DR using fundus images, in patients having diabetes with no DR. It reports a risk stratification tool and learns the various associated risk factors causing the development of DR from the stage of no DR. It uses a large retrospective longitudinal dataset for the analysis- 575,431 eyes/single images of the development set has 28,899 known outcomes, 546,532 to augment the training process via multitask learning, 3678 images in the internal validation set and 2345 images in the external validation set. Deepa et al. [18] have proposed a multistage ensemble DCNN using InceptionV3 and Xception which concatenates multi-stage patch based and image-based probability vectors for deep feature extraction and SVM based ensemble classification, for the detection of DR. It uses voting and stacking for probability vector concatenation which is classified using an Artificial Neural Network (ANN). An ensemble of SVM classifier is used for further prediction on the ensembled classification output. It is observed that the proposed architecture incorporates a single-tier ensemble for feature extraction and a double-tier ensemble using different classifiers for classification which makes the overall architecture computationally expensive and time consuming.

Saeed et al. [65] have developed a two-stage Principal Component Analysis based transfer learning algorithm and initializes the pretrained model with extracted (64 × 64) ROIs of MAs, EXs and normal features of the fundus images of Kaggle’s EyePACs and MESSIDOR datasets, for detection of DR. It introduces a fixed-dimension based adaptive maxpooling to predict the label of the ROI, and compares the performance of DL models namely VGG-19, ResNet152 and Dual Path Network 107(DPN107) in which the fully connected layers are replaced with PCA for unsupervised feature discrimination. It uses Decision Tree (DT), RF and Gradient Boosting (GB) for classification and compares the overall performance of ResNet152 with all the other models. Tsai et al. [82] have proposed a DL model using Inception V3, ResNet101 and DenseNet121 on global Kaggle’s EyePACs dataset and local dataset from Taipei City Hospital (TCH), for DR detection and sets a higher overestimation rate on local dataset than global dataset due to differences in regional and ethnic factors. The proposed model uses a significant dataset for the DL models employed. However, it is biased and takes ethnic factors into account for the detection of DR and fails to learn region specific features. On comparison, DenseNet121 has performed better than InceptionV3 and ResNet101.Atwany et al. [3] have performed a literature review and analyses the significance of supervised, semi-supervised and vision transformer methodologies and learning paradigms and their significance in application to DL models for DR detection. It uses Kaggle’s EyePACs dataset of 88,702 images, DDR dataset of 13 K images and 30,244 fundus images from Beijing Tongren Eye Centre for the study and have concluded that supervised learning is not a good paradigm for noisy data. Instead, the study identifies Semi-Supervised Learning (SSL) effective but unexplainable, and less prone to inductive bias to be able to handle variance due to cross domain shift. However, it identifies SSL as non-robust for small-scale datasets. The study also reviews various amalgamation techniques for synthesis of dataset such as Generative Adversarial Network (GAN) and Variational Autoencoders (VAE), and identifies less complex DL attention models such as Vision Transformers (ViT).

Das, S. et al. [16] have proposed a two-way CNN classifier based on Squeeze-and-Excitation memory module and CNN, using DIARETDB1 and a local dataset, for DR detection. The model introduces the significance of data augmentation in better model performance. Lim et al. [55] have performed a literature review on gradient-based interpretability methods in DL models such as saliency map, integrated gradient, layer-wise relevance propagation, occlusion testing, sensitivity analysis, class activation map, gradient-weighted class activation map and layer-wise relevance propagation, in the detection of DR. It identifies the drawbacks of these interpretability methods in detection of correct lesions for a given class and the lack of reliable ground truth. AbdelMaksoud et al. [1] have proposed an integrated CNN model called E-DenseNetBC-121 which is a combination of EyeNet [63] and DenseNet [39] and uses datasets such as EyePACS, IDRiD, MESSIDOR and APTOS 2019 for the detection of DR. Li et al. [54] have use a DL algorithm-based software for grading 1674 images for the detection of DR. However, it fails to detect DME which is responsible for DR, for early DR detection. It uses 1, 40,000 fundus images from publicly available EyePACs dataset and 1200 fundus images from Shanghai General Hospital. Deepa et al. [19] have proposed a comprehensive two-phase feature extraction algorithm which uses Xception and textural and transform based techniques to detect MAs for DR detection. It uses Siamese Network based CNN to perform hierarchical clustering along with Xception-based cluster selection, and a fine-tuned Xception model for extraction of patch-wise local and global features, respectively. The features are extracted from 2290 images and are classified using a Radial Basis Function (RBF) kernel based SVM which is compared with other classifiers such as RF, Adaboost and Multilayer Perceptron (MLP).

Sau and Bansal [70] have proposed a Fitness based Newly Updated Grasshopper Optimization Algorithm (FNU-GOA) for the optimization of a DL model and to optimize the threshold value in active contour method for the segmentation of blood vessels, MAs, EXs and HEs for DR detection. It is compared with several other optimization algorithms such as Particle Swarm Optimization (PSO), Grey wolf optimization algorithm (GWO), Whale optimization algorithm (WOA) and Grasshopper Optimization Algorithm (GOA), and ML classifiers such as Neural Network (NN), RNN, Long Short Term Memory (LSTM) and Deep NN. The meta-heuristic optimized algorithm achieves a poor specificity and fails to classify negative samples correctly. Shaik and Cherukuri [72] have proposed a multi-stage end-to-end DNN pipeline called Hinge Attention Network (HA-Net) using gated attention VGG-16 discriminator and reconstruction autoencoder to produce attention maps, for learning latent representations in 3662 images of Kaggle’s APTOS 2019 dataset, and ISBI-2018 IDRiD² dataset for the detection of DR using various optimizers. It achieves a poor accuracy on limited-graded dataset. The model employs various attention descriptors with minimum samples and suffers from the curse of dimensionality. It fails to generate a decision boundary and learn inter-spatial and inter-channel correlation present in latent features. It also fails to reduce latent spatial representations learned from baseline models such as VGG-16, VGG-19, ResNet50, ResNet50V2, Xception, MobileNet, Inception V3 and InceptionResNetV2, and from overlapping data.

On the basis of the literature review, it is clear that conventional models are outperformed by state-of-the-art DL models. Many conventional models are shallow in nature, in contrast to DL models where DL models have shown convincing results. Based on this inspiration, the proposed DRFEC aims to perform an exhaustive analysis on DR image classification to define the characteristics, behaviour and pattern of DR data, and to identify, interpret and implement a convincing model for the problem. The use of conventional modes with a very small and limited dataset is the main drawback in DR detection. In addition to this, the use of the training set of Kaggle’s EyePACS dataset is very limited due to data imbalance and lack of features. Besides, processing a larger dataset through increase in the number of samples and data augmentation causes data explosion which is a huge challenge with constraints in adequate resources. Various research works have considered only a few images which ranges between a few thousand images. The existing models have shown resemblance in their performances through the use of a similar number of images with no significant improvement in the pattern of the collected data. Moreover, these models also cannot generalize real-time datasets because of biasness towards regional and ethnic factors, which remains an unsolved problem. They have also shown high variance thereby reflecting the poor learning process of the methodologies for the detection. The proposed model thus aims to train a DL model to gain insights on the problem, and identify and differentiate parameters responsible for various performance with respect to (w.r.t.) the optimal model. It aims to solve critical problems through identification and analysis of DR data from the inception using an imbalanced dataset and gradually optimizing the proposed baseline framework through hyperparameter tuning, model training and evaluation [64].

The proposed DRFEC

The manual examination of DR is a feasible but lethargic process, and hence not recommendable for early DR detection. Therefore, it is essential to examine DR using a proficient system which employs ML techniques such as DL for better detection. Various explorative works have been performed earlier for DR detection, using a variety of datasets and especially using the Kaggle’s EyePACS dataset. The proposed DRFEC accomplishes inbuilt preprocessing, deep feature extraction and image classification using a comprehensive DL model which constitutes VGG-19, VGG-16 [74], Xception [15], InceptionV3 [79], MobileNet [35], MobileNetV2 [67], EfficientNet B0-B7 [81], DenseNet121, DenseNet169, DenseNet201 [36], ResNet50, ResNet50V2, ResNet101, ResNet101V2, ResNet152, ResNet152V2 [34], NASNetLarge [89], NASNetMobile [89] and InceptionResNetV2 [80]. The model uses popularly known ImageNet [51] pre-trained DL models for training and evaluation of Kaggle DR Detection dataset obtained from EyePACS [17, 21].

DL models have the proficiency to extract new and deep features, from previously learned features using representation learning. Thus, the proposed model emphases on finding scope for various improvements upon the data and the model, and thereafter determines the most suitable network amongst them for DR image classification. Figure 1 depicts a pictorial demonstration of DRFEC deploying DL models. It is implemented using a 64-bit operating system, ×64-based processor, Windows 10 Pro, DL package Keras from TensorFlow, Python 3.8, TensorFlow version 2.4, 64GB RAM and Intel(R) Xeon(R) W-2155 CPU @ 3.30GHz 3.31 GHz. The outline of the different steps of DRFEC is given in Fig. 1 below.

The proposed DRFEC processes the input data acquired from Kaggle repository and downsamples it accordingly based on different architectures employed in the model. Then the employed CNN is used to perform feature extraction and multi-class classification of DR fundus images. Flowchart 1 is an illustration of the working principle of DRFEC where each time a CNN is chosen to train the model. Every CNN employed is individually used to compute the training and validation/test accuracy values to check for model overfitting and generalization. Based on the performances and inferences drawn from every single DL CNN model, the best DL architecture for deep DR feature extraction and image classification is extracted.

Results

The proposed model is a composition of DL CNN models such as VGG-16, VGG-19, Xception, InceptionV3, InceptionResNetV2, MobileNet, MobileNetV2, EfficientNet B0-B7, DenseNet121, DenseNet169, DenseNet201, ResNet50,, ResNet50V2, ResNet101, ResNet101V2, ResNet152, ResNet152V2, NASNetLarge and NASNetMobile which are trained on a training dataset of 27,446 fundus images and tested on a test dataset of 7680 fundus images, where both the training and test dataset contains images belonging to 5 classes of DR. The DL models are individually trained and tested upon the dataset of interest to evaluate their performance. The proposed model illustrates and demonstrates the behaviour of these models in a chronological fashion, and has achieved different training and validation values. The DL models are trained for a target size of (224 × 224), using inbuilt preprocessing of the CNNs, a batch size of 32, and using the fully connected classification layers for classification accompanied with softmax activation function. Besides, the DL models uses the state-of-the-art Adam optimizer, a learning rate of 0.0001 and categorical cross entropy loss function, for training for 50 epochs.

Time complexity of DRFEC

The proposed model is a comprehensive evaluation of 26 CNN Deep Learning networks. The ML models especially Neural Networks (NNs) when applied on image data requires processing of thousands of features from thousands of pixels in a given image. Moreover, NNs are fully connected since inception and hence are highly parameterized. When they perform image learning, the number of parameters increases exponentially, as we will see in this section while analysing the working of a fully connected network in one of the most popular DL models. These networks cause overfitting and requires a large number of samples for proper training, in addition to larger memory and greater computational power for prediction. In image data, there are two major characteristics namely – feature localization or feature correlation and feature independence of location. CNNs thus, have the ability to solve the problem associated with too-many-parameters, with shared parameters (feature independence of location) of locally connected networks (feature localization) called the Convolution Net (ConvNet), which reduces the total number of parameters. These ConvNets have undergone evolution in its depth, architecture and search space. They are incorporated factors such as activation function, dropout, augmentation, filter size for parameter reduction and non-linear transformation, doubling of channels to recover lost information, to excel in performance. However, issues such as vanishing gradient deaccelerates their performance and makes them hard to optimize. They have introduced the concepts of variable filter sizes, residual learning, skip connections, identity mappings, depthwise separable convolutions and reinforcement learning to overcome problems associated with the depth of the network, and in the search space to identify the best combination of parameters.

In a CNN, the number of features is at most a constant time the number of input pixels say p, where the constant is <1. Therefore, the convolution of a fixed size filter across an image with p number of pixels takes O(p) time because each output is a sum of the product between k pixels in the image and k weights in the filter, where k does not vary with p. Similar is the case with maxpooling and average pooling operations of a CNN which does not take more than liner time in the given input size. Therefore, the overall runtime is linear [86]. With increasing depth, these CNNs transform to a Deep Neural Network (DNN). With the increase in the number of fully connected layers DNNs processes a feed-forward pass algorithm and a backpropagation algorithm [46]. Theoretically and mathematically, the training of a neural network-based model is implemented using matrices, and the time complexity of matrix multiplication (M_ij∗M_jk) is O (i∗j∗k). For e.g., to compute the time complexity of the forward pass algorithm for VGG-16 with 16 layers and 5 convolutional blocks, let i denote the number of nodes of the input convolutional layer, j denotes the number of nodes in the second convolutional layer, k denotes the number of nodes in the third convolutional layer, l denotes the number of nodes in the MaxPooling layer, m denotes the number of nodes in the dense layer 1, n denotes the number of nodes in dense layer 2 and o denotes the number of nodes in the output layer. Since there are 7 layers, 6 matrices are required to represent weights between the layers.

Let W_ji, W_kj, W_lk, W_ml, W_nm, W_on be the matrices where W_ji is a matrix with j rows and i columns and contains the weights from layer i to layer j, W_kj is a matrix with k rows and j columns and contains the weights from layer j to layer k. Similarly, W_lk is a matrix with l rows and k columns and contains the weights from layer k to layer l, W_ml is a matrix with m rows and l columns and contains the weights from layer l to layer m. Again, W_nm is a matrix with n rows and m columns and contains the weights from layer m to layer n, and W_on is a matrix with o rows and n columns and contains the weights from layer n to layer o. For t training examples (here t = 27,446), to propagate from layer i to j, the following matrix multiplication is performed S_jt = W_ji∗Z_it and has O (j∗i∗t) time complexity. The activation function Z_jt = f (S_jt) has O (j∗t) time complexity because it is an element-wise linear operation. In total, from i → j there is a time complexity of

Similarly, from j → k there is a time complexity of O (k∗j∗t), from k → l there is a time complexity of O (l∗k∗t), from l → m there is a time complexity of O (m∗l∗t), from m → n there is a time complexity of O (n∗m∗t), and from n → o there is a time complexity of O (o∗n∗t). In total, the time complexity for feedforward propagation will be

ii

Thus, the time complexity of the DNN during forward propagation is.

This is repeated for all the convolution operations in a CNN- here VGG-16 with 5 convolutional blocks, and thus the total time-complexity for forward propagation in VGG-16 is.

which is linear. The similar concept is followed across all the other DL models depending upon the number of layers i.e., the depth of the network. The structures such as inception modules, skip connections, dense blocks in InceptionV3, ResNets and DenseNets, respectively helps in minimization of parameters and optimizes the depth of the network thereby optimizing the computational time. Thus, all the DL models have a time complexity which is linear and is only affected by the number of parameters in the network. In case, where the input size and the size of the kernel is same, the time complexity is O (p) [87].

The backpropagation algorithm proceeds as follows: Initially, the error E_O, from the output layer o to n is computed and the matrix containing the error for the nodes at layer o is represented as -.

E_ot has o rows and t columns meaning each column is an error for the training example t, and ⨂ is the element-wise operation. The small change ‘delta, D’ in the weights called ‘delta weights’ between the layer o and n is computed as -.

The weights are adjusted as W_on = W_on – D_on, and for o to n the time complexity is.

From layer n to m, E_nt = f ` (S_nt) ⨂ (W_no ∗ E_ot), D_nm = E_nt * Z_tm and W_nm = W_nm – D_nm, where W_no is the transpose of W_on. The time complexity for n to m is O (nt + not + ntm + nm) = O (n*t (o + m)). Similarly, for m to l, l to k, k to j and j to i, the time complexities are O(m*t(n + l)), O(l*t(m + k)), O(k*t(l + j)) and O(j*t(k + i)), respectively. Therefore, the total complexity is.

which is the same as the feed-forward pass algorithm.

Therefore, the time complexity for one epoch will be.

In case for e = 50 number of epochs, the DNN will be showing a time complexity of -.

which is again linear. Here, the time complexity of the proposed model is based on the number of parameters in the DL networks. It is directly proportional to the number of parameters i.e. it increases and decreases with the increase and decrease in the total number of parameters in the network, respectively. However, it is observed that models with higher complexity have performed comparatively better than models with lower complexity such as NasNetLarge [2]. A similar thing is observed in the proposed model where larger architectures such as DensNet169, NasNetLarge, and EfficientNetB4 have shown significant performances despite huge architectural, space and computational complexity. Each of these models have taken a maximum of 72 hrs of execution on 50 epochs using 35,126 images in the available experimental environment.

The complexity of DL models is computed using multiply-accumulates (MACC) operation and Floating-Point Operations Per Second (FLOPs) [9] with a batch size of 32. The input and output to convolutional layers are not vectors but three-dimensional feature maps of size H × W × C where H is the height of the feature map, W the width, and C the number of channels at each location. The number of MACCs for a convolution layer with kernel size K is:

Each pixel in the output feature map is of size H_out × W_out and performs a dot product of the weights, and a K × K window of input values across all input channels, C_in. The layer has C_out different convolution kernels, therefore it is repeated C_out times to create all the output channels. For an e.g., a 3 × 3 convolution with 512 filters, on a 14 × 14 input feature map with 7 channels, the MACC is 3 × 3 × 7 × 14 × 14 × 512 = 6,322,176. The MACC-FLOPs obtained individually for each of the DL model is depicted in Table 3. Figures 63 and 64 illustrates the graphical presentation of the time complexity of 26 DL models in terms of MACCs and FLOPs, respectively.

Observations

The proposed work aims to show a comparative analysis of state-of-the-art DL models for early detection of DR using a fundus image dataset. On comparison, it is observed that the DL architectures belonging to different ensembles/family have shown different behaviours using the image data. Tables 4 and 5 highlights the comparative performance of the DL models in terms of training and validation accuracy, training and validation loss, and loss difference. It is observed that the validation loss of the architectures implemented in the proposed approach is higher than the corresponding training loss. However, the training accuracy is significantly higher than the validation accuracy. This implies that the architectures inclines towards majority classification due to data imbalance which causes overfitting and poor generalization. Additionally, significant observations are analysed to identify the best DL architecture for early detection of DR which are discussed below:

• i.

  In Table 4, it is shown that VGG-16 has the lowest training accuracy amongst all the DL architectures, due to its simple-deep architecture. However, VGG-19 has comparatively a better training and testing accuracy than VGG-16, due to increase in the number of layers. This implies that VGG can perform better with increasing depth. This difference in the behaviour of both VGG-16 and VGG-19 infers that the depth of the network has a crucial role to play in achieving better accuracy and minimum loss. However, with the increasing depth and number of parameters in the networks, performance issues such as vanishing gradient problem arises, which minimizes the effect of the loss function on the activation function. This causes VGG to perform poorly than the family of ResNets. The ResNet family has a better performance than VGG due to the initialization of skip connections in the network which not only retains the depth of the network but also eliminates the vanishing gradient problem.

• ii.

  VGG-16 has achieved the lowest validation accuracy in comparison to all the other DL models. It imposes implicit regularization through greater depths and pre-initialization of network weights to avoid instability of gradients. It uses small convolutional filters for additional non-linearity in structure and captures spatial context based on non-trivial receptive fields. Thus, it is inferred that deeper models are beneficial for larger datasets, however VGG-16 has many weight parameters which makes the model heavy and causes larger inference time. This leads to vanishing gradient problem with higher loss and thus higher error with lesser ability to generalize, which ultimately affects validation performance. In addition to this, the model greatly suffers from overfitting as it is a simple yet overly deep network stacking 3 × 3 convolutional layers, and higher training error caused due to loss. VGG-19 is 11.3% better than VGG-16, 5.88% better than NasNetMobile, 1.73% better than ResNet50, and 0.40% better than ResNet50V2 and EfficientNetB7 respectively. Thus, VGGs are simple yet a better learning DL model than various state-of-the-art DL models.

• iii.

  ResNet101 is 15.25% better than VGG-16, 9.83% better than NasNetMobile, 5.68% better than ResNet50, 4.35% better than ResNet50V2, 4.35% better than EfficientNetB7, 3.95% better than VGG-19, 3.74% better than ResNet121, 3.6% better than Inception V3, 1.92% better than ResNet101V2 and 1.69% better than EfficientNetB0. Moreover, it is 1.29% better than ResNet152V2, 0.97% better than EfficientNetB2, 0.69% better than EfficientNetB1, 0.67% better than ResNet152, 0.52% better than ResNet201, 0.16% better than EfficientNetB5 and 0.12% better than MobileNet. ResNets take lesser memory due to skip connections, faster inference time, and identity mapping leading to optimized training error.

• iv.

  Amongst the inception-based modules which have better and optimized receptive fields of (1 × 1) and (3 × 3) instead of (5 × 5) and (7 × 7), to optimize the total number of parameters in the DL models of greater depth. This extreme inception module simplifies the learning task through explicit factoring at a smaller input space and maps all correlations using 1 × 1, 3 × 3 or 5 × 5 convolutions. It permits entire decoupling of spatial correlations and cross-channel correlations in the feature maps forming rich representations with fewer parameters. It has performed better with a training accuracy of 99.46% which is less than DenseNet201 having a training accuracy of 99.58%, by 0.12%. However, it is 15.98% better than VGG-16, 10.56% better than NasNetMobile, 6.41% better than ResNet50, and 5.08% better than ResNet50V2 and EfficientNetB7, respectively. Moreover, it is 4.68% better than VGG-19, 4.47% better than ResNet121, 4.33% better than Inception V3, 2.65% better than ResNet101V2 and 2.42% better than EfficientNetB0. Additionally, it is 2.02% better than ResNet152V2, 1.7% better than EfficientNetB2, 1.42% better than EfficientNetB1, 1.4% better than ResNet152, 1.25% better than ResNet201, 0.89% better than EfficientNetB5, 0.85% better than MobileNet, 0.73% better than ResNet101, 0.63% better than MobileNetV2, and 0.19% better than EfficientNetB6. Xception is 0.10% better in training accuracy but 1% lower in testing accuracy, than InceptionResNetV2. It achieves performance gains due to the increase in the capacity of the network.

• v.

  The family of DenseNets have especially a better performance than VGGs and ResNets because it reuses the features for learning, and eliminates the generation and learning of redundant features. It also has a better performance than various other DL models – it is 16.95% better than VGG-16, 11.53% better than NasNetMobile, 7.38% better than ResNet50, and 6.05% better than ResNet50V2 and EfficientNetB7, respectively. Moreover, it is 5.65% better than VGG-19, 5.44% better than ResNet121, 5.3% better than Inception V3, 3.62% better than ResNet101V2 and 3.39% better than EfficientNetB0. Additionally, it is 2.99% better than ResNet152V2, 2.67% better than EfficientNetB2, 2.39% better than EfficientNetB1, 2.37% better than ResNet152, 2.22% better than ResNet201, 1.86% better than EfficientNetB5, 1.82% better than MobileNet, 1.7% better than ResNet101, 1.6% better than MobileNetV2, 1.16% better than EfficientNetB6, 0.97% better than Xception and 0.88% better than EfficientNetB3.

• vi.

  The family of NasNets is equipped with enriched computational power and automated engineering and solves the problem of finding a better CNN for image classification problem using Reinforcement Learning. It introduces a search space to find the best suited parameters, filter sizes, output channels, strides and total number of layers. On the basis of reinforcement learning it is able to determine the best searched architectures for the dataset. Despite the huge computational power required, NasNetLarge is 17.02% better than VGG-16, 11.60% better than NasNetMobile, 7.45% better than ResNet50, and 6.12% better than ResNet50V2 and EfficientNetB7, respectively. Moreover, it is 5.72% better than VGG-19, 5.51% better than ResNet121, 5.37% better than InceptionV3, 3.69% better than ResNet101V2 and 3.46% better than EfficientNetB0. Additionally, it is 3.06% better than ResNet152V2, 2.74% better than EfficientNetB2, 2.46% better than EfficientNetB1, 2.44% better than ResNet152, 2.29% better than ResNet201, 1.93% better than EfficientNetB5, 1.89% better than MobileNet, 1.77% better than ResNet101, 1.67% better than MobileNetV2, 1.23% better than EfficientNetB6, 1.04% better than Xception, 0.95% better than EfficientNetB3, 0.07% better than DenseNet169 and 0.04% better than InceptionResNetV2.

• vii.

  DenseNet201 has achieved the highest training accuracy of 99.58% amongst all the DL models as well its corresponding performance-resembling architectures. It is observed that DenseNets have shown best results in representation of images, with strong gradient flow, better model complexity, diversified features, flexibility in feature complexity, smooth decision boundaries, and computational and parameter efficiency. However, the excessive connections of DenseNets can decrease its computational and parameter efficiency, and make it more prone to overfitting with highly skewed data. In brief, the depth of DenseNets have a crucial role to play in performance.

• viii.

  EfficientNetB4 is 17.04% better than VGG-16, 11.62% better than NasNetMobile, 7.47% better than ResNet50, and 6.14% better than ResNet50V2 and EfficientNetB7, respectively. Moreover, it is 5.74% better than VGG-19, 5.53% better than ResNet121, 5.39% better than InceptionV3, 3.71% better than ResNet101V2 and 3.48% better than EfficientNetB0. Additionally, it is 3.08% better than ResNet152V2, 2.76% better than EfficientNetB2, 2.48% better than EfficientNetB1, 2.46% better than ResNet152, 2.31% better than ResNet201, 1.95% better than EfficientNetB5, 1.91% better than MobileNet, 1.79% better than ResNet101, 1.69% better than MobileNetV2, 1.25% better than EfficientNetB6, 1.06% better than Xception, 0.97% better than EfficientNetB3, 0.09% better than DenseNet169, 0.06% better than InceptionResNetV2 and 0.02% better than NasNetLarge.

• ix.

  EfficientNetB4 and ResNet50 achieved an optimal and equivalent training accuracy of 99.37%, but a testing accuracy 79.09% and 77.32%, respectively. Besides, ResNet50V2 and EfficientNetB7, and EfficientNetB1 and ResNet152 have the same testing accuracy of 72.97% and 76.6%, respectively. From this behaviour, it can be implied that the ResNets outperforms the EfficientNets in terms of training accuracy but the EfficientNets outperforms the ResNets in terms of validation accuracy. However, comparatively ResNets do not explicitly address generalization and suffers greatly from optimization difficulty. On the other hand, the fastidious nature of EfficientNets and the engineering of hyperparameters w.r.t the choice of ML frameworks for training affects their top-line accuracy. Additionally, they induce greater cost for data movement between layers due to larger number of channels, and have lesser applications because current hardware accelerators are engineered on traditional learning frameworks. From, Fig. 65 it is observed that the training curve increases in ResNet50 but decreases in EfficientNetB7 and further decreases in ResNet101V2. However, it increases in ResNet152 and then in DenseNet201 -the highest, and then decreases in Xception and then in InceptionResNetV2.

• x.

  NASNetLarge and EfficientNetB4 have shown a near equivalent performance in training and validation accuracy. However, they are of different platforms. NASNet is based on reinforcement learning search method to optimize and find the best convolutional layer instead of searching for the entire network, and reduces computational complexity and improves generalization. It generates a search space that decouples complexity and depth of the network. However, EfficientNets such as EfficientNetB4 is a portable structure with depth and resembles MobileNets and NASNetMobile. From Tables 4 and 5, it is observed that these portable, less memory-based structures have shown better performance due to computational and parameter efficiency than all other large DL models such as DenseNets, ResNets, VGGs, etc.

• xi.

  EfficientNetB4 has achieved the highest validation accuracy amongst all the DL networks. Moreover, it has comparatively achieved a better training accuracy, training loss, validation loss and loss difference to that of DenseNet201 which has achieved the highest training accuracy. It can be inferred that EfficientNetB4 has shown remarkable performance compared to all the other DL models, despite overfitting and application of an imbalanced image dataset. ResNet50 and ResNet50V2 have shown highest overfitting while training the model and have achieved a highest loss difference of 2.49 each, whereas DenseNet121 and InceptionV3 have achieved a lowest loss difference of 1.48 each, thereby representing lowest overfitting.

Figure 65 is a graphical illustration of the accuracy of 26 DL models and their performance difference w.r.t* EfficientNetB7 in terms of accuracy.

Figure 66 is a graphical illustration of the accuracy of 26 DL models and their performance difference w.r.t* EfficientNetB7 in terms of loss.

Discussion

DRFEC is a competitive model and is able to achieve significant results. Dong et al. [23] have implemented an InceptionV3-VGG-16 based hybrid CNN through feature concatenation and have achieved 96.11% accuracy on 2693 images only whereas the proposed model has individually achieved an accuracy of 99.03% and 73.72% in InceptionV3 and 90.91% and 62.07% in VGG-16 using 35,126 images, which is significantly and comparatively larger to fit in a data hungry CNN. Deepa et al. [18] and Deepa et al. [19] have implemented a fine-tuned InceptionV3 and Xception based multi-stage patch-based and image-based CNN and have achieved an accuracy of 96.2% and 96%, respectively using a computationally powerful processing platform on various multi-sized image datasets, which are however overall, less than the Kaggle training dataset employed in proposed DRFEC. It has achieved comparatively an accuracy of 99.03% and 73.72% in InceptionV3 and 99.46% and 78.05% in Xception than Deepa et al. [18]. Tsai et al. [82] and AbdelMaksoud et al. [1] have implemented a DenseNet121 based CNN using Kaggle’s EyePACs train-test dataset and TCH dataset, and using EyePACS Kaggle training dataset, IDRiD, MESSIDOR and APTOS 2019 datasets, and have achieved an accuracy of 84.05% and 84.67%, and 91.2% respectively. Whereas DRFEC has achieved an accuracy of 98.80% and 73.58% using DenseNet121 on 35,126 images. Das, S. et al. [16] have implemented Squeeze-and-Excitation CNN using limited DIARETDB1 and local dataset and achieved an accuracy of 96.92% whereas DRFEC has consistently achieved an accuracy of more than 98% and 70% on 35,126 images. Sau and Bansal [70] have implemented a FNU-GOA-MDNN for optimization using a comparatively limited ISBI 2018 IDRiD dataset and have achieved an accuracy of 95.27%, whereas DRFEC has consistently achieved an accuracy of more than 98% and 70% on 35,126 images. Shaik and Cherukuri [72] have implemented a VGG-16 based HA-Net using 3662 Kaggle’s APTOS 2019 images and ISBI 2018 IDRiD images, and achieved an accuracy 85.54% and 66.41% respectively, whereas DRFEC has achieved an accuracy of 90.91% and 62.07% using VGG-16 and 97.98% and 73.37% using VGG-19 respectively on 35,126 images. It is observed the performance of the models, degrades with decreasing and limited data. The more is the data made available to a DL model the better it fits to be a better learning model which maintains its integrity and consistency, which is represented in DRFEC. Table 6 is an illustration which compares the performance of various existing and state-of-the-art DL models with the proposed DRFEC.

Data availability

There is no additional data associated with this manuscript.