PatchResNet: Multiple Patch Division–Based Deep Feature Fusion Framework for Brain Tumor Classification Using MRI Images

Literature Review

Nowadays, many studies have been conducted on the accurate classification of brain tumors using artificial intelligence (AI) techniques [10, 17, 18]. A summary of studies conducted on brain tumor classification using AI techniques is provided in Table 1.

It can be noted from the table that the majority of proposed methods have used deep learning methods. These methods need big data to train deep networks, and data augmentation techniques are to be applied to such datasets [36]. Nevertheless, such methods generally achieve high classification accuracy.

Motivation and Our Framework

The main motivation of this research is to increase the classification ability of ResNet [39] in transfer learning mode. Today, patch-based models have attained high classification ability for computer vision. Notable models include vision transformers (ViT) [40], multilayer perceptron mixers (MLP-mixer) [41], and convolutional mixers (ConvMixer) [42], all of which have attained high classification ability. In ConvMixer, the authors show that the high classification capability has been attained using the mixer layer with CNN. In ViT, they used fixed-sized patches and transformers to classify an image. In the experiments with ViT, they used 14 × 14, 16 × 16, and 32 × 32 sized patches to show experiments. We have used three types of patches together in this framework to attain different results.

We proposed a new framework called PatchResNet, which uses two feature extractors and three feature selectors for patches. This architecture produces 18 results. In addition, IHMV was used in this study. A total of 34 (= 18 + 16) results were generated, and best among them were chosen using IHMV. Hence, our developed architecture is self-organizing image classification.

Novelties and Contributions

The novelties of the proposed work are given below:

• In this work, three different types of fixed-size patch divisions have been applied.

• In our framework, features have been generated using pretrained ResNet50. By deploying the last pooling and fully connected layer, two feature extractors have been created, and these feature extractors have been applied to these patches to get different features.

• Three feature selectors have been used in our framework together.

• By applying IHMV, voted results have been created.

Multiple Patch-Based Deep Feature Extraction

The primary novelty of this layer is multiple patch divisions. 32 × 32, 56 × 56, and 112 × 112 sized fixed-size patches have extracted features from local areas. Using these sizes of the patches (32 × 32, 56 × 56, and 112 × 112), 49, 16, and 4 patches have been created. Six feature vectors have been generated by extracting features from each group by deploying the last pooling and fully connected layers of the pretrained ResNet50. A graphical explanation has been given to explain the proposed feature extraction layer.

In this figure (see Fig. 2), the abbreviations used are given as follows: FC: fully connected layer, Ff: features of the fully connected layer, Pf: features of the pooling layer, F: final feature vector. Herein, 32 × 32, 56 × 56, and 112 × 112 sized patches have been applied to the image to create patches. Using 32 × 32, 56 × 56, and 112 × 112 sized non-overlapping blocks, 49, 16, and 4 patches have been created from 224 × 224 sized image, and these are named p¹, p², and p³ in this image. Using each patch group, FC (fully connected), and pooling layer of the pretrained ResNet50, 138 feature vectors are generated. Sixty-nine (49, 16, and 4 of them belong to first, second, and third patch groups) of them are generated using the FC layer, and 69 out of them are generated by deploying the pooling layer. In this layer, the generated 138 features are divided into six groups. These groups are named Ff¹ (this group contains 49 feature vectors, and they are generated using 32 × 32 sized patches and FC layer), Pf¹ (this group contains 49 feature vectors, and they are generated using 32 × 32 sized patches and pooling layer), Ff² (this group contains 16 feature vectors and they generated using 56 × 56 sized patches and FC layer), Pf² (this group contains 16 feature vectors and they generated using 56 × 56 sized patches and pooling layer), Ff³ (this group contains 4 feature vectors and they generated using 112 × 112 sized patches and FC layer), and Pf³ (this group contains 4 feature vectors, and they generated using 112 × 112 sized patches and pooling layer).

By merging these groups, six feature vectors have been generated.

The steps of the proposed multiple patch-based feature generation layers are:

Step 0: Load the image and resize the image to 224 × 224 sized images.

Step 1: Divide the image into patches with sizes of 32 × 32, 56 × 56, and 112 × 112.

Step 2: Generate features deploying the last pooling layer (global average pooling layer – avg_pool) and fully connected layer (fc1000). The used ResNet50 was trained on the ImageNet1K dataset.

Herein, $\mathit{Ff}$ and $\mathit{Pf}$ are fully connected and pooling features. These features are generated from pretrained ResNet50 ($R50(.)$).

Step 3: Concatenate the feature vectors generated.

Herein, $F_h$ is hth created the final feature vector, and we generated six feature vectors.

In this layer, six feature vectors have been calculated, and presented in Table 2.

Feature Selection Layer

This layer is needed to decrease the number of features and increase the number of feature vectors. Three commonly known feature selectors are used in this layer. These feature selectors are NCA [46], Chi2 [47], and ReliefF [48]. NCA and ReliefF are distance base feature selectors that calculate weights for each feature. NCA only generates positive weights, but ReliefF can generate positive and negative weights to qualify features. Chi2 is one of the fastest feature selection functions since it uses a simple statistical moment.

This paper proposes multiple selectors based on the most informative features selection layer. The model developed in this paper uses the pooling and fully connected layers of the ResNet50 architecture to extract six different feature vectors (two feature vectors for each patch size). Then, these six feature vectors are fed to the NCA, Chi2, and ReliefF methods for feature selection. This way, three qualified index values are calculated for each feature vector. This phase generates 18 feature vectors containing qualified index information. The graphical outline of this layer is demonstrated in Fig. 3.

The steps of the proposed feature selection model are:

Step 4: Calculate qualified indexes of each feature vector by deploying NCA, Chi2, and ReliefF.

Herein, $\mu (.,.)$ is NCA, $\chi (.,.)$ represents Chi2, and $\varpi (.,.)$ defines ReliefF functions. The input parameters of these feature selectors are feature vectors and actual output ($y$). Three qualified indexes ($\mathit{idx}$) have been generated using these three functions,.

Step 5: Select the top 512 features by deploying the indexes generated.

where $s_c$ is the selected cth feature vector with a length of 512, and $\mathit{dim}$ represents the number of images/observations.

Classification

A simple/shallow classifier (kNN) has been used in the classification layer [49]. We used kNN to demonstrate the classification capabilities of the 18 feature vectors. A MATLAB classification learner was used to select the most appropriate classifier, and Fine kNN (1NN) was selected. We only changed the distance parameter of the Fine kNN. We changed the distance parameter to L1-norm (City block) instead of L2-norm (Euclidean) since NCA and ReliefF use L1-norm to calculate distances. Tenfold cross-validation (tenfold CV) has been used to get robust results.

Step 8: Classify the generated each feature by deploying kNN.

Herein, $r_c$ is cth validation prediction vector with a length of $\mathit{dim}$.

Majority Voting Layer

The primary goal of this layer is to increase the calculated classification performance in the classification layer. Therefore, the IHMV algorithm was used. IHMV is a loop-based majority voting model and uses a mode function. The steps of this layer are:

Step 9: Sort the calculated results (r) in accordance with their accuracy.

where $\mathit{ind}$ defines sorted/qualified indexes by descending, $\varsigma (.)$ is the sorting function, and $ac{c}^r$ is an accuracy vector with a.

Step 10: Create an array using a loop.

Herein, $\mathit{arr}$ is an array.

Step 11: Calculate the voted results by deploying the mode function.

Herein, $v_k$ is kth voted result (validation prediction vector) and $\psi (.)$ is the mode function. In this step, 16 voted results are generated from 18 validation prediction vectors.

In the last step, the best accurate validation prediction vector was chosen as a result.

Step 12: Select the best accurate vector among the 34 (18 kNN results + 16 voted results) results.

Experimental Setup

We used the MATLAB (2022a) programming tool to implement PatchResNet. The pretrained ResNet50 was also imported to MATLAB. We used a simple configured laptop for implementation. This laptop has Intel Core i7 10870H processor, 16 GB main memory, and 512 GB hard disk. We did not use any graphical processing units since we used ResNet50 in transfer learning mode. The transition of the proposed PatchResNet is tabulated in Table 3.

The parameters of the proposed PatchResNet are tabulated in Table 3. The calculated results have been generated using these parameters. Using different sizes of patches, feature extractors, feature selectors, classifiers, and voted algorithms, variable classification models can be proposed.

Dataset

We used an open-access MRI dataset that is popular for computer vision applications and is publicly available on the Kaggle website (https://www.kaggle.com/). This dataset has four categories with 3264 MR images. The distribution of this dataset is given as follows. There are 926 scans of brains with glioblastoma multiforme (GBM), 937 meningioma images, 901 pituitary tumor images, and 500 control scans of healthy individuals [50, 51]. The sample images of this dataset have been demonstrated in Fig. 4.

Performance Evaluation Metrics

Standard performance evaluation metrics—accuracy, F1-score, precision, and recall—were used to evaluate classification results. Accuracy is the oldest classification evaluation performance metric and is calculated using the number of true predicted observations. Recall defines the ratio of the number of true positives to the sum of true positives and false negatives which is a useful performance measure to evaluate the unbalanced datasets. Finally, precision is used to calculate the ratio of the true positives with all positives and is very important to show the diagnosis rate. To express precision and recall using a parameter, the F1-score (it is the harmonic mean of the precision and recall) has been used.

Results

Precision, recall, accuracy, and F1-score have been calculated to compute results. The obtained confusion matrix is presented in Fig. 5.

The results obtained from deploying the confusion matrix (see Fig. 5) are presented in Table 4.

Table 4 demonstrates that the proposed PatchResNet attained 98.10% classification accuracy, 98.15% unweighted average recall, 97.91% average precision, and 98.01% overall F1-score. Moreover, the best results class is Pituitary since the recall of this class is equal to 100%. GBM also attained 100% precision.

We proposed a new framework named PatchResNet because three types of patch divisions were used in this work. We selected pretrained ResNet50 to extract features as the fully connected layer of ResNet50 has generally been used to extract deep features in the literature. This research used both global average pooling and fully connected layers of the pretrained ResNet50 to obtain two deep feature extractors. Variable-sized feature vectors have been obtained by using different patch divisions and two feature extractors. In the feature selection phase, the most informative 512 features were selected, three feature selectors were used, and 18 (= 3 × 2 × 3) feature vectors were calculated. kNN was applied to these 18 selected feature vectors to calculate classification results, and the calculated accuracies of these 18 selected feature vectors are depicted in Fig. 6.

Figure 6 demonstrates that the best accurate feature vector is the 7th feature vector and the 7th feature vector yielded 96.54% classification accuracy. This vector is created using fixed-size patches of 56 × 56, feature extraction using the global average pooling layer of the ResNet50 and NCA feature selector. Using the outputs of the proposed PatchResNet, comparative results have been calculated in accordance with the size of the patch, feature extractor, and feature selectors. These comparative results are demonstrated in Fig. 7.

Figure 7 demonstrates the average classification accuracies of the methods used. We have used three types of patch divisions, and these are 32 × 32, 56 × 56, and 112 × 112. Our calculated average classification accuracies are 92.82%, 95.04%, and 94.85% for 32 × 32, 56 × 56, and 112 × 112 sized patch divisions, respectively. Two feature selectors were used. The average classification accuracy of the pooling layer–based feature extractor is 94.58%, and the average classification accuracy of the FC-based feature extractor is 93.89%. According to the feature selectors performance evaluation, the best selector is NCA since the average classification of NCA is 96%. Average classification accuracies ReliefF and Chi2 are 93.39% and 93.32%, respectively. According to Fig. 7, the best size for patch division is 56 × 56, the best feature extraction model pooling layer of the ResNet50, and the most suitable selector is NCA. Moreover, the 7th selected feature vector used these components and achieved the best classification accuracy among the 18 generated classification results by deploying a kNN classifier with a tenfold CV (see Fig. 6). Moreover, the statistical analysis of these components is given in Table 5.

We applied the statistical t-test to the generated 18 feature vectors to obtain clinically significant features. Herein, our reference point is p-value since features with a p-value less than 0.005 are considered distinct features. In each feature vector, there are 512 features, and this dataset has four classes. Therefore, p-values of $6=\left(\begin{array}{c}4\\ 2\end{array}\right)$ couples have been calculated. Using p-values, the number of distinct features has been calculated and is shown in Fig. 8.

Figure 8 demonstrates that our generated features are distinctive based on p-value analysis and yielded high classification performance.

IHMV was used to calculate voted classification accuracies, and 16 voted results have been calculated. These classification accuracies of the voted vectors are demonstrated in Fig. 9.

Figure 9 demonstrates classification accuracies via the number of predicted vectors used to calculate voted vectors. According to Fig. 9, the best classification accuracy was 98.10%, attained by voting the best 11 results. Moreover, all voted results are higher than 97%.

We used 18 pretrained CNNs to get comparative results. The used CNNs are (1) ResNet18, (2) ResNet50, (3) ResNet101, (4) DarkNet19, (5) MobileNetV2, (6) DarkNet53, (7) Xception, (8) ShuffleNet, (9) NasNetMobile, (10) NasNetLarge, (11) DenseNet201, (12) InceptionV3, (13) InceptionResNetV2, (14) GoogLeNet, (15) AlexNet, (16) VGG16, (17) VGG19, and (18) SqueezeNet. Pooling/fully connected layers of these networks have been used to extract features. By deploying NCA, the top 512 features were selected, and classification was performed by deploying kNN. The calculated classification accuracies of these pretrained CNNs are shown in Table 6.

Table 6 demonstrates that the best feature extractor among the used 18 CNNs is pretrained DenseNet201, and it attained 95.83% classification accuracy. Moreover, ResNet50 attained 93.90% classification accuracy without using patch division. By deploying patch division, the accuracy rate of the ResNet50 was increased from 93.90 to 96.45% (see Fig. 6). Moreover, PatchResNet increased classification performance to 98.10%. Table 6 tabulates that our proposed PatchResNet increased the classification performance of the pretrained ResNet50. To show the superiority of the developed framework, the class-wise comparison results with ViT are given in Table 7.

Tummala et al. [52] used the ViT method with the same dataset in their study. As can be seen in Table 7, our method has achieved better classification performance for 2 classes (Meningioma and Pituitary). For GBM, the ViT method achieved higher classification accuracy. However, they applied a 70:30 hold-out validation strategy in their study. But, we have used a tenfold CV to obtain more generalized robust results.

In this study, we used the second dataset to evaluate the proposed method’s performance. The dataset presented by Cheng et al. [27] contains 3064 images belonging to three classes: GBM (1426), Meningioma (708), and Pituitary (930) [53]. The test results obtained on this dataset are given in Table 8.

As seen in Table 8, the proposed framework has achieved more than 95% classification accuracy using the second dataset. Hence, our proposed method illustrates high classification performance using both datasets.

Conflict of Interest

The authors declare no competing interests.