Multi-Attention Integrated Deep Learning Frameworks for Enhanced Breast Cancer Segmentation and Identification

Gamma correction serves as the initial preprocessing step tailored specifically for breast ultrasound images. It plays a pivotal role in enhancing the visibility of crucial anatomical structures and subtle details within the images. By adjusting the image’s brightness and contrast, gamma correction improves the delineation of tumor boundaries and enhances the visibility of tumor features. This step is particularly critical in breast cancer tumor segmentation, where accurate visualization of tumor margins is essential for precise delineation and subsequent analysis.

Gamma correction can be represented mathematically as follows:

$$\underbrace{I_{out}}_{\text{Output Pixel Intensity}}=\underbrace{I_{in}}_{\text{Input Pixel Intensity}}^{\gamma}$$

(1)

Following gamma correction, Gaussian filtering is employed to mitigate speckle noise, a common artifact in ultrasound images that can obscure tumor boundaries and hinder accurate segmentation. By selectively smoothing out noise while preserving essential details, Gaussian filtering improves the clarity of tumor features and enhances the accuracy of segmentation algorithms. This step is crucial in breast cancer tumor segmentation and classification, as it reduces noise artifacts and improves the fidelity of tumor delineation, leading to more accurate and reliable segmentation results. Gaussian filtering ensures that the images are cleaner and more conducive to subsequent segmentation and classification tasks, facilitating the accurate identification and characterization of breast tumors.

Gaussian filtering is given by:

$$\underbrace{I_{out}(x,y)}_{\text{OutputImage}}=\sum_{i=-\frac{N}{2}}^{\frac{N}{2}}\sum_{j=-\frac{N}{2}}^{\frac{N}{2}}\underbrace{I_{in}(x+i,y+j)}_{\text{InputImage}}\cdot\underbrace{G(i,j)}_{\text{GaussianKernel}}$$

(2)

Once the images have undergone gamma correction and Gaussian filtering, resizing is performed to standardize image dimensions, facilitating compatibility with segmentation and classification algorithms. Standardized image dimensions are essential for ensuring consistency and comparability across different datasets and analysis pipelines. Resizing enables researchers to create a uniform framework for analysis, simplifying the processing pipeline and reducing computational complexity.

Resizing of breast cancer images can be represented mathematically as:

$$\underbrace{I_{out}(x^{\prime},y^{\prime})}_{\text{OutputImage}}=\underbrace{I_{in}\left(\frac{x}{r_{x}},\frac{y}{r_{y}}\right)}_{\text{InputImage}}$$

(3)

The last preprocessing stage, normalization, scales the pixel values of images to a standardized range, typically from 0 to 1. This normalization process is crucial for ensuring consistency in pixel intensity across different images, which is essential for training machine learning models and neural networks. Normalization enhances the comparability of images and improves the convergence speed of machine learning algorithms during training. By eliminating variations in intensity that may arise due to differences in acquisition parameters or imaging conditions, normalization ensures that segmentation and classification algorithms can learn effectively from the data, leading to more accurate and reliable analysis results.

The normalization process is denoted as:

$$\underbrace{I_{out}}_{\text{Output Image}}=\frac{\underbrace{I_{in}}_{\text{Input Image}}-\underbrace{\min(I_{in})}_{\text{Minimum Of Input Image}}}{\underbrace{\max(I_{in})}_{\text{Maximum Of Input Image}}-\underbrace{\min(I_{in})}_{\text{Minimum Of Input Image}}}$$

(4)

The encoder section of the segmentation architecture is designed using an InceptionResNet CNN backbone and thus consists of a stem block, three types of InceptionResNet blocks, and two types of reduction blocks.

The stem block begins with three convolution layers and is followed by a max pooling layer and a convolution layer where the layers get executed at the same time. This is followed by a filter concatenation layer and this is split into two paths that are parallel to each other. One of the paths contains two convolutional layers while the other is composed of four convolutional layers. Both paths are combined using filter concatenation and are followed by a parallel convolution and max pooling again which is further followed by filter concatenation.

$$\underbrace{\text{Conv}(I_{(i,j)},F)}_{\text{Convolutional Operation}}=\sum_{m=0}^{M-1}\sum_{n=0}^{N-1}\underbrace{I_{(i+m,j+n)}}_{\text{Input feature map}}\cdot\underbrace{F_{(m,n)}}_{\text{Filter}}+\underbrace{b}_{\text{Bias}}$$

(5)

The Inception Resnet blocks are of three types, named A, B and C respectively. Block A is composed of three different paths and a residual connection. The first path consists of a single convolution operation while the second and third paths consist of three and two convolutional operations respectively. The three paths are combined with the help of another convolution operation followed by concatenation with the residual connection. Blocks B and C are similar but the major difference is with the size of the feature maps since an average pooling operation is responsible for downsampling the data from block B to block C. They are composed of two different paths, one with three convolution operations and the other with one convolution operation. The convolution paths are combined by utilizing another convolution operation. There also exists a residual connection which is combined with the result of the convolution operations by utilizing a convolution operation.

$$\underbrace{\text{Concatenation}(A,B)_{(i,j,k)}}_{\text{Concatenation Operation}}=\begin{cases}\underbrace{A_{(i,j,k)}}_{\text{Input feature map}}&\text{if }1\leq k\leq\text{depth}(A)\\ \underbrace{B_{(i,j,k-\text{depth}(A))}}_{\text{Input feature map}}&\text{if }\text{depth}(A)\leq k\leq\text{depth}(A)+\text{depth}(B)\end{cases}$$

(6)

$$\underbrace{\text{ReLU}(x)}_{\text{Rectified Linear Unit Activation}}=\begin{cases}\underbrace{x}_{\text{Feature map}}&\text{if }x>0\\ 0&\text{otherwise}\end{cases}$$

(7)

The Reduction blocks are of two variants which are called A and B respectively. Block A begins with a filter concatenation operation which is then split into three paths. The first and third paths are composed of a max pooling and a convolution operation respectively and the second path is composed of three convolution layers. The three paths are then combined with the help of a filter concatenation operation. Block B also consists of a max pooling operation and three convolution operations which are present parallelly. Unlike block A, block B is composed of four different parallel paths where the first two paths are described in the previous statement. The other two paths are two convolution operations respectively and all the four paths are combined by utilising a filter concatenation operation.

$$\underbrace{\text{MaxPooling}(O)_{(i,j)}}_{\text{Max Pooling Operation}}=\max_{p=0}^{k-1}\max_{q=0}^{k-1}\underbrace{I_{(i\cdot s+p,j\cdot s+q)}}_{\text{Input feature map}}$$

(8)

$$\underbrace{\text{FilterConcat}(F_{1},F_{2})(X)}_{\text{Filter Concatenation}}=\underbrace{\text{Concatenation}(\text{Conv}(X,F_{1}),\text{Conv}(X,F_{2}))}_{\text{Concatenated Convolutions}}$$

(9)

The Decoder section is composed of decoder blocks, spatial-channel attention blocks, convolution and transpose convolution layers, and a softmax activation function. The decoder block begins with convolution and a batch normalization operation followed by a transpose convolution operation and another batch normalization operation which is followed by convolution and batch normalization operations again.

$$\underbrace{\text{BN}(x)}_{\text{Batch Normalization}}=\underbrace{\gamma}_{\text{Learnable parameter}}\left(\frac{\underbrace{x}_{\text{Input}}-\underbrace{\mu}_{\text{Mean}}}{\sqrt{\underbrace{\sigma^{2}}_{\text{Variance}}+\underbrace{\epsilon}_{\text{Small constant}}}}\right)+\underbrace{\beta}_{\text{Learnable parameter}}$$

(10)

$$\underbrace{\text{TransposeConv}(X,K)_{(i,j,d)}}_{\text{Transpose Convolution}}=\sum_{p=0}^{F-1}\sum_{q=0}^{F-1}\sum_{c=0}^{C-1}\underbrace{X_{(i+s\cdot p,j+s\cdot q,c)}}_{\text{Input feature map}}\cdot\underbrace{K_{(p,q,c,d)}}_{\text{Filter kernel}}$$

(11)

The spatial channel attention block begins with two double convolution operations taking place simultaneously followed by the addition of the two feature maps obtained from the operation. The addition operation is followed by the introduction of non-linearity using ReLU activation followed by another convolution operation. The convolution operation is followed by a sigmoid activation function to restrict the values to lie within the range 0 and 1.

$$\underbrace{\text{AveragePooling}(X)_{(i,j)}}_{\text{Average Pooling}}=\frac{1}{k^{2}}\sum_{p=0}^{k-1}\sum_{q=0}^{k-1}\underbrace{X_{(i\cdot s+p,j\cdot s+q)}}_{\text{Input feature map}}$$

(12)

$$\underbrace{\text{Sigmoid}(x)}_{\text{Sigmoid Activation}}=\frac{1}{1+e^{-\underbrace{x}_{\text{Input}}}}$$

(13)

$$\underbrace{\text{Addition}(A,B)_{(i,j)}}_{\text{Addition Operation}}=\underbrace{A_{(i,j)}}_{\text{Input feature map}}+\underbrace{B_{(i,j)}}_{\text{Input feature map}}$$

(14)

This is followed by channel attention. The Channel attention block is composed of two different pooling operations (max pooling and average pooling) which happen simultaneously and the obtained feature maps are given as input to a shared multi-layered perceptron. The shared MLP is composed of a flatten layer, gaussian error linear unit (GELU) activation function and dropout layers. Once these three operations are done flatten and dropout operations are performed again. The decoder operation ends with a transpose convolution operation, two convolution operations, and a softmax activation function thus displaying the segmented output.

$$y=\underbrace{\text{GELU}(W\cdot x+b)}_{\text{Gated Linear Unit}}$$

(15)

$$\underbrace{\text{GELU}(x)}_{\text{Gaussian Error Linear Unit}}=x\cdot\underbrace{\Phi(x)}_{\text{Gaussian error function}}$$

(16)

$$\underbrace{O_{(i,j)}}_{\text{Output feature map}}=\frac{\underbrace{I_{(i,j)}}_{\text{Input feature map}}}{1-\underbrace{\text{rate}}_{\text{Dropout rate}}}$$

(17)

$$\underbrace{\text{softmax}(z)}_{\text{Softmax Activation}}=\frac{e^{\underbrace{z}_{\text{Output score}}}}{\sum_{i,j}e^{z}}$$

(18)

Initially, the image is processed through a stem block, which captures low-level features such as edges and textures. These edges outline the boundaries of potential tumors, while textures reveal the internal structure of these masses, which often differ significantly between healthy tissue and malignancies. Following the stem block, the image progresses through five InceptionResNet-A Blocks. Microcalcifications require finer resolution, whereas architectural distortions span larger areas. The residual connections, facilitate deeper networks by mitigating the vanishing gradient problem, and capture a broad spectrum of features at various scales. This multi-scale feature capture is crucial for analyzing ultrasound images of breast tissue, where abnormalities can manifest at different scales.

Subsequently, the image encounters a reduction block, which reduces the spatial dimensions of the feature maps. This reduction allows the model to focus on higher-level features and significantly reduces computational complexity, facilitating more efficient processing. This is particularly useful for identifying broader patterns indicative of cancer, such as the overall shape and orientation of a mass.

The image then navigates through two InceptionResNet Block B layers, further through another stem block. This refines the detection of mid-level features, such as more nuanced textural patterns and subtle edge variations. The stem block repetition extracts additional low-level features that complement the more complex features identified in the intermediate stages, ensuring the model has a comprehensive grasp of the ultrasound image’s content.

Following this, the image passes through a reduction block, which reduces the spatial dimensions of the feature maps. This reduction allows the model to focus on low-level features, essential for the precise delineation of tumor boundaries. The image then enters five InceptionResNet Block C layers, and finally into an average pooling layer. These operations are optimized for extracting high-level semantic features for differentiating between various types of tissues present in the image.

This compressed feature map is then fed to the LinkNet decoder which transforms the abstracted feature map into a spatially coherent segmentation map. In the decoder, upsampling refines the segmentation map generated by the encoder, and the attention mechanism focuses specifically on the tumor region, enhancing its emphasis. By integrating spatial and channel attention mechanisms, the model can enhance feature maps by emphasizing spatial locations and informative channels. This comprehensive approach improves the model’s capability to understand intricate tumor patterns and structures, thereby enhancing segmentation performance.

Initially, the feature map is fed to 2 convolutional blocks, followed by a spatial-channel attention block, which is repeated thrice. They perform a preliminary enhancement of the map, focusing on sharpening the details and adjusting the contrast to make the underlying structures more prominent. This ensures that the feature map contains clear and distinguishable elements that correspond to the anatomical structures within the breast ultrasound images. It is then passed to the first decoder block.

The initial decoder block is designed to capture high-level semantic features essential for segmenting larger structures within breast ultrasound images. It facilitates the reconstruction of the spatial relationships and contextual information abstracted away during the encoding process. The spatial-channel attention block that follows this decoder block scrutinizes the feature map to identify and accentuate the regions that are most likely to contain tumor structures. This is achieved by assigning higher weights to the spatial locations that exhibit characteristics typical of tumors, such as irregular shapes and unusual textural patterns. The channel attention mechanism analyses the feature map across different channels to determine the ones that carry the most relevant information for segmentation. By amplifying the signals from these informative channels, the model can better discern the unique features that differentiate tumor tissue from the surrounding healthy tissue.

Finer textures and structures within the tumors are captured as the feature map moves up to the second decoder block. The spatial-channel attention block adjusts the feature map’s weights to emphasize the spatial locations where these detailed features are most prominent, resulting in more precise segmentation of smaller tumor components. Channel attention further identifies the most relevant feature map channels for the task, focusing on the texture and shape of the tumors.

In the third decoder block, the feature map captures even more detailed features, including intricate patterns and structures within the tumors. The attention mechanism in this block focuses on the boundaries of the tumor region, which helps the model improve the quality of the produced segmentation map. This makes the output more accurate and minimizes extraneous markings.

The final decoder block is responsible for capturing the most detailed features, including the specific patterns and structures unique to each tumor. The attention mechanism allows the model to distinguish between benign and malignant types of tumors and identify subtle variations within a single tumor type. The output from this decoder block is transpose-convolved to ensure a consistent output shape of the segmentation map, followed by convolutions to correct the output channels. This finally transforms the abstracted feature map into a detailed and accurate segmentation map.

The model was trained by backpropagating over a custom loss function (21), equal to an aggregate of focal loss (19) and dice (Jaccard) loss (20) obtained after each training epoch.

$$\underbrace{\text{loss}_{\text{focal}}(p_{t})}_{\text{Focal loss}}=-(1-\underbrace{p_{t}}_{\text{True class probability}})^{\underbrace{\gamma}_{\text{Focal loss focusing parameter}}}\log(\underbrace{p_{t}}_{\text{True class probability}})$$

(19)

$$\underbrace{\text{loss}_{\text{Jaccard}}}_{\text{Jaccard loss}}=1-\frac{\underbrace{V_{p}\cap V_{g}}_{\text{Intersection of predicted and ground truth}}}{\underbrace{V_{p}\cup V_{g}}_{\text{Union of predicted and ground truth}}}$$

(20)

$$\underbrace{\text{loss}_{\text{total}}}_{\text{Total loss}}=\underbrace{\text{loss}_{\text{focal}}}_{\text{Focal loss}}+\underbrace{\text{loss}_{\text{Jaccard}}}_{\text{Jaccard loss}}$$

(21)

The model specifications and parameters of the proposed Spatial-Channel Attention LinkNet Framework with InceptionResNet Backbone are shown in Table 1.

The convolutional block within DCNNIMAF consists of a convolutional layer, followed by a batch normalization layer, and finally an activation layer. The activation function used varies between Leaky ReLU and SiLU in different convolutional blocks. The operations performed by the block on the input feature map are mathematically represented as follows:

$$\underbrace{O_{i,j}}_{\text{Output feature map value}}=\sum_{m=0}^{M-1}\sum_{n=0}^{N-1}\underbrace{I_{i+m,j+n}}_{\text{Input pixel value}}\cdot\underbrace{F_{m,n}}_{\text{Filter weight}}+\underbrace{b}_{\text{Bias}}$$

(22)

$$\underbrace{\text{BN}(x)}_{\text{Batch Normalization}}=\underbrace{\gamma}_{\text{Scale parameter}}\frac{\underbrace{x}_{\text{Input value}}-\underbrace{\mu}_{\text{Mean}}}{\sqrt{\underbrace{\sigma^{2}}_{\text{Variance}}+\underbrace{\epsilon}_{\text{Small constant}}}}+\underbrace{\beta}_{\text{Shift parameter}}$$

(23)

$$\text{LeakyReLU}(x)=\begin{cases}\underbrace{x}_{\text{Input value}}&\text{if }x>0\\ \underbrace{\alpha x}_{\text{Leaky slope}}&\text{otherwise}\end{cases}$$

(24)

$$\underbrace{\text{SiLU}(x)}_{\text{Sigmoid Linear Unit}}=\frac{x}{1+e^{-x}}$$

(25)

The double convolutional block comprises two consecutive convolutional layers with 256 filters, a kernel size of 3, and a padding of 1. Mathematically, the operation of this block can be represented as

$$\underbrace{O}_{\text{Output}}=\underbrace{\text{conv}_{2}}_{\text{Second convolution}}\left(\underbrace{\text{conv}_{1}}_{\text{First convolution}}\left(\underbrace{I}_{\text{Input}}\right)\right)$$

(26)

The self-attention block in DCNNIMAF computes the attention weights $\alpha_{ij}$ for each pair of positions $(i,j)$ within the feature map.

$$\underbrace{\alpha_{ij}}_{\text{Attention weight for position }(i,j)}=\underbrace{\text{softmax}\left(\frac{\underbrace{QK^{T}}_{\text{Dot product of query and key}}}{\sqrt{\underbrace{d_{k}}_{\text{Dimensionality of key vectors}}}}\right)}_{\text{Softmax normalization}}\underbrace{V}_{\text{Value matrix}}$$

(27)

The flow of information through DCNNIMAF begins with an input layer of shape (256, 256, 3). Initially, the segmentation map undergoes a convolutional block with 512 filters, a padding of 2 and a kernel size of 3. This extracts low-level features such as textures and edges from the input. Following this, the output from the first convolutional block is passed through another convolutional block with 256 filters, the same kernel size, and padding, but with SiLU activation. The introduction of SiLU activation enhances the non-linearity for higher-level feature extraction, which helps to distinguish between different breast tissue characteristics indicative of cancerous growth.

Subsequently, a double convolutional block is applied to further refine feature extraction. By employing consecutive convolutional layers with 256 filters each, this block extracts deeper and more abstract features from the input. Following this, a convolutional block with 128 filters, a kernel size of 4, and padding of 2 is employed, accompanied by a leaky ReLU activation. This operation aims to distill the extracted features into more compact and discriminative representations, facilitating the model’s capability to detect, and interpret complex patterns within the tumor’s structure such as textural anomalies to irregular shapes Continuing the feature refinement process, another convolutional block with 128 filters, a padding of 1, and a kernel size of 3 is applied, this time utilizing SiLU activation.

Subsequently, two convolutional blocks are utilized - the first with 128 filters, a kernel size of 4, and a padding of 2, and the second with 64 filters, a padding of 1, and a kernel size of 3. These features are then fed to a spatial attention mechanism, enhancing the model’s capacity to adjust to subtle differences between various tissue characteristics associated with malignant and benign tumors.

The feature map obtained from the preceding operations is then concatenated with the output from a convolution and batch normalization layer with 64 filters, a padding of 2, and a kernel size of 3. This model integrates both high-level and low-level features across different layers through a concatenation approach, enabling a more comprehensive representation of the input image. This allows the model to learn about the presence of microcalcifications and the density of the tumor tissue, that are most indicative of malignancy.

This concatenated output undergoes further processing through convolutional and activation layers before being upsampled and concatenated again with intricate feature attention results. This iterative refinement process ensures that the model can effectively leverage both global and local contextual details present in the input segmentation map. This is then fed through additional convolutional blocks and pooling layers before being passed through a self-attention block. By incorporating self-attention mechanisms, it allows the model to highlight more weightage to the distribution of cells or the presence of necrosis, filtering out less relevant information and potential artifacts that could obscure diagnosis.

Ultimately, the result from the self-attention block is flattened and subjected to dropout regularization to mitigate overfitting. Dropout prevents the model from relying on specific features or patterns within the training data that may not generalize well to unseen samples, thereby improving its robustness and generalization performance.

The feature map is then directed into a fully connected layer containing 128 neurons, then proceeds to an output layer with three neurons and softmax activation for classification into malignant, benign, or normal categories. This final step consolidates the extracted features into a compact representation suitable for classification, enabling the model to make accurate predictions concerning the existence and severity of breast tumors based on the input ultrasound image. The model’s training parameters were updated after each epoch via backpropagation using the categorical cross entropy loss criterion (29).

$$\underbrace{\text{loss}_{\text{CE}}}_{\text{Categorical Cross Entropy Loss}}=-\log\left(\frac{\underbrace{e^{x_{p}}}_{\text{Exponential of the true class score}}}{\underbrace{\sum_{j=1}^{N}e^{x_{j}}}_{\text{Sum of exponentials of all class scores}}}\right)$$

(28)

The working algorithm in classifying breast cancer as benign, malignant, or normal, is demonstrated in Algorithm 3.

The model specifications and parameters of the proposed DCNNIMAF classifier have been shown in Table 2.

Accuracy measures the proportion of pixels that were classified correctly in the segmentation map compared to the ground truth.

$$\underbrace{\text{accuracy}}_{\text{Segmentation Accuracy}}=\frac{\underbrace{\text{correctly\_classified\_pixels}}_{\text{Number of correctly classified pixels}}}{\underbrace{\text{total\_pixel\_count}}_{\text{Total number of pixels in the image}}}$$

(29)

The IoU score, often termed the Jaccard index, assesses the intersection of the ground truth mask with the predicted segmentation mask divided by their union. It represents the amount of tumor region correctly segmented regarding the total tumor region (ground truth).

$$\underbrace{\text{IoU}_{\text{Score}}}_{\text{IoU}}=\frac{\underbrace{\text{Area}_{\text{segmentation}}\cap\text{Area}_{\text{groundTruth}}}_{\text{Intersection}}}{\underbrace{\text{Area}_{\text{segmentation}}\cup\text{Area}_{\text{groundTruth}}}_{\text{Union}}}$$

(30)

The Dice coefficient, often recognized as the Dice similarity index, assesses the overlap between the ground truth and the predicted segmentation mask.

$$\underbrace{\text{Dice}_{\text{Coefficient}}}_{\text{Dice}}=\frac{2\times|\underbrace{\text{Area}_{\text{segmentation}}\cap\text{Area}_{\text{groundTruth}}}_{\text{Intersection}}|}{|\underbrace{\text{Area}_{\text{segmentation}}}_{\text{Segmentation}}|+|\underbrace{\text{Area}_{\text{groundTruth}}}_{\text{Ground Truth}}|}$$

(31)

Figures 8 and 9 depict the training and validation curves for accuracy and total loss, respectively, obtained while training the proposed segmentation framework. From the graphs, it is evident that the model has achieved a high accuracy of 98.1%, with a minimal loss of 0.06 at the end of 100 epochs. The model also achieved an impressive Dice Coefficient score of 97.2% and an IoU score of 96.9%. The training and validation curves of these metrics have been shown in Figures 10 and 11 respectively.

From the segmentation results, it can be inferred that this model has demonstrated impressive performance. The high values obtained from IoU, Dice Coefficient, and Accuracy scores, along with the minimal total loss imply that the InceptionResNet backbone managed to successfully extract important characteristics from input preprocessed images, and the dual-attention mechanism in the decoder blocks helped fine-tune the segmentation maps during segmentation.

Grad-CAMs, which stands for Gradient-weighted Class Activation Mapping, is a method in deep learning used to visualize important regions in an input image that guide the model’s decision-making process [31]. They are particularly useful in understanding how Convolutional Neural Networks (CNNs) make their predictions, especially in tasks like medical image segmentation, where it is necessary to observe if the attention mechanism carries out its operations properly.

U-Net [32] model attains a Dice coefficient of 82.52% and an IoU score of 69.76%. These scores reflect a foundational capability in segmenting tumors from breast ultrasound images and highlight the model’s limitations in capturing the full extent of tumor boundaries and internal structures, particularly in the nuanced textures and densities often found in breast tissues. Res U-Net [33] enhances the original U-Net with a Dice coefficient of 88% and an IoU score of 80%, demonstrating enhanced performance through the incorporation of residual connections, but further refinements in its network architecture and feature extraction are necessary to achieve optimal segmentation accuracy, especially in dealing with the variable echo intensities and shadowing effects commonly encountered in breast ultrasound imaging. By integrating a DenseNet backbone, the U-Net with DenseNet Backbone [34] reaches a Dice coefficient of 89.8% and an IoU score of 79.1%, showcasing the benefits of dense connectivity in improving segmentation outcomes. However, additional strategies may be required to fully leverage the complex patterns inherent in breast ultrasound images, such as the differentiation between cystic and solid components of tumors, which is critical for accurate diagnosis. The Multi-scale Fusion U-Net [35] achieves a Dice coefficient of 95.35% and an IoU score of 91.12%, marking a significant improvement over earlier models. But it shows suboptimal performance when handling the heterogeneity of breast tissues and the dynamic nature of tumor growth observed in ultrasound sequences. The proposed Spatial-Channel Attention LinkNet Framework with InceptionResNet Backbone stands out with a Dice coefficient of 97.20% and an IoU score of 96.91%. This performance is attributed to the integration of spatial-channel attention mechanisms and the robust InceptionResNet backbone, which together enable precise localization and delineation of tumors, including the ability to distinguish between different types of breast lesions based on their texture, shape, and boundary characteristics.

Accuracy is a fundamental metric that evaluates the overall performance of a model across all classes. It measures the proportion of true classifications (both true positives and true negatives) in the total images classified, providing a comprehensive view of the model’s effectiveness in correctly classifying instances.

$$\text{Accuracy}=\frac{\sum_{k=1}^{n}(TP_{k}+TN_{k})}{\sum_{k=1}^{n}(TP_{k}+TN_{k}+FP_{k}+FN_{k})}$$

(32)

Where:

• •

  $TP$ denotes the number of true positives.

• •

  $TN$ denotes the number of true negatives.

• •

  $FP$ denotes the number of false positives.

• •

  $FN$ denotes the number of false negatives.

• •

  $n$ denotes the total number of classes.

Precision focuses on the proportion of true positive predictions among all positive predictions made by the classifier. It is particularly important in situations where false positives are costly, as it helps in minimizing the impact of false positives on the overall performance of the model.

$$\text{Precision}=\frac{\sum_{k=1}^{n}TP_{k}}{\sum_{k=1}^{n}(TP_{k}+FP_{k})}$$

(33)

Recall, also known as sensitivity, measures the ability of the classifier to identify all relevant instances within a specific class. It is crucial in situations where missing a positive instance (false negative) is more detrimental than identifying a negative instance as positive (false positive). Recall helps in ensuring that the model does not overlook any relevant instances.

$$\text{Recall}=\frac{\sum_{k=1}^{n}TP_{k}}{\sum_{k=1}^{n}(TP_{k}+FN_{k})}$$

(34)

F1-Score combines precision and recall into a single measure, providing a balanced view of the model’s performance. It is useful in scenarios where both false positives and false negatives are equally important, and a balance between these two metrics is desired.

$$\text{F1 Score}=\frac{2\sum_{k=1}^{n}TP_{k}}{\sum_{k=1}^{n}(2TP_{k}+TN_{k}+FP_{k})}$$

(35)

The proposed DCNNIMAF classifier was trained for 100 epochs, and the evaluation metrics were recorded after each epoch. The training and validation curves obtained for accuracy coupled with categorical cross-entropy loss have been depicted in Figures 13 and 14 respectively. From the graph plots, it can be seen that the classification model has obtained a high accuracy of 99.2% at a minimal loss of 0.03. Figures 15, 16, and 17 display the training and validation precision, recall, and F1-score curve, respectively. It can be inferred from the graphs, that the proposed model has minimized false positives and false negatives, thereby achieving a remarkable precision of 99.3% and a recall of 99.1%. The high values of precision and recall contribute to the high F1-score value of 99.1%.

The normalized confusion matrix obtained on the validation data using the trained DCNNIMAF classification model has been presented in Figure 18. A normalized confusion matrix is a type of confusion matrix where the values are normalized to show proportions or percentages. It is useful for comparing classification performance across classes, since the values are between 0 to 1, making it easy to interpret.

In Figure 18, the normalized confusion matrix depicts the proposed model’s classification performance across the three breast cancer classes: "benign," "normal," and "malignant." Each row corresponds to the actual class, with each column representing the predicted class. The matrix’s values show the proportion of true-class cases that were successfully classified (along the diagonal) or misclassified (off-diagonal).

From the matrix, it can be observed that the model has obtained remarkable accuracy. With most values along the diagonal close to one, it indicates that the majority of the samples were categorized correctly. For the "benign" class, the model had a true positive rate of 0.99, indicating that 99% of benign tumors were properly categorized. In the "normal" class, the true positive rate was 0.98, implying that 98% of normal cases were correctly identified. Similarly, in the "malignant" class, the true positive rate was 0.99, indicating that 99% of malignant tumors were correctly identified. Misclassification errors were minor, with extremely low false positive and false negative rates.

The proposed DCNNIMAF model is compared with other pretrained CNNs, including EfficientNetV2[36], MobileNetV2 [37], [38], NASNetMobile[39], Xception[40], InceptionV3[41], InceptionResNetV2[30], MobileNet[42], VGG16[43], and ResNet50[44]. This comparison aims to provide an overall assessment of the proposed model relative to existing baseline CNNs widely utilized for breast cancer classification. All models, including the proposed one, are trained utilizing the identical dataset, and the outcomes are presented in Table 5. The performance of these models is evaluated based on the following metrics: Accuracy (Acc), Precision (Prec), Recall (Rec), and F1 Score (F1).

From Table 5, it is evident that the proposed DCNNIMAF model has outperformed all baseline CNN models in terms of performance evaluation metrics. EfficientNetV2 overfits on the data due to difficulty in generalizing the nuanced features of breast cancer like irregular margins of malignant lesions or varying degrees of echogenicity observed in ultrasound images. MobileNetV2’s lightweight architecture struggles with the detailed analysis required to detect early signs of breast cancer, such as subtle changes in echotexture or the presence of microcalcifications within lesions. While DenseNet121 benefits from dense connectivity for feature reuse, its performance in identifying specific breast cancer markers like the orientation and distribution of calcifications or the assessment of lesion vascularity is compromised. NASNetMobile, designed for mobile applications, lacks the precision needed to capture the complex interplay of features indicative of breast cancer, such as the irregular shapes of masses or variations in posterior acoustic shadowing. Xception does not fully exploit the spatial dependencies crucial for identifying specific indicators of breast cancer, such as the pattern of calcifications or the echogenicity of surrounding tissue.

InceptionV3’s design compromise for computational efficiency limits its capacity to analyze the multidimensional data characteristic of breast cancer ultrasound images, particularly in detecting subtle architectural distortions or changes in tissue echotexture. Despite its sophisticated architecture, InceptionResNetV2 does not optimally align with the need to identify specific, disease-related features like the texture and margin irregularities of masses or the presence of ductal abnormalities. MobileNet’s focus on efficiency limits its depth necessary for detailed feature extraction from breast cancer ultrasound images. VGG16’s simplicity and relative shallowness struggles with the detailed analysis required to detect and classify features such as the presence of posterior acoustic enhancement, leading to lower accuracy in validation tests. Features such as the assessment of lesion margins might not be adequately learned due to limitations in the ResNet50’s depth and focus. The proposed DCNNIMAF model distinguishes itself by effectively integrating multiple spatial and self-attention mechanisms, enabling precise identification of critical features such as calcifications, architectural distortions, and mass margins. These enhancements allow the model to capture the complex, heterogeneous pathology of breast cancer evident in ultrasound imagery.

From the results presented in Table 6, it is apparent that the DCNNIMAF model proposed in this research outperforms all other models in existing research. The assembly of Fine Tuned VGG16 and VGG19 [45] achieves moderate performance with accuracy and F1-scores around 95%. Its performance is relatively low, indicating potential limitations in its ability to capture the complexity of breast cancer pathology fully. CNN-based Ensemble Learner with MLP Meta Classifier [46] has shown high performance with an accuracy of 98% but has struggled with identifying subtle changes in the irregular shapes of masses. BCCNN [47] shows promising results with metrics around 98%. However, the slight variation in F1-score compared to the highest performers suggests it faces challenges in maintaining a balance between precision and recall, essential for minimizing errors in breast cancer diagnosis.

ResNet50 Hybrid with SVM [48] presents strong recall but exhibits a lower precision score. This discrepancy indicates that while the model is capable of identifying many positive cases, it struggles with accurately distinguishing between benign and malignant lesions, leading to potential false positives. The precision score of Deep CNN with Fuzzy Merging [49] drops significantly highlighting a critical issue in its ability to classify breast cancer cases precisely. This suggests that while the model captures broad patterns effectively, it overlooks finer details necessary for accurate diagnosis. Xception combined with SVM R [50] shows a balanced performance of around 96% but indicates a relative inefficiency in comparison to other models in terms of feature extraction capabilities, leading to inefficiency in real-world use. Grid-based Deep Feature Generator with DNN Classifier [51] demonstrates a high precision score, but the minor discrepancies in recall and F1-score indicate potential inefficiencies in capturing all relevant pathological features, affecting its overall efficacy.

InceptionV3 with Residual Connections [52] achieves a high recall but significantly lower precision, indicating a significant imbalance in its diagnostic capabilities. This suggests challenges in accurately discriminating between similar-looking benign and malignant cases, which is crucial for reducing false positives. EDLCDS-BCDC [53] presents moderate performance across metrics, around 95% to 97%, highlighting potential shortcomings in accurately identifying subtle differences. AlexNet, ResNet50, and MobileNetV2 Hybrid Feature Extractor with mRMR and SVM [54]. shows solid performance with accuracy and F1-scores around 95%. However, its limitations suggest shortcomings in fully adapting to the complex and varied nature of breast cancer pathology, indicating areas for potential enhancement.

The proposed DCNNIMAF model demonstrates remarkable performance across all metrics evaluated, surpassing all other models in this comparison. This can be attributed to its meticulously designed architecture that incorporates advanced feature extraction techniques and multiple attention mechanisms, allowing for the precise and effective identification of the nuanced pathological features associated with breast cancer. This specialized approach ensures not only high accuracy but also maintains excellent precision and recall, showcasing its robustness and reliability in clinical applications for breast cancer classification.