Concurrent Spatial and Channel ‘Squeeze &

Excitation’ in Fully Convolutional Networks

Abhijit Guha Roy1,2 , Nassir Navab2,3 , Christian Wachinger1

1
Artificial Intelligence in Medical Imaging (AI-Med), KJP, LMU München, Germany.
2
Computer Aided Medical Procedures, Technische Universität München, Germany.
3
Computer Aided Medical Procedures, Johns Hopkins University, USA.
arXiv:1803.02579v2 [cs.CV] 8 Jun 2018

Abstract. Fully convolutional neural networks (F-CNNs) have set the

state-of-the-art in image segmentation for a plethora of applications. Ar-
chitectural innovations within F-CNNs have mainly focused on improving
spatial encoding or network connectivity to aid gradient flow. In this pa-
per, we explore an alternate direction of recalibrating the feature maps
adaptively, to boost meaningful features, while suppressing weak ones.
We draw inspiration from the recently proposed squeeze & excitation
(SE) module for channel recalibration of feature maps for image clas-
sification. Towards this end, we introduce three variants of SE modules
for image segmentation, (i) squeezing spatially and exciting channel-wise
(cSE), (ii) squeezing channel-wise and exciting spatially (sSE) and (iii)
concurrent spatial and channel squeeze & excitation (scSE). We effec-
tively incorporate these SE modules within three different state-of-the-
art F-CNNs (DenseNet, SD-Net, U-Net) and observe consistent improve-
ment of performance across all architectures, while minimally effecting
model complexity. Evaluations are performed on two challenging appli-
cations: whole brain segmentation on MRI scans and organ segmentation
on whole body contrast enhanced CT scans.

1 Introduction

Deep learning, in particular, convolutional neural networks (CNN) have become

the standard for image classification [1, 2]. Fully convolutional neural networks
(F-CNNs) have become the tool of choice for many image segmentation tasks in
medical imaging [3–5] and computer vision [6–9]. The basic building block for all
these architectures is the convolution layer, which learns filters capturing local
spatial pattern along all the input channels and generates feature maps jointly
encoding the spatial and channel information. While much effort is put into
improving this joint encoding of spatial and channel information, encoding of the
spatial and channel-wise patterns independently is less explored. Recent work
attempted to address this issue by explicitly modeling the interdependencies
between the channels of feature maps. An architectural component called squeeze
& excitation (SE) block [10] was introduced, which can be seamlessly integrated
within any CNN model. The SE block factors out the spatial dependency by
global average pooling to learn a channel specific descriptor, which is used to
2

recalibrate the feature map to emphasize on useful channels. Its nomenclature

is motivated by the fact that the SE block ‘squeezes’ along the spatial domain
and ‘excites’ or reweights along the channels. A convolutional network with SE
blocks won the first place in the ILSVRC 2017 classification competition on the
ImageNet dataset, indicating its effectiveness [10].
In this work, we want to leverage the high performance of SE blocks for image
classification to image segmentation with F-CNNs. We refer to the previously
introduced SE block as channel SE (cSE), because it only excites channel-wise,
which proved to be effective for classification. We hypothesize that for image
segmentation, the pixel-wise spatial information is more informative. Hence, we
introduce another SE block, which ‘squeezes’ along the channels and ‘excites’
spatially, termed spatial SE (sSE). Finally, we propose to have concurrent spatial
and channel SE blocks (scSE) that recalibrate the feature maps separately along
channel and space, and then combines the output. Encouraging feature maps
to be more informative both spatially and channel-wise. To the best of our
knowledge, this is the first time that spatial squeeze & excitation is proposed for
neural networks and the first integration of squeeze & excitation in F-CNNs.
We integrate the proposed SE blocks (cSE, sSE and scSE) in three state-of-
the-art F-CNN models for image segmentation to demonstrate that SE blocks
are a generic network component to boost performance. We evaluate the seg-
mentation performance in two important medical applications: whole-brain and
whole-body segmentation. In whole-brain segmentation, we automatically iden-
tify 27 cortical and subcortical structures on magnetic resonance imaging (MRI)
T1-weighted brain scans. In whole-body segmentation, we automatically label
10 visceral organs on contrast-enhanced CT scan of the abdomen.
Related Work: F-CNN architectures have successfully been used in a wide
variety of medical image segmentation tasks to provide state-of-the-art perfor-
mance. A seminal F-CNN model is U-Net [3], which has an encoder/decoder
based architecture combined with skip connections between encoder and decoder
blocks with similar spatial resolution. SkipDeconv-Net (SD-Net) [4] builds upon
U-Net, using unpooling layers used in [7] for decoding, learnt by jointly optimiz-
ing logistic and Dice loss functions. A more recent architecture introduces dense
connectivity within the encoder and decoder blocks, unlike U-Net and SD-Net
which uses normal convolutions, termed fully convolutional DenseNet [9].

2 Methods
0
Let us assume an input feature map X ∈ RH×W ×C that passes through an
encoder or decoder block Ftr (·) to generate output feature map U ∈ RH×W ×C ,
Ftr : X → U. Here H and W are the spatial height and width, with C 0 and C
being the input and output channels, respectively. The generated U combines
the spatial and channel information of X through a series of convolutional layers
and non-linearities defined by Ftr (·). We place the SE blocks FSE (·) on U to
recalibrate it to Û. We propose three different variants of SE blocks, which are
detailed next. The SE blocks can be seamlessly integrated within any F-CNN
 3

model by placing them after every encoder and decoder block, as illustrated in
Fig. 1(a). Û is used in the subsequent pooling/upsampling layers.

Fig. 1: Illustration of network architecture with squeeze & excitation (SE) blocks. (a)
The proposed integration of SE blocks within F-CNN. (b-d) The architectural design
of cSE, sSE and scSE blocks, respectively, for recalibrating feature map U.

2.1 Spatial Squeeze and Channel Excitation Block (cSE)

We describe the spatial squeeze and channel excitation block, which was pro-
posed in [10]. We consider the input feature map U = [u1 , u2 , · · · , uC ] as a
combination of channels ui ∈ RH×W . Spatial squeeze is performed by a global
average pooling layer, producing vector z ∈ R1×1×C with its k th element
H X
W
1 X
zk = uk (i, j). (1)
H ×W i j
This operation embeds the global spatial information in vector z. This vector
C C
is transformed to ẑ = W1 (δ(W2 z)), with W1 ∈ RC× 2 , W2 ∈ R 2 ×C being
weights of two fully-connected layers and the ReLU operator δ(·). This encodes
the channel-wise dependencies. The dynamic range of the activations of ẑ are
brought to the interval [0, 1], passing it through a sigmoid layer σ(ẑ). The resul-
tant vector is used to recalibrate or excite U to

ÛcSE = FcSE (U) = [σ(zˆ1 )u1 , σ(zˆ2 )u2 , · · · , σ(zˆC )uC ]. (2)
th
The activation σ(ẑi ) indicates the importance of the i channel, which are
rescaled. As the network learns, these activations are adaptively tuned to ignore
less important channels and emphasize the important ones. The architecture of
the block is illustrated in Fig. 1(b).
4

2.2 Channel Squeeze and Spatial Excitation Block (sSE)

We introduce the channel squeeze and spatial excitation block that squeezes
the feature map U along the channels and excites spatially, which we con-
sider important for fine-grained image segmentation. Here, we consider an al-
ternative slicing of the input tensor U = [u1,1 , u1,2 , · · · , ui,j , · · · , uH,W ], where
ui,j ∈ R1×1×C corresponding to the spatial location (i, j) with i ∈ {1, 2, · · · , H}
and j ∈ {1, 2, · · · , W }. The spatial squeeze operation is achieved through a con-
volution q = Wsq ? U with weight Wsq ∈ R1×1×C×1 , generating a projection
tensor q ∈ RH×W . Each qi,j of the projection represents the linearly combined
representation for all channels C for a spatial location (i, j). This projection is
passed through a sigmoid layer σ(.) to rescale activations to [0, 1], which is used
to recalibrate or excite U spatially

ÛsSE = FsSE (U) = [σ(q1,1 )u1,1 , · · · , σ(qi,j )ui,j , · · · , σ(qH,W )uH,W ]. (3)
Each value σ(qi,j ) corresponds to the relative importance of a spatial information
(i, j) of a given feature map. This recalibration provides more importance to
relevant spatial locations and ignores irrelevant ones. The architectural flow is
shown in Fig. 1(c).

2.3 Spatial and Channel Squeeze & Excitation Block (scSE)

Finally, we introduce a combination of the above two SE blocks, which concur-
rently recalibrates the input U spatially and channel-wise. We obtain the con-
current spatial and channel SE, ÛscSE , by element-wise addition of the channel
and spatial excitation, ÛscSE = ÛcSE + ÛsSE . A location (i, j, c) of the input
feature map U is given higher activation when it gets high importance from
both, channel re-scaling and spatial re-scaling. This recalibration encourages the
network to learn more meaningful feature maps, that are relevant both spatially
and channel-wise. The architecture of the combined scSE block is illustrated in
Fig. 1(d).

Model Complexity: Let us consider an encoder/decoder block, with an output

feature map of C channels. Addition of a cSE block introduces C 2 new weights,
while a sSE block introduces C weights. So, the Phincrease in model complexity
of a F-CNN with h encoder/decoder blocks is i=1 (Ci2 + Ci ), where Ci is the
number of output channels for the ith encoder/decoder block. To give a concrete
example, the U-Net in our experiments has about 2.1 × 106 parameters. The
scSE block adds 3.3 × 104 parameters, which is an approximate increase by
1.5%. Hence, SE blocks only increase overall network complexity by a very small
fraction.

3 Experimental Results
In this section, we conducted extensive experiments to explore the impact of
our proposed modules. We chose three state-of-the-art F-CNN architectures, U-
 5

Net [3], SD-Net [4] and Fully Convolutional DenseNet [9]. All of the networks
have an encoder/decoder based architecture. The encoding and decoding paths
consist of repeating blocks separated by down-sampling and up-sampling, respec-
tively. We insert (i) channel-wise SE (cSE) blocks, (ii) spatial SE (sSE) blocks
and (iii) concurrent spatial and channel-wise SE (scSE) blocks after every en-
coder and decoder block of the F-CNN architecture and compare against its
vanilla version.
Datatsets: We use two datasets in our experiments. (i) Firstly, we tackle the
task of segmenting MRI T1 brain scans into 27 cortical and sub-cortical struc-
tures. We use the Multi-Atlas Labelling Challenge (MALC) dataset [11], which is
a part of OASIS [12], with 15 scans for training and 15 scans for testing consistent
to the challenge instructions. The main challenge associated with the dataset are
the limited training data with severe class imbalance between the target struc-
tures. Manual segmentations for MALC were provided by Neuromorphometrics,
Inc.? (ii) Secondly, we tackle the task of segmenting 10 organs on whole-body
contrast enhanced CT (ceCT) scans. We use data from the Visceral dataset [13].
We train on 65 scans from the silver corpus, and test on 20 scans with manual
annotations from the gold corpus. The silver corpus was automatically labeled
by fusing the results of multiple algorithms, yielding noisy labels. The main chal-
lenge associated with the whole-body segmentation is the highly variable shape
of the visceral organs and the capability to generalize when trained with noisy
labels. We use Dice score for performance evaluation.
Model Learning: In our experiments, all of the three F-CNN architectures had
4 encoder blocks, one bottleneck layer, 4 decoder blocks and a classification layer
at the end. The logistic loss function was weighted with median frequency bal-
ancing [8] to compensate for the class imbalance. The learning rate was initially
set to 0.01 and decreased by one order after every 10 epochs. The momentum was
set to 0.95, weight decay constant to 10−4 and a mini batch size of 4. Optimiza-
tion was performed using stochastic gradient descent. Training was continued till
validation loss converged. All the experiments were conducted on an NVIDIA
Titan Xp GPU with 12GB RAM.
Quantitative Results: Table 1 lists the mean Dice score on test data for both
datasets. Results of the standard networks together with the addition of cSE,
sSE and scSE blocks are reported. Comparing along the columns, we observe
that inclusion of any SE block consistently provides a statistically significant
(p ≤ 0.001, Wilcoxon signed-rank) increase in Dice score in comparison to the
normal version for all networks, in both applications. We further observe that
the spatial excitation yields a higher increase than the channel-wise excitation,
which confirms our hypothesis that spatial excitation is more important for seg-
mentation. Spatial and channel-wise SE yields the overall highest performance,
with an increase of 4 − 8% Dice for brain segmentation and 2 − 3% Dice for
whole-body segmentation compared to the standard network. Particularly for
brain, the performance increase is striking, given the limited increase in model

?
http://Neuromorphometrics.com/
6

Table 1: Mean and standard deviation of the global Dice scores for the different F-
CNN models without and with cSE, sSE and scSE blocks on both datasets.
MALC Dataset
Networks No SE Block + cSE Block + sSE Block + scSE Block
DenseNets [9] 0.842 ± 0.058 0.865 ± 0.069 0.876 ± 0.061 0.882 ± 0.063
SD-Net [4] 0.771 ± 0.150 0.790 ± 0.120 0.860 ± 0.062 0.862 ± 0.082
U-Net [3] 0.763 ± 0.110 0.825 ± 0.063 0.837 ± 0.058 0.843 ± 0.062
Visceral Dataset
Networks No SE Block + cSE Block + sSE Block + scSE Block
DenseNets [9] 0.892 ± 0.068 0.903 ± 0.058 0.912 ± 0.056 0.918 ± 0.051
SD-Net [4] 0.871 ± 0.064 0.892 ± 0.065 0.901 ± 0.057 0.907 ± 0.057
U-Net [3] 0.857 ± 0.106 0.865 ± 0.086 0.872 ± 0.080 0.881 ± 0.082

Fig. 2: Boxplot of Dice scores for all brain structures on the left hemisphere (due to
space constraints), using DenseNets on MALC dataset, without and with proposed cSE,
sSE, scSE blocks. Grey and white matter are abbreviated as GM and WM, respectively.

complexity. Comparing the results across network architectures, DenseNets yield

the best performance.
Fig. 2 and Fig. 3 present structure-wise results for whole brain and whole
body segmentation, respectively, for DenseNets. In Fig. 2, we observe that sSE
and scSE outperform the normal model consistently for all the structures. The
cSE model outperforms the normal model in most structures except some chal-
lenging structures like 3rd/4th ventricles, amygdala and ventral DC where its
performance degrades. One possible explanation could be the small size of these
structures, which might have got overlooked by only exciting the channels. The
performance of sSE and scSE is very close. For whole body segmentation, in
Fig. 3, we observe a similar pattern.
Qualitative Results: Fig. 4 presents segmentation results for MRI T1 brain
scan in Fig. 4(a-d) and for Whole body ceCT scans in Fig. 4(e-h). We show
the input scan, ground truth annotations, DenseNet segmentation along with
our proposed DenseNet+scSE segmentation. We highlight ROIs with a white
box and red arrow, to show regions where inclusion of scSE block improved
the segmentation. For MRI brain scan segmentation, we indicate the structure
left putamen, which is under segmented using DenseNet (Fig. 4(c)), but the
 7

Fig. 3: Structure-wise Dice performance of DenseNets on Visceral dataset, without

and with proposed cSE, sSE, scSE blocks. Left and right are indicated as L. and R.
Psoas major muscle is abbreviated as PM.

segmentation improves with the inclusion of the scSE block (Fig. 4(d)). For whole
body ceCT, we indicate the spleen, which is over segmented using DenseNet
(Fig. 4(g)), and which is rectified with adding scSE block (Fig. 4(h)).

4 Conclusion
We proposed the integration of squeeze & excitation blocks within F-CNNs for
image segmentation. Further, we introduced the spatial squeeze & excitation,
which outperforms the previously proposed channel-wise squeeze & excitation.
We demonstrated that SE blocks yield a consistent improvement for three differ-
ent F-CNN architectures and for two different segmentation applications. Hence,
recalibration with SE blocks seems to be a fairly generic concept to boost per-
formance in CNNs. Strikingly, the substantial increase in segmentation accuracy
comes with a negligible increase in model complexity. With the seamless integra-
tion, we believe that squeeze & excitation can be a crucial component for neural
networks in many medical applications.
Acknowledgement: We thank the Bavarian State Ministry of Education, Sci-
ence and the Arts in the framework of the Centre Digitisation.Bavaria (ZD.B)
for funding and NVIDIA corporation for GPU donation.

References
1. Krizhevsky, A., Sutskever, I. and Hinton, G.E., 2012. Imagenet classification with
deep convolutional neural networks. In NIPS pp. 1097-1105.
2. He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image
recognition. In CVPR, pp. 770-778, IEEE.
3. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical
image segmentation. In Proc. MICCAI, Springer 2015, pp. 234-241.
4. Roy, A.G., Conjeti, S., Sheet, D., Katouzian, A., Navab, N. and Wachinger, C.,
2017, September. Error Corrective Boosting for Learning Fully Convolutional Net-
works with Limited Data. In MICCAI, pp. 231-239, Springer.
5. Roy, A.G., Conjeti, S., Navab, N. and Wachinger, C., 2018. QuickNAT: Segmenting
MRI Neuroanatomy in 20 seconds. arXiv:1801.04161.
6. Long J, Shelhamer E, and Darrell T. Fully convolutional networks for semantic
segmentation. In CVPR 2015, pp. 3431-40, IEEE.
8

Fig. 4: Input scan, ground truth annotations, DenseNet segmentation and

DenseNet+scSE segmentation for both whole-brain MRI T1 (a-d) and whole-body
ceCT (e-h) are shown. ROIs are indicated by white box and red arrow highlighting
regions where the scSE block improved the segmentation, for both applications.

7. Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation.

In ICCV 2015, pp. 1520-28, IEEE.
8. Badrinarayanan V, Kendall A, Cipolla R. Segnet: A deep convolutional encoder-
decoder architecture for image segmentation. In arXiv preprint:1511.00561, 2015.
9. Jégou, S., Drozdzal, M., Vazquez, D., Romero, A. and Bengio, Y., 2017, July. The
one hundred layers tiramisu: Fully convolutional densenets for semantic segmenta-
tion. In CVPR Workshop, pp. 1175-1183, IEEE.
10. Hu, J., Shen, L. and Sun, G. Squeeze-and-excitation networks. In CVPR 2018.
11. Landman, B, Warfield, S., Miccai Workshop on Multiatlas Labeling. In MICCAI
Grand Challenge, 2012.
12. Marcus, D.S., Fotenos, A.F., Csernansky, J.G., Morris, J.C. and Buckner, R.L.,
2010. Open access series of imaging studies: longitudinal MRI data in nondemented
and demented older adults. Journal of cognitive neuroscience, 22(12), pp.2677-2684.
13. Jimenez-del-Toro, O., et al. 2016. Cloud-based evaluation of anatomical structure
segmentation and landmark detection algorithms: VISCERAL anatomy bench-
marks. IEEE TMI, 35(11), pp.2459-2475.