ECG Arrhythmia Classification Using Transfer

Learning from 2-Dimensional Deep CNN Features

Milad Salem, Shayan Taheri, Jiann-Shiun Yuan

Department of Electrical and Computer Engineering
University of Central Florida
Orlando, Florida, 32816 U.S.A.
Email: {miladsalem, shayan.taheri, jiann-shiun.yuan}@ucf.edu

Abstract— Due to the recent advances in the area of deep Memory Networks (LSTM) [2], and Convolutional Neural
learning, it has been demonstrated that a deep neural network, Network (CNN) [3]. One of the main advantages of using
trained on a huge amount of data, can recognize cardiac Deep Neural Networks (DNNs) is the fact that a neural
arrhythmias better than cardiologists. Moreover, traditionally network can automatically learn complex representative
feature extraction was considered an integral part of ECG features directly from the data itself, therefore, eliminating the
pattern recognition; however, recent findings have shown that need for using manual feature extraction. Utilizing this merit
deep neural networks can carry out the task of feature extraction gives an opportunity to create end-to-end learning systems that
directly from the data itself. In order to use deep neural networks take ECG signals as input and output arrhythmia class
for their accuracy and feature extraction, high volume of training
prediction, while extracting the “deep features” autonomously.
data is required, which in the case of independent studies is not
Another advantage of using a DNN is that deeper networks, in
pragmatic. To arise to this challenge, in this work, the
identification and classification of four ECG patterns are studied presence of sufficient amount of data, can deliver higher
from a transfer learning perspective, transferring knowledge accuracy and better results in classification of fine-grained
learned from the image classification domain to the ECG signal ECG signals.
classification domain. It is demonstrated that feature maps In [3], it was shown that a 34-layered deep convolutional
learned in a deep neural network trained on great amounts of Neural Network can outperform board-certified cardiologist in
generic input images can be used as general descriptors for the detecting abnormalities and arrhythmias in ECG signal. In their
ECG signal spectrograms and result in features that enable
work a dataset of near 64000 ECG recordings from
classification of arrhythmias. Overall, an accuracy of 97.23
percent is achieved in classifying near 7000 instances by ten-fold
approximately thirty thousand patients was used for training
cross validation. the DNN. However, this dataset is nearly 500 times larger than
other datasets of its kind, demonstrating the fact that data
Keywords— DenseNet, electrocardiogram, feature extraction, volume was one of the main factors in achieving such
machine learning, neural network, transfer learning. performance.
In spite of the advantages that DNNs present in the ECG
I. INTRODUCTION domain, one disadvantage is dominantly hindering the wide
Traditionally, identifying patterns in an electrocardiogram usage of these tools, the disadvantage of data volume. As
(ECG) signal and classifying the type of arrhythmia witnessed compared to the traditional classification approaches DNNs
in the signal relied heavily on the features extracted from the require a considerable amount of data to be trained. This
signal. These features included statistical features, signal problem creates a gap between the dataset size and deep
procession features, and medical features, which in the end features, since the datasets that are publicly available in this
required many engineering and optimization or domain domain lack in volume [3].
expertise to deliver high accuracy [1]. Once the features were In order to fill this gap and arise to the problem of low ECG
extracted, classification is traditionally done using support data volume versus high-performing deep features, in this
vector machine (SVM), Random Forest (RF), K-Nearest work, we propose using transfer learning from the 2-
Neighbor (KNN), Feed Forward Neural Network, and a myriad dimensional domain. The image classification and object
of other classification tools that showed capability in this task recognition domain are of the richest domains regarding the
[2]. training data volume, in contrast to the ECG signal domain
Recent state-of-the-art performances achieved by deep where datasets are relatively small. These domains contain
learning methods in popular pattern recognition problems have sufficient amount of data to train DNNs and find feature maps
motivated researchers and engineers to implement these that are capable of representing complex patterns in images.
techniques to the field of biomedical image and signal These learnt feature maps can be transferred to the ECG
processing. In this regard, the deep learning methods have domain, if the 1-D ECG signal is transformed into a 2-D image
shown promising results in the ECG domain using Recurrent using spectrograms. It will be shown that a Deep Neural
Neural Networks (RNN), specifically Long Short Term Network, DenseNet, pre-trained on ImageNet classes can act

978-1-5386-3603-9/18/$31.00 ©2018 IEEE

Authorized licensed use limited to: National Taiwan Univ of Science and Technology. Downloaded on September 07,2022 at 10:02:50 UTC from IEEE Xplore. Restrictions apply.
 thoroughly as feature extractor from ECG signal spectrograms C. Transfer Learning and off-the-shelf CNN features
on classification of four different rhythms including Normal Knowledge learnt from pattern in one domain or task may
Sinus Rhythm (Normal), Atrial Fibrillation and Flutter (AF), be applicable to patterns in another domain or task. Utilization
Ventricular Fibrillation (VF), and ST Segment Change (ST). of Transfer Learning (TL) technique allows this knowledge to
be transferred between two domains and used in the latter
II. BACKGROUND domain to enable classification. An application of this
technique is leveraging a pre-trained deep CNN for automatic
A. The Types of Arrhythmias Studied feature extraction. The convolutional layers inside this network
A heart arrhythmia is a group of illness conditions in which contain feature maps that are learnt during training on the
the heart beats faster, slower, or in an irregular manner, original dataset and hold knowledge regarding the patterns
commonly due to a disease. In this work three types of existing in that dataset. These feature maps can act as feature
arrhythmias are of interest: (1) Ventricular Fibrillation, which extractor from another dataset. These “off-the-shelf” extracted
is rapid and irregular electrical activity that causes the features from the intermediate layers of a deep neural network
ventricles to fail to contract in a synchronized manner, are strong enough to conquer the hand-crafted features and be
resulting in cessation of cardiac output, (2) Atrial Fibrillation, an ideal candidate for feature extraction [4].
which is defined as a very rapid and random disturbance of the Here, we aim to bring the knowledge learnt from millions
atrium, (3) Change of the ST segment, including elevation or of images in the ImageNet dataset, via a deep neural network
depression of the ST segment, based on the level of the (DenseNet), to the ECG domain and classify a small dataset of
segment is raised above or below the baseline. thousands of instances using this neural network as a feature
extractor. We show that the patterns learnt from the ImageNet
B. The Challenge of Fine-Grained Arrhythmia Classification dataset, consisting of many classes of images such as animals
In independent studies, similar to this work, when the and objects, can be used to represent ECG spectrograms.
challenge of examining a unique abnormal rhythm is faced, the
1) DenseNet
existence of sufficient amount of training data (containing
Densely Connected CNN (DenseNet) is a deep CNN with
records of the target rhythm) is vital. The details that need to be
connections between each layer and every other layer in a feed-
examined in an ECG signal are often fine-grained and similar,
forward approach [5]. This type of connection helps attenuate
therefore, have patterns that are hard to detect, even for trained
the problem of vanishing-gradient, resulting in better training
cardiologists [1]. However, the datasets existing in this domain
and feature propagation. DenseNet showed promising results in
contain a small amount of data or none for many abnormal
object recognition benchmark tasks. The architecture of
rhythms. This challenge is amplified when one aims to
DenseNet involves four Dense Blocks variable in length. In
approach it from a deep learning perspective; while deep
this work we use a pre-trained DenseNet, which has 161
learning can help the detection of the fine-grained patterns,
convolutional layers through-out its structure and examine the
training a deep neural network requires a humongous amount
outputs of these layers for feature extraction.
of data. If one attempts to train a deep neural network on a
small amount of training data, overfitting may occur, which
results in the classifier failing to detect patterns in unseen data. III. METHODOLOGY
Therefore, the availability of datasets, the amount of data, A. Method Overview
and reliability of the results of classifying unseen data propose
a challenge in training deep neural networks for ECG pattern To classify the input ECG signal into 4 classes of interest,
recognition. However, for the same reasons, we were the recordings are first cut, and the data is selected based on the
encouraged to use deep learning from another perspective, annotations. Each data instance is transformed into an image
which not only is resilient towards the aforementioned issues, via using spectrograms. The images are fed into a pre-trained
but also helps resolve them. In this work, we examine the DenseNet, a 161 layered deep CNN, and features are extracted
usage of “Transfer Learning” and “Off-the-Shelf CNN by finding the output of 12 intermediate convolutional layers.
Features” for achieving high accuracy and reliability in These features are classified using SVM by ten-fold cross
abnormal rhythm detection in ECG signals. The achieved validation and the optimum layer is found based on their
results is for the case of small training dataset. performance. The classification steps are shown in Fig. 1.

Fig. 1. Different stages of the classification system

Authorized licensed use limited to: National Taiwan Univ of Science and Technology. Downloaded on September 07,2022 at 10:02:50 UTC from IEEE Xplore. Restrictions apply.
 B. Data Sets in Use
The data classified in this work is extracted from four
different datasets: 1) MIT-BIH Atrial Fibrillation Dataset: This
dataset consists of 24 recordings of nearly ten hours long,
recorded from human subjects with atrial fibrillation (mostly
paroxysmal). 2) The MIT-BIH Malignant Ventricular
Arrhythmia Database: Containing 22 half-hour recordings, this
dataset included patterns for ventricular tachycardia,
ventricular flutter, and ventricular fibrillation. 3) European ST-
T Database: including 90 annotated excerpts of ambulatory
ECG recordings with different lengths, this dataset contains
367 episodes for ST change. 4) The MIT-BIH Normal Sinus
Rhythm Database: this database includes 18 long-term ECG Fig. 2. Spectrogram of a sample data instance belonging to each class
recordings of subjects that had no significant arrhythmias.
This theory is intuitively understandable since each layer
holds different feature maps which are activated in the
C. Data Selection presence of specific patterns. To perform feature extraction, the
The aforementioned datasets contain many recordings and output of a few convolutional layers inside DenseNet are
consequently many beats. In order to select data instances that examined and considered as feature vectors. We randomly
hold enough information, we define a window size of 500 select twelve layers and extract a feature vector from each
samples and cut each recording into data instances that contain layer’s output for each input image.
this number of samples. This window size enables
approximately 3 to 7 beats in each data instance. F. Feature Selection and classification
The datasets in use contain arrhythmias that have been It is crucial to understand that each extracted feature vector
annotated by different cardiologists in different approaches. contains many feature maps that exist on our pre-trained
The AF and VF dataset have annotations marking the change DenseNet and were formed to be able to distinguish between
in the type of the rhythm. For the ST dataset, the annotations the thousand classes of ImageNet. Therefore, not all the
mark the boundaries of each change in the ST level. In order to patterns that exist on the feature maps of each layer may act as
stay accurate, the data selection occurs near the annotation a good feature extractor. To increase the accuracy and to select
mark or near the peak of the change on all of the recordings in these specific feature maps, we reduce the dimensionality of
the dataset. In the Normal dataset, only the beats are annotated, these vectors by selecting the feature maps that have high
therefore, the data selection occurs randomly around the importance using the chi-squared test. The classification of
annotated beats. In order to be diverse, data selections in all each feature vector is done via a support vector classifier with a
classes are done from all of the recordings in each dataset. linear kernel. The classification is repeated with feature
selection and the results are recorded.
Overall, 7008 data instances are selected that show the
symptoms and patterns of the rhythms of interest. This data
volume is considerably small compared to the millions of data IV. RESULTS
instances used in popular image classification tasks.
A. Experimental Setup
D. Transformation from 1D Signal to 2D Image In order to extract the feature vectors from the
Since in this work a deep neural network pre-trained on spectrograms, we acquired a Keras implementation of
images is used as feature extractor for ECG signals, the data DenseNet from [7]. The DenseNet-161 pre-trained weights and
instances need to be transformed into an image. In order to do model were used, resulting in a deep neural network with 161
so, we use spectrograms. Spectrograms are able to capture the convolutional layers trained on a thousand classes of generic
changes in the power of the signal in an image by taking the images. We examined 12 layers randomly and used the output
Fourier transform of each partition of the signal. The number of these layers as feature vectors. After feature extraction, the
of partitions can be seen as a tune-able hyper-parameter which vectors were classified via linear SVM using ten-fold cross-
depends on the details of the signal and the relevant changes of validation. The process was repeated when feature selection
the signal between classes. In this work 31 partitions were was applied inside each fold.
chosen to create the spectrograms. A sample of each class’s
spectrogram is shown in Fig. 2. B. Results
The result of feature extraction via DenseNet and
E. Feature Representation Via DenseNet classification via SVM are shown in Fig. 3. It is visible that
It has been shown that generic descriptors extracted from some layers contain feature maps that perform well on the task
the inner layers of convolutional neural networks can be very of recognizing the patterns of the input data and result in higher
potent in representing the input. It was shown in Nguyen et al. accuracy. Moreover, the optimum layer for feature extraction is
[6] that the layer of which the features have been extracted layer 112 both before and after feature selection is applied. Via
from can have direct effect on the accuracy of the recognition selecting the features, the overall accuracy is generally
task. increased nearly 2 percent, showing the importance of this task.

Authorized licensed use limited to: National Taiwan Univ of Science and Technology. Downloaded on September 07,2022 at 10:02:50 UTC from IEEE Xplore. Restrictions apply.
 When examining different layers’ performance on each class, it spectrograms, respectively. The results of which are shown in
is clear that ST was the hardest class to classify in all layers, Table II, alongside the results from using DenseNet as feature
having the lowest F1 score regardless of the selected layer. extractor, and two related works.
Furthermore, this figure demonstrates that each layer has
different capabilities in extracting features from different Table II demonstrates three points. Firstly, transforming
classes’ inputs, having different patterns residing within. ECG signals via spectrograms improves the accuracy of the
model greatly. Secondly, utilization of DenseNet as feature
Fig. 3. Results of SVM model on each layer’s feature vectors, overall (left extractor results in better discrimination of the spectrograms
plot) and for each class (right plot). with furthered accuracy. Thirdly, our results have
outperformed [8] and are arguably scalable to [3]. While [3]
had a high number of classes with a higher possible overlap
between them, the humongous size of the training data allowed
having high precision and recall. On the other hand, [8] with a
close number of classes to our work and more instances,
achieved a lower F1-score. Therefore, via transfer learning and
translation to 2-D domain, we were able to extract better
representatives of ECG signals and improve classification
results despite the small amount of data.

V. CONCLUSION
The best result for feature extraction via DenseNet is from
the 112th layer, combined with classification via SVM and This work, to our understanding, is the first effort to use a
feature selection. This layer delivers 97.23 percent accuracy. deep CNN, pre-trained on millions of images, as a feature
The confusion matrix for the final result is shown in Fig. 4. extractor from spectrograms of ECG signals. An accuracy of
Table 1 compares the results for different classes. 97.23 percent is achieved in classifying a small dataset of
approximately seven thousand samples of ECG signals,
Fig. 4. The normalized confusion matrix of the best result achieved containing four classes of rhythms, using a pre-trained 161
layered DenseNet as a feature extractor and SVM for
classification. The results show that firstly using spectrograms
to transform ECG signals to images can preserve their fine-
grained details; secondly feature maps learnt from a colossal
amount of generic data in a deep neural network can act very
well to represent spectrograms of ECG signals.

REFERENCES
[1] S. Hong and M. Wu and Y. Zhou and Q. Wang and J. Shang and H. Li
and J. Xie, "ENCASE: An ENsemble ClASsifiEr for ECG classification
using expert features and deep neural networks," 2017 Computing in
Cardiology (CinC), Rennes, 2017, pp. 1-4.
[2] Özal Yildirim, A novel wavelet sequence based on deep bidirectional
LSTM network model for ECG signal classification, Computers in
TABLE I. THE FINAL RESULT OF CLASSIFICATION FOR EACH CLASS Biology and Medicine, Volume 96, 2018, Pages 189-202, ISSN 0010-
USING DENSENET AS FEATURE EXTRACTOR. 4825.
Class Precision Recall F1 [3] Rajpurkar, Pranav & Hannun, Awni & Haghpanahi, Masoumeh &
Bourn, Codie & Y. Ng, Andrew. (2017). Cardiologist-Level Arrhythmia
Control 99.17 99.61 99.39
Detection with Convolutional Neural Networks.
AF 94.76 98.27 96.48
[4] Razavian, Ali Sharif, Hossein Azizpour, Josephine Sullivan, and Stefan
VF 99.22 99.30 99.26 Carlsson. "CNN features off-the-shelf: an astounding baseline for
ST 97.89 90.92 94.28 recognition." In Computer Vision and Pattern Recognition Workshops
(CVPRW), 2014 IEEE Conference on, pp. 512-519. IEEE, 2014.
TABLE II. COMPARISON OF DIFFERENT CLASSIFICATION APPROACHES. [5] Huang, Gao, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der
Maaten. "Densely connected convolutional networks." In Proceedings of
Approach #Classes #Instances Precision Recall Accuracy F1 the IEEE conference on computer vision and pattern recognition, vol. 1,
1D Signal+ SVM 4 7008 57.46 56.66 61.00 48.27 no. 2, p. 3. 2017.
2D Spec.+ SVM 4 7008 89.43 88.33 89.04 88.85 [6] K. Nguyen, C. Fookes, A. Ross and S. Sridharan, "Iris Recognition With
Off-the-Shelf CNN Features: A Deep Learning Perspective," in IEEE
2D Spec.+ DenseNet 4 7008
97.76 97.02 97.23 97.35 Access, vol. 6, pp. 18848-18855, 2018.
+ SVM
[3] 14 ~1,502,000 80.0 78.4 -- 77.6 [7] Felix Yu, DenseNet Implementation (2017), GitHub repository.
Available online: https://github.com/flyyufelix/ (accessed Apr. 16th
[8] 5 100,389 63.5 60.3 97.6 61.85 2018)
In order to show the effects of bringing 1-D ECG signals to [8] Kiranyaz S, Ince T, Gabbouj M (2016). “Real-time patient-specific ECG
the 2-D domain using spectrograms, two additional SVM classification by 1-D convolutional neural networks.” IEEE
models were trained on raw 1-D signals and their Transactions on Biomedical Engineering, Volume 63, pp. 664-675

Authorized licensed use limited to: National Taiwan Univ of Science and Technology. Downloaded on September 07,2022 at 10:02:50 UTC from IEEE Xplore. Restrictions apply.