13674 IEEE SENSORS JOURNAL, VOL. 20, NO.

22, NOVEMBER 15, 2020

Detection of Respiratory Infections Using

RGB-Infrared Sensors on Portable Device
Zheng Jiang , Graduate Student Member, IEEE, Menghan Hu , Member, IEEE, Zhongpai Gao ,
Lei Fan, Ranran Dai, Yaling Pan, Wei Tang, Guangtao Zhai , and Yong Lu

Abstract —Coronavirus Disease 2019 (COVID-19) caused by

severe acute respiratory syndrome coronaviruses 2 (SARS-
CoV-2) has become a serious global pandemic in the past few
months and caused huge loss to human society worldwide.
For such a large-scale pandemic, early detection and isolation
of potential virus carriers is essential to curb the spread of
the pandemic. Recent studies have shown that one important
feature of COVID-19 is the abnormal respiratory status caused
by viral infections. During the pandemic, many people tend to
wear masks to reduce the risk of getting sick. Therefore, in this
paper, we propose a portable non-contact method to screen
the health conditions of people wearing masks through analy-
sis of the respiratory characteristics from RGB-infrared sen-
sors. We first accomplish a respiratory data capture technique for people wearing masks by using face recognition. Then,
a bidirectional GRU neural network with an attention mechanism is applied to the respiratory data to obtain the health
screening result. The results of validation experiments show that our model can identify the health status of respiratory
with 83.69% accuracy, 90.23% sensitivity and 76.31% specificity on the real-world dataset. This work demonstrates that
the proposed RGB-infrared sensors on portable device can be used as a pre-scan method for respiratory infections, which
provides a theoretical basis to encourage controlled clinical trials and thus helps fight the current COVID-19 pandemic.
The demo videos of the proposed system are available at: https://doi.org/10.6084/m9.figshare.12028032.
Index Terms — COVID-19 pandemic, deep learning, dual-mode tomography, health screening, recurrent neural network,
respiratory state, SARS-CoV-2, thermal imaging.

Manuscript received June 18, 2020; accepted June 21, 2020. Date of I. I NTRODUCTION
publication June 24, 2020; date of current version October 16, 2020. This
work was supported in part by the National Natural Science Foundation
of China under Grant 61901172, Grant 61831015, and Grant U1908210,
in part by the Shanghai Sailing Program under Grant 19YF1414100,
T O TACKLE the outbreak of the COVID-19 pandemic,
early control is essential. Among all the control measures,
efficient and safe identification of potential patients is the
in part by the Science and Technology Commission of Shanghai Munic-
ipality (STCSM) under Grant 18DZ2270700 and Grant 19511120100, most important part. Existing researches show that the human
in part by the Foundation of Key Laboratory of Artificial Intelligence, physiological state can be perceived through breathing [1],
Ministry of Education under Grant AI2019002, and in part by the Funda- which means respiratory signals are vital signs that can reflect
mental Research Funds for the Central Universities. The associate editor
coordinating the review of this article and approving it for publication was human health conditions to a certain extent [2]. Many clinical
Dr. Ioannis Raptis. (Zheng Jiang and Menghan Hu contributed equally literature suggests that abnormal respiratory symptoms may
to this work.) (Corresponding authors: Guangtao Zhai; Yong Lu.) be important factors for the diagnosis of some specific dis-
Zheng Jiang, Zhongpai Gao, Lei Fan, and Guangtao Zhai are with the
Institute of Image Communication and Information Processing, Shanghai eases [3]. Recent studies have found that COVID-19 patients
Jiao Tong University, Shanghai 200240, China, and also with the Key have obvious respiratory symptoms such as shortness of
Laboratory of Artificial Intelligence, Ministry of Education, Shanghai breath fever, tiredness, and dry cough [4], [5]. Among those
200240, China (e-mail: zhaiguangtao@sjtu.edu.cn).
Menghan Hu is with the Shanghai Key Laboratory of Multidimen- symptoms, atypical or irregular breathing is considered as
sional Information Processing, East China Normal University, Shanghai one of the early signs according to the recent research [6].
200062, China, and also with the Key Laboratory of Artificial Intelligence, For many people, early mild respiratory symptoms are diffi-
Ministry of Education, Shanghai 200240, China.
Ranran Dai is with the Department of Pulmonary and Critical Care cult to be recognized. Therefore, through the measurement
Medicine, Ruijin Hospital, School of Medicine, Shanghai Jiao Tong of respiration conditions, potential COVID-19 patients can
University, Shanghai 200240, China. be screened to some extent. This may play an auxiliary
Yaling Pan and Yong Lu are with the Department of Radiology, Ruijin
Hospital Luwan Branch, School of Medicine, Shanghai Jiao Tong Univer- diagnostic role, thus helping find potential patients as early
sity, Shanghai 200240, China (e-mail: 18917762053@163.com). as possible.
Wei Tang is with the Department of Respiratory Disease, Ruijin Traditional respiration measurement requires attachments of
Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai
200240, China. sensors to the patient’s body [7]. The monitor of respiration
Digital Object Identifier 10.1109/JSEN.2020.3004568 is measured through the movement of the chest or abdomen.

© IEEE 2020. This article is free to access and download, along with rights for full text and data mining, re-use and analysis

Authorized licensed use limited to: Mapua University. Downloaded on March 31,2021 at 03:47:34 UTC from IEEE Xplore. Restrictions apply.
JIANG et al.: DETECTION OF RESPIRATORY INFECTIONS USING RGB-INFRARED SENSORS 13675

Contact measurement equipment is bulky, expensive, and time- depth camera to classify abnormal respiratory patterns in
consuming. The most important thing is that the contact during real-time and achieved excellent results [9].
measurement may increase the risk of spreading infectious dis- In this paper, we propose a remote, portable and intelli-
eases such as COVID-19. Therefore, the non-contact measure- gent health screening system based on respiratory data for
ment is more suitable for the current situation. In recent years, pre-screening and auxiliary diagnosis of respiratory diseases
many non-contact respiration measurement methods have been like COVID-19. To be more practical in situations where
developed based on image sensors, doppler radar [8], depth people often choose to wear masks, the breathing data capture
camera [9] and thermal camera [10]. Considering factors such method for people wearing masks is introduced. After extract-
as safety, stability and price, the measurement technology of ing breathing data from the videos obtained by the thermal
thermal imaging is the most suitable for extensive promotion. camera, a deep learning neural network is performed to work
So far, thermal imaging has been used as a monitoring tech- on the classification between healthy and abnormal respiration
nology in a wide range of medical fields such as estimations of conditions. To verify the robustness of our algorithm and the
heart rate [11] and breathing rate [12]–[14]. Another important effectiveness of the proposed equipment, we analyze the influ-
thing is that many existing respiration measurement devices ence of mask type, measurement distance and measurement
are large and immovable. Given the worldwide pandemic, angle on breathing data collection.
the portable and intelligent screening equipment is required to The main contributions of this paper are threefold. First,
meet the needs of large-scale screening and other application we combine the face recognition technology with dual-mode
scenarios in a real-time manner. For thermal imaging-based imaging to accomplish a respiratory data extraction method for
respiration measurement, nostril regions and mouth regions people wearing masks, which is quite essential for the current
are the only focused regions since only these two parts have situation. Based on our dual-camera algorithm, the respiration
periodic heat exchange between the body and the outside data is successfully obtained from masked facial thermal
environment. However, until now, researchers have barely con- videos. Subsequently, we propose a classification method
sidered measuring thermal respiration data for people wearing to judge abnormal respiratory states with a deep learning
masks. During the epidemic of infectious diseases, masks framework. Finally, based on the two contributions mentioned
may effectively suppress the spread of the virus according to above, we have implemented a non-contact and efficient health
recent studies [15], [16]. Therefore, to develop the respiration screening system for respiratory infections using the collected
measurement method for people wearing masks becomes quite data from the hospital, which may contribute to finding the
practical. In this study, we develop a portable and intelligent possible cases of COVID-19 and keeping the control of
health screening device that uses thermal imaging to extract the second spread of SARS-CoV-2.
respiration data from masked people, which is then used
to do the health screening classification via deep learning II. M ETHOD
architecture. A brief introduction to the proposed respiration condition
In classification tasks, deep learning has achieved state- screening method is shown below. We first use the portable
of-the-art performance in most research areas. Compared and intelligent screening device to get the thermal and the
with traditional classifiers, classifiers based on deep learning corresponding RGB videos. During the data collection, we also
can automatically identify the corresponding features and perform a simple real-time screening result. After getting the
their correlations rather than extracting features manually. thermal videos, the first step is to extract respiration data
Recently, many researchers have developed detection methods from faces in thermal videos. During the extraction process,
of COVID-19 cases through medical imaging techniques such we use the face detection method to capture people’s masked
as chest X-ray imaging and chest CT imaging [17]–[19]. areas. Then a region of interest (ROI) selection algorithm is
These studies have proved that deep learning can achieve proposed to get the region from the mask that stands for
high accuracy in detection of COVID-19. Based on the the characteristic of breath most. Finally, we use a bidi-
nature of the above methods, they can only be used for rectional GRU neural network with an attention mechanism
the examination of highly suspected patients in hospitals, (BiGRU-AT) model to work on the classification task with the
and may not meet the requirements for the larger-scale input respiration data. A key point in our method is to collect
screening in public places. Therefore, this paper pro- respiration data from facial thermal videos, which has been
poses a scheme based on breath detection via a thermal proved to be effective by many previous studies [26]–[28].
camera.
For breathing tasks, deep learning-based algorithm can also
better extract the corresponding features such as breathing rate A. Overview of the Portable and Intelligent Health
and inhale-to-exhale ratio, and make more accurate predic- Screening System for Respiratory Infections
tions [20]–[23]. Recently, many researchers made use of deep Our data collection obtained by the system is shown in
learning to analyze the respiratory process. Cho et al. used a Fig. 1. The whole screening system includes a FLIR one
convolutional neural network (CNN) to analyze human breath- thermal camera, an Android smartphone and the corresponding
ing parameters to determine the degree of nervousness through application we have written, which is used for data acquisition
thermal imaging [24]. Romero et al. applied a language model and simple instant analysis. Our screening equipment, whose
to detect acoustic events in sleep-disordered breathing through main advantage is portable, can be easily applied to measure
related sounds [25]. Wang et al. utilized deep learning and abnormal breathing in many occasions of instant detection.

Authorized licensed use limited to: Mapua University. Downloaded on March 31,2021 at 03:47:34 UTC from IEEE Xplore. Restrictions apply.
13676 IEEE SENSORS JOURNAL, VOL. 20, NO. 22, NOVEMBER 15, 2020

is proposed to further excavate high-level contextual features.

Then, the output and those low-level features are combined in
low-level feature pyramid layers. Finally, the result is obtained
after another two layers of deep neural network. For faces
that a lot of features are lost due to the cover of a mask,
such a context-sensitive structure can obtain more feature
correlations and thus improve the accuracy of face detection.
In our experiment, we use the open-source model from the
paddle hub which is specially trained for masked faces to
detect the face area on RGB videos.
The next step is to extract the masked area and map the area
from RGB video to thermal video. Since the position of the
mask on the human face is fixed, after obtaining the position
coordinates of the human face, we obtain the mask area of the
Fig. 1. Overview of the portable and intelligent health screening system
for respiratory infections: a) device appearance; b) analysis result of face by scaling down in equal proportions. For a detected face
the application. (Notice that the system can simultaneously collect body with width w, and height h, the location of left-up corner is
temperature signals. In the current work, this body temperature signal defined as (0, 0), the location of right-bottom corner is then
is not considered in the model and is only used as a reference for the
users.) (w, h). The corresponding coordinate of the two corners of
the mask region is declared as (w/4, h/2) and (3w/4, 4h/5).
Considering that the background to the boundary of the mask
As shown in Fig. 1, the FLIR one thermal camera consists will produce a large contrast with the movement, which is
of two cameras, an RGB camera and a thermal camera. easy to cause errors, we choose the center area of the mask
We collect the face videos from both cameras and use face through this division. Then the selected area is mapped from
recognition method to get the nostril area and forehead area. the RGB image to thermal image to obtain the masked region
The temperatures of the two regions are calculated in time in thermal videos.
series and shown in the screening result page in Fig. 1(b).
The red line stands for the body temperature and the blue line C. Extract Respiration Data From ROI
stands for breathing data. From the breathing data, we can After getting the masked region in thermal videos, we need
predict the respiratory pattern of the test case. Then, a simple to get the region of interest (ROI) that represents breathing fea-
real-time screening result is given directly in the application tures. Recent studies often characterize breathing data through
according to the extracted features shown in Fig. 1. We use the temperature changes around the nostril [11], [30]. However
raw face videos collected from both RGB camera and thermal when people wear masks, there exists another problem that
camera as the data for further study to ensure accuracy and the nostrils are also blocked by the masks, and when people
higher performance. wearing different masks, the ROI may be different. Therefore,
we perform an ROI tracking method based on maximizing the
variance of thermal image sequence to extract a certain area
B. Detection of Masked Region From Dual Mode Image
on the masked region of the thermal video which stands for
When continuous breathing activities perform, there is a the breath signals most.
fact that periodic temperature fluctuations occur around the Due to the lack of texture features in masked regions com-
nostril due to the inspiration and expiration cycles. Therefore, pared to human faces, we judge the ROI from the temperature
respiration data can be obtained by analyzing the temperature change of the thermal image sequence. The main idea is to
data around the nostril based on the thermal image sequence. traverse the masked region in the thermal images and find a
However, when people wear masks, many facial features are small block with the largest temperature change as the selected
blocked because of this. Merely recognizing the face through ROI. The position of a certain block is fixed in the masked
thermal image will lose a lot of geometric and textural facial region among all the frames since the nostril area is fixed on
details, resulting in recognition errors of the face and mask the face region. We do not need to consider the movement of
parts. In order to solve this problem, we adopt the method the block since our face recognition algorithm can detect the
based on two parallel located RGB and infrared cameras for mask position in each frame’s thermal image. For a certain
face and mask region recognition. The masked region of the block with height m, and width n, we define the average pixel
face is first captured in the RGB camera, then such a region is intensity at frame t as:
mapped to the thermal image with a related mapping function.
1 
m−1 n−1
The algorithm for masked face detection is based on the
pyramidbox model created by Tang et al. [29]. The main s̄(t) = s(i, j, t) (1)
mn
i=0 j =0
idea is to apply tricks like the Gaussian pyramidbox in deep
learning to get the context correlations as further character- For thermal images, s̄(t) represents the temperature value
istics. The face image is first used to extract features of at frame t. For every block we obtained, we calculate their
different scales using the Gaussian pyramid algorithm. For s̄(t) on time line. Then, for each block n, the total variance
those high-level contextual features, a feature pyramid network of the list of average pixel intensity with T frames σs2 (n) is

Authorized licensed use limited to: Mapua University. Downloaded on March 31,2021 at 03:47:34 UTC from IEEE Xplore. Restrictions apply.
JIANG et al.: DETECTION OF RESPIRATORY INFECTIONS USING RGB-INFRARED SENSORS 13677

Fig. 2. The pipeline of the respiration data extraction: a) record the RGB video and thermal video through a FLIR one thermal camera; b) use face
detection method to detect face and mask region in the RGB frames and then map the region to the thermal frames; c) capture the ROIs in the
thermal frames of mask region by tracking method; d) extract the respiration data from the ROIs.

Fig. 3. The structure of the BiGRU-AT network: the network consists of four layers: the input layer, the bidirectional layer, the attention layer and an
output layer. The output is a 2 dimension tensor which indicates normal or abnormal respiration condition.

calculated as shown in Eq. 2, where μ stands for the mean healthy or not as shown in Fig. 3. The input of the network is
value of s̄(t). the respiration data obtained by our extraction method. Since
 the respiratory data is time series, it can be regarded as a
(s̄(t) − μ)2
σs2 (n) = (0 < t < T ) (2) time series classification problem. Therefore, we choose the
T Gate Recurrent Unit (GRU) network with the bi-direction and
Since respiration is a periodic data spread out from the attention layer to work on the sequence prediction task.
nostril area, we can consider that the block with the largest Among all the deep learning structures, recurrent neural
variance is the position where the heat changes most in network (RNN) is a type of neural network which is specially
both frequency and value within the mask, which stands for used to process time series data samples [31]. For a time step
the breath data mostly in the masked region. We adjust the t, the RNN model can be represented by:
corresponding block size according to the size of the masked  
region. For a masked region with N blocks, the final ROI is h (t ) = φ U x (t ) + W h (t −1 ) + b (4)
selected by: o(t ) = V h (t ) + c (5)
 
RO I = arg maxσs2 (n) (3) y (t ) = σ o(t )
 (6)
1≤n<N
where x (t ), h (t ) and o(t ) stand for the current input state,
For each thermal video, we traverse all possible blocks
hidden state and output at time step t respectively. V, W, U
in the mask regions of each frame and find the ROIs for
are parameters obtained by training procedure. b is the bias
each frame by the method above. The respiration data is then
and σ and φ are activation functions. The final prediction
defined as s̄(t)(0 < t < T ), which is the pixel intensities of
y (t ).
is 
ROIs in all the frames.
Long-short term memory network is developed based on
RNN [32]. Compared to RNN, which can only memorize
D. BiGRU-AT Neural Network and analyze short-term information, it can process relatively
We apply a BiGRU-AT neural network to do the classi- long-term information, and is suitable for problems with
fication task on judging whether the respiration condition is short-term delays or long time intervals. Based on LSTM,

Authorized licensed use limited to: Mapua University. Downloaded on March 31,2021 at 03:47:34 UTC from IEEE Xplore. Restrictions apply.
13678 IEEE SENSORS JOURNAL, VOL. 20, NO. 22, NOVEMBER 15, 2020

many related structures are proposed in recent years [33]. GRU research areas. The structure of attention layer is:
is a simplified LSTM which merges three gates of LSTM (for-
get, input and output) into two gates (update and reset) [34]. u t = tanh (Wu h t + bw ) (12)


For tasks with a few data, GRU may be more suitable than exp u  t uw
LSTM since it includes less parameters. In our task, since the at =   
(13)
t exp u t u w
input of the neural network is only the respiration data in time 
sequence, the GRU network may perform a better result than s= αt h t (14)
t
LSTM network. The structure of GRU can be expressed by
the following equations: where h t represents the BiGRU layer output at time step t,
which is bidirectional. Wu and bw are also parameters that
 
vary in the training process. at performs a softmax function
rt = σ Wr · h t −1 , x t + br (7)
 
on u t to get the weight of each step t. Finally, the output of the
z t = σ Wz · h t −1 , x t + bz (8) attention layer s is a combination of all the steps from BiGRU
 

h̃ t = tanh Wh̄ · rt ∗ h t −1 , x t + bh (9) with different weights. By applying another softmax function
h t = (1 − z t ) ∗ h t −1 + z t ∗ h̃ t (10) to the output s, we get the final prediction of the classification
task. The structure of the whole network is shown in Fig. 3.

where rt is the reset gate that controls the amount of infor-

mation being passed to the new state from the previous states. III. E XPERIMENTS
z t stands for the update gate which determines the amount A. Dataset Explanation and Experimental Settings
of information being forgotten and added. Wr , Wz and Wh are Our goal is to distinguish whether there is an epidemic
trained parameters that vary in the training procedure. h̃ t is the of infectious disease such as COVID-19 according to the
candidate hidden layer which can be regarded as a summary abnormal breathing in the respiratory system. We collect the
of the above information h t −1 and the input information x t at healthy dataset from people around our authors. And the
time step t. h t is the output layer at time step t which will be abnormal dataset was obtained from the inpatients of the respi-
sent to the next unit. ratory disease department and cardiology department in Ruijin
The bidirectional RNN has been widely used in natural Hospital. Most of the patients we collected data from caught
language processing [35]. The advantage of such a network basic or chronic respiratory diseases. Only some of them have
structure is that it can strengthen the correlation between the a fever, which is one of the typical respiratory symptoms of
context of the sequence. As the respiratory data is a periodic infectious diseases. Therefore, the body temperature is not
sequence, we use bidirectional GRU to obtain more infor- taken into consideration in our current screening system.
mation from the periodic sequence. The difference between We use a FLIR one thermal camera connected to an Android
bidirectional GRU and normal GRU is that the backward phone to work on the data collection. We collected data
sequence of data is spliced to the original forward sequence from 73 people. For each person, we collected two 20-second
of data. In this way, the hidden layer of the original h(t) is infrared and RGB videos with a sampling frequency of 10 Hz.
changed to: During the data collection process, all testers were required to
be about 50 cm away from the camera and face the camera
→ ←
− − directly to ensure data consistency. The testers were asked
ht = [ ht , ht ] (11)
to stay still and breathe normally during the whole process,
→
− ←− but small movements were allowed. For each infrared and
where h t is the original hidden layer and h t is the backward RGB videos collected, we sampled in step of 100 frames with
→
−
sequence of h t . a sampling interval of 3 frames. Thus we finally obtained
During the analysis of respiratory data, the entire waveform 1,925 healthy breathing data and 2,292 abnormal breathing
in time sequence should be taken into consideration. For some data, a total of 4,217 data. Each piece of data consists
specific breathing patterns such as asphyxia, several particular of 100 frames of infrared and RGB videos in 10 seconds.
features such as sudden acceleration may occur only at a In the BiGRU-AT network, the hidden cells in the BiGRU
certain point in the entire process. However, if we only use layer and attention layers are 32 and 8 respectively. The
the BiGRU network, these features may be weakened as the breathing data is normalized before input into the neural
time sequence data is input step by step, which may cause a network and we use cross-entropy as the loss function. During
larger error in prediction. Therefore, we add an attention layer the training process, we separate the dataset into two parts.
to the network, which can ensure that certain keypoint features The training set includes 1,427 healthy breathing data and
in the breathing process can be maximized. 1,780 abnormal breathing data. And the test set contains
Attention mechanism is a choice to focus only on those 498 healthy breathing data and 512 abnormal breathing data.
important points among the total data [36]. It is often com- Once this paper is accepted, we will release the dataset used
bined with neural networks like RNN. Before the RNN model in the current work for non-commercial users.
summarizes the hidden states for the output, an attention layer Among the whole dataset, we choose 4 typical respiratory
can make an estimation of all outputs and find the most data examples as shown in Fig. 4. Fig. 4(a) and Fig. 4(b) stand
important ones. This mechanism has been widely used in many for the abnormal respiratory patterns extracted from patients.

Authorized licensed use limited to: Mapua University. Downloaded on March 31,2021 at 03:47:34 UTC from IEEE Xplore. Restrictions apply.
JIANG et al.: DETECTION OF RESPIRATORY INFECTIONS USING RGB-INFRARED SENSORS 13679

Fig. 4. Comparison of normal and abnormal respiratory data extracted

by our method: a), b) are abnormal data collected from patients in the
general ward of the respiratory department in Ruijin Hospital; c), d) are
normal data collected from healthy volunteers.

TABLE I
E XPRIMENTAL R ESULTS ON THE T EST S ET

Fig. 5. Confusion matrices of the four models. Each row is the number
of real labels and each column is the number of predicted labels. The left
one is the result of BiGRU-AT, the right one is the result of LSTM.

Fig. 4(c) and Fig. 4(d) represent the normal respiratory pat-
tern called Eupnea from healthy participants. By comparison, from the results, the performance improvement of BiGRU-AT
we can find that the respiratory of normal people are in strong compared to LSTM is mainly in the accuracy rate of the
periodic and evenly distributed while abnormal respiratory negative class. This is because many scatter-like abnormalities
data tend to be more irregular. Generally speaking, most in the time series of abnormal breathing are better recognized
abnormal breathing data from respiratory infections have faster by the attention mechanism. Besides, the misclassification rate
frequency and irregular amplitude. of the four networks is relatively high to some extent which
may be because many positive samples do not have typical
B. Experimental Result respiratory infections characteristics.
The experimental results are shown in Table. I. We consider
four evaluation metrics viz. Accuracy, Sensitivity, Specificity C. Analysis
and F1. To measure the performance of our model, we com- During the data collection process, all testers were required
pare the result of our model with three other models which to be about 50 cm away from the camera and face the
are GRU-AT, BiLSTM-AT and LSTM respectively. The result camera directly to ensure data consistency. However, in real-
of sensitivity reaches 90.23% which is far more higher than time conditions, the distance and angles of the testers towards
the specificity of 76.31%. This may have a positive effect the device cannot be so accurate. Therefore, in the analysis
on the screening of potential patients since the false nega- section, we give 3 comparisons from different aspects to prove
tive rate is relatively low. Our method performs better than the robustness of our algorithm and device.
any other network in all evaluation metrics with the only 1) Influence of Mask Types on Respiratory Data: To measure
exception in the sensitivity value of GRU-AT. By comparison, the robustness of our breathing data acquisition algorithm and
the experimental result demonstrates that attention mechanism the effectiveness of the proposed portable device, we analyze
is well-performed in keeping important node features in the the breathing data of the same person wearing different
time series of breathing data since the networks with attention masks. We design 3 mask-wearing scenarios that cover most
layer all perform a better result than LSTM. Another point is situations: wearing one surgical mask (blue line); wearing one
that GRU based networks achieve better results than LSTM KN95 (N95) mask (red line) and wearing two surgical masks
based networks. This may because our data set is relatively (green line). The results are shown in Fig. 6. It can be seen
small which can’t fill the demand of the LSTM based net- from the experimental results that no matter what kind of
works. GRU based networks require less data than LSTM and mask is worn, or even two masks, the respiratory data can be
perform better result in our respiration condition classification well recognized. This proves the stability of our algorithm and
task. device. However, since different masks have different thermal
To figure out the detailed information about the classifica- insulation capabilities, the average breathing temperature may
tion of the respiratory state, we plotted the confusion matrix vary as the mask changes. To minimize this error, respiratory
of the four models as demonstrated in Fig. 5. As can be seen data are normalized before input into the neural network.

Authorized licensed use limited to: Mapua University. Downloaded on March 31,2021 at 03:47:34 UTC from IEEE Xplore. Restrictions apply.
13680 IEEE SENSORS JOURNAL, VOL. 20, NO. 22, NOVEMBER 15, 2020

Fig. 6. The raw respiratory data obtained through the breathing data Fig. 8. The raw respiratory data obtained while the rotation angle varies
extraction algorithm with different types of masks. from 45 degrees to 0 degrees at a stable speed. The blue line stands for
the rotation in pitch axis and the red line stands for the rotation in yaw
axis.

3) Limitation of Rotations Toward the Camera During Mea-

surement: Considering that breath detection will be applied
in different scenarios, we cannot assure that the testers would
face the device directly with no error angle. Therefore, this
experiment is designed to show the respiratory data extraction
performance when faces rotate in pitch axis and yaw axis.
We define the camera directly towards the face to be 0 degrees,
and design an experiment in which the rotation angle gradually
changed from 45 degrees to 0 degrees. We consider the
transformation of two rotation axes: yaw and pitch, which
respectively represent left and right turning and nodding.
The results in the two cases are quite different as shown in
Fig. 8. Our algorithm and device maintain good results in
Fig. 7. The raw respiratory data which is obtained under the distance yaw rotation, but it is difficult to obtain precise respiratory
between camera and device from 0 to 2 meters. During the 200 frames’ data in pitch rotation. This means participants can turn left
acquisition process, the distance between the tester and the device
varies from 0.1 meters to 2 meters at a stable speed. or turn right during the measurement but can’t nod or head
up a lot since this may impact the current measurement
result.
2) Limitation of Distance to the Camera During Measurement:
To verify the robustness of our algorithm and device in dif- IV. C ONCLUSION
ferent scenarios, we design experiments to collect respiratory In this paper, we propose an abnormal breathing detection
data at different distances. Considering the limitations of method based on a portable dual-mode camera which can
handheld devices, we test the collection of facial respiration record both RGB and thermal videos. In our detection method,
data from a distance of 0 to 2 meters. During one 20 seconds’ we first accomplished an accurate and robust respiratory data
data collection process, the distance between the tester and detection algorithm which can precisely extract breathing data
the device varies from 0 to 2 meters at a stable speed. from people wearing masks. Then, a BiGRU-AT network is
The result is demonstrated in Fig. 7. The signal tends to be applied to work on the screening of respiratory infections.
periodic from the position of 0.1 meters, and it does not In validation experiments, the obtained BiGRU-AT network
lose regularity until about 1.8 meters. At a distance of about achieves a relatively good result with an accuracy of 83.7% on
10 centimeters, the complete face begins to appear in the the real-world dataset. For patients caught respiratory diseases,
camera video. When the distance comes to 1.8 meters, our the sensitivity value is 90.23%. It is foreseeable that among
face detection algorithm begins to fail gradually due to the patients with COVID-19 who have more clinical respiratory
distance and pixel limitation. This experiment verifies that symptoms, this classification method may yield better results.
our algorithm and device can guarantee relatively accurate The result of our experiments provide a theoretical basis for
measurement results in the distance range of 0.1 meters to scanning respiratory diseases through thermal respiratory data,
1.8 meters. In future research, we will improve the effec- which serves as an encouragement for controlled clinical trials
tive distance through improvements in algorithm and device and call for labeled breath data. During the current spread of
precision. COVID-19, our research can work as a pre-scan method for

Authorized licensed use limited to: Mapua University. Downloaded on March 31,2021 at 03:47:34 UTC from IEEE Xplore. Restrictions apply.
JIANG et al.: DETECTION OF RESPIRATORY INFECTIONS USING RGB-INFRARED SENSORS 13681

abnormal breathing in many scenarios such as communities, [16] C. C. Leung, T. H. Lam, and K. K. Cheng, “Mass masking in
campuses and hospitals, contributing to distinguishing the the COVID-19 epidemic: People need guidance,” Lancet, vol. 395,
no. 10228, p. 945, Mar. 2020.
possible cases, and then slowing down the spreading of the [17] L. Wang and A. Wong, “COVID-Net: A tailored deep convolu-
virus. tional neural network design for detection of COVID-19 cases from
In future research, based on ensuring portability, we plan chest X-ray images,” 2020, arXiv:2003.09871. [Online]. Available:
http://arxiv.org/abs/2003.09871
to use a more stable algorithm to minimize the effects caused [18] O. Gozes et al., “Rapid AI development cycle for the coronavirus
by different masks on the measurement of breathing condi- (COVID-19) pandemic: Initial results for automated detection &
tions. Besides, temperature may be taken into consideration to patient monitoring using deep learning CT image analysis,” 2020,
arXiv:2003.05037. [Online]. Available: http://arxiv.org/abs/2003.05037
achieve a higher detection accuracy on respiratory infections. [19] L. Li et al., “Artificial intelligence distinguishes COVID-19 from
community acquired pneumonia on chest CT,” Radiology, Mar. 2020,
Art. no. 200905.
R EFERENCES [20] J. Chauhan, J. Rajasegaran, S. Seneviratne, A. Misra, A. Seneviratne,
[1] M. A. Cretikos, R. Bellomo, K. Hillman, J. Chen, S. Finfer, and and Y. Lee, “Performance characterization of deep learning models for
A. Flabouris, “Respiratory rate: The neglected vital sign,” Med. J. Aust., breathing-based authentication on resource-constrained devices,” Proc.
vol. 188, no. 11, pp. 657–659, Jun. 2008. ACM Interact., Mobile, Wearable Ubiquitous Technol., vol. 2, no. 4,
[2] A. D. Droitcour et al., “Non-contact respiratory rate measurement pp. 1–24, Dec. 2018.
validation for hospitalized patients,” in Proc. Annu. Int. Conf. IEEE Eng. [21] B. Liu et al., “Deep learning versus professional healthcare equipment:
Med. Biol. Soc., Sep. 2009, pp. 4812–4815. A fine-grained breathing rate monitoring model,” Mobile Inf. Syst.,
[3] R. Boulding, R. Stacey, R. Niven, and S. J. Fowler, “Dysfunctional vol. 2018, pp. 1–9, Jan. 2018.
breathing: A review of the literature and proposal for classification,” [22] Q. Zhang, X. Chen, Q. Zhan, T. Yang, and S. Xia, “Respiration-based
Eur. Respiratory Rev., vol. 25, no. 141, pp. 287–294, Sep. 2016. emotion recognition with deep learning,” Comput. Ind., vols. 92–93,
[4] Z. Xu et al., “Pathological findings of COVID-19 associated with acute pp. 84–90, Nov. 2017.
respiratory distress syndrome,” Lancet Respiratory Med., vol. 8, no. 4, [23] U. M. Khan, Z. Kabir, S. A. Hassan, and S. H. Ahmed, “A deep learning
pp. 420–422, Apr. 2020. framework using passive WiFi sensing for respiration monitoring,” in
[5] C. Sohrabi et al., “World health organization declares global emergency: Proc. IEEE Global Commun. Conf. (GLOBECOM), Dec. 2017, pp. 1–6.
A review of the 2019 novel coronavirus (COVID-19),” Int. J. Surg., [24] Y. Cho, N. Bianchi-Berthouze, and S. J. Julier, “DeepBreath: Deep
vol. 76, pp. 71–76, Apr. 2020. learning of breathing patterns for automatic stress recognition using low-
[6] E. J. Chow et al., “Symptom screening at illness onset of health care cost thermal imaging in unconstrained settings,” in Proc. 7th Int. Conf.
personnel with SARS-CoV-2 infection in King County, Washington,” Affect. Comput. Intell. Interact. (ACII), Oct. 2017, pp. 456–463.
JAMA, vol. 323, no. 20, p. 2087, May 2020. [25] H. E. Romero, N. Ma, G. J. Brown, A. V. Beeston, and M. Hasan,
[7] F. Q. Al-Khalidi, R. Saatchi, D. Burke, H. Elphick, and S. Tan, “Deep learning features for robust detection of acoustic events in sleep-
“Respiration rate monitoring methods: A review,” Pediatric Pulmonol., disordered breathing,” in Proc. IEEE Int. Conf. Acoust., Speech Signal
vol. 46, no. 6, pp. 523–529, Jun. 2011. Process. (ICASSP), May 2019, pp. 810–814.
[8] J. Kranjec, S. Beguš, J. Drnovšek, and G. Geršak, “Novel [26] Z. Zhu, J. Fei, and I. Pavlidis, “Tracking human breath in infrared
methods for noncontact heart rate measurement: A feasibility imaging,” in Proc. 5th IEEE Symp. Bioinf. Bioeng. (BIBE), Oct. 2005,
study,” IEEE Trans. Instrum. Meas., vol. 63, no. 4, pp. 838–847, pp. 227–231.
Apr. 2014. [27] C. B. Pereira, X. Yu, M. Czaplik, V. Blazek, B. Venema, and
[9] Y. Wang, M. Hu, Q. Li, X.-P. Zhang, G. Zhai, and N. Yao, S. Leonhardt, “Estimation of breathing rate in thermal imaging videos:
“Abnormal respiratory patterns classifier may contribute to large- A pilot study on healthy human subjects,” J. Clin. Monit. Comput.,
scale screening of people infected with COVID-19 in an accurate vol. 31, no. 6, pp. 1241–1254, Dec. 2017.
and unobtrusive manner,” 2020, arXiv:2002.05534. [Online]. Available: [28] A. Ishida and K. Murakami, “Extraction of nostril regions using period-
http://arxiv.org/abs/2002.05534 ical thermal change for breath monitoring,” in Proc. Int. Workshop Adv.
[10] M.-H. Hu, G.-T. Zhai, D. Li, Y.-Z. Fan, X.-H. Chen, and X.-K. Yang, Image Technol. (IWAIT), Jan. 2018, pp. 1–5.
“Synergetic use of thermal and visible imaging techniques for contact- [29] X. Tang, D. K. Du, Z. He, and J. Liu, “Pyramidbox: A context-assisted
less and unobtrusive breathing measurement,” J. Biomed. Opt., vol. 22, single shot face detector,” in Proc. Eur. Conf. Comput. Vis. (ECCV),
no. 3, Mar. 2017, Art. no. 036006. 2018, pp. 797–813.
[11] M. Hu et al., “Combination of near-infrared and thermal imaging [30] Y. Cho, S. J. Julier, N. Marquardt, and N. Bianchi-Berthouze, “Robust
techniques for the remote and simultaneous measurements of breathing tracking of respiratory rate in high-dynamic range scenes using mobile
and heart rates under sleep situation,” PLoS ONE, vol. 13, no. 1, thermal imaging,” Biomed. Opt. Express, vol. 8, no. 10, pp. 4480–4503,
Jan. 2018, Art. no. e0190466. 2017.
[12] C. B. Pereira, X. Yu, M. Czaplik, R. Rossaint, V. Blazek, and [31] J. L. Elman, “Finding structure in time,” Cognit. Sci., vol. 14, no. 2,
S. Leonhardt, “Remote monitoring of breathing dynamics using infrared pp. 179–211, Mar. 1990.
thermography,” Biomed. Opt. Express, vol. 6, no. 11, pp. 4378–4394, [32] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural
2015. Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
[13] G. F. Lewis, R. G. Gatto, and S. W. Porges, “A novel method [33] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and
for extracting respiration rate and relative tidal volume from J. Schmidhuber, “LSTM: A search space odyssey,” IEEE Trans. Neural
infrared thermography,” Psychophysiology, vol. 48, no. 7, pp. 877–887, Netw. Learn. Syst., vol. 28, no. 10, pp. 2222–2232, Oct. 2017.
Jul. 2011. [34] K. Cho et al., “Learning phrase representations using RNN encoder-
[14] L. Chen, N. Liu, M. Hu, and G. Zhai, “RGB-thermal imaging system decoder for statistical machine translation,” 2014, arXiv:1406.1078.
collaborated with marker tracking for remote breathing rate measure- [Online]. Available: http://arxiv.org/abs/1406.1078
ment,” in Proc. IEEE Vis. Commun. Image Process. (VCIP), Dec. 2019, [35] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by
pp. 1–4. jointly learning to align and translate,” 2014, arXiv:1409.0473. [Online].
[15] S. Feng, C. Shen, N. Xia, W. Song, M. Fan, and B. J. Cowling, “Rational Available: http://arxiv.org/abs/1409.0473
use of face masks in the COVID-19 pandemic,” Lancet Respiratory [36] A. Vaswani et al., “Attention is all you need,” in Proc. Adv. Neural Inf.
Med., vol. 8, no. 5, pp. 434–436, May 2020. Process. Syst., 2017, pp. 5998–6008.

Authorized licensed use limited to: Mapua University. Downloaded on March 31,2021 at 03:47:34 UTC from IEEE Xplore. Restrictions apply.