An Affordable Bio-Sensing and Activity Tagging

Platform for HCI Research

Siddharth1,2, Aashish Patel1, Tzyy-Ping Jung2, and Terrence J. Sejnowski2,3

Department of Electrical and Computer Engineering, University

of California, San Diego
Institute for Neural Computation, University of California,
San Diego
The Computational Neurobiology Laboratory, Salk Institute

{ssiddhar@eng.ucsd.edu, anp054@eng.ucsd.edu, jung@sccn.ucsd.edu,

terry@salk.edu}

Abstract.
We present a novel multi-modal bio-sensing platform capable of integrating
multiple data streams for use in real-time applications. The system is composed
of a central compute module and a companion headset. The compute node col-
lects, time-stamps and transmits the data while also providing an interface for a
wide range of sensors including electroencephalogram, photoplethysmogram,
electrocardiogram, and eye gaze among others. The companion headset con-
tains the gaze tracking cameras. By integrating many of the measurements sys-
tems into an accessible package, we are able to explore previously unanswera-
ble questions ranging from open-environment interactions to emotional-
response studies. Though some of the integrated sensors are designed from the
ground-up to fit into a compact form factor, we validate the accuracy of the sen-
sors and find that they perform similarly to, and in some cases better than, al-
ternatives.

Keywords: bio-sensing, multi-modal bio-sensing, emotion studies, brain-

computer interfaces

1 Introduction

Electroencephalogram (EEG) systems have experienced a renewed interest by the

research community for use in non-clinical studies. Though being deployed in large-
scale studies, many of the advances have not been translated to substantial real-world
applications. A major challenge is that the hardware and software typically used to
make measurements limit their use to controlled environments. Additionally, the low
spatial resolution of EEG itself limits the amount of usable information that can be
extracted from noise in dynamic recording environments. Lastly, the absence of a
method to automatically extract user-environment interactions for tagging with EEG
data introduces an immense overhead to researchers - having to manually tag events
or limit experimental design by requiring the subjects to provide information during
the experiments.
Most of the EEG research from the past decades has been conducted under labora-
tory based controlled environments as opposed to practical daily-use applications. On
the other hand, there are many fitness trackers available today capable of providing
accurate heart-rate, blood pressure, galvanic skin response (GSR), steps taken etc.
Under controlled laboratory conditions, EEG researchers have been able to control a
quadcopter [1], control robots [2], control wheelchair to move around [3] etc. Unfor-
tunately, research labs have been unable to show applications of EEG “into the wild”
due to constraints imposed by the existing EEG decoders [17].
EEG research often studies event-related brain responses evoked or elicited by a
visual or auditory stimulus. But, for real-world experiments with EEG, the stimulus
onset is not measured or is ill-defined. A solution is to use saccadic eye movements
and fixations as the time-locking mechanism for analyzing naturalistic visual stimuli
[26, 27, 28]. Hence, we need to simultaneously record and synchronize EEG and eye-
gaze data in real-world neuroimaging studies. For real-world experiments with EEG
there is also a need to pinpoint the stimulus that is causing the changes in EEG.
Hence, user’s visual perspective is necessary to be recorded for EEG recordings in
real-world experiments.
For the analysis of emotional responses, recent research [8, 10] (in NeuroCardiolo-
gy) has shown that the heart also has a role to play in generation of emotions. This
falsifies the wide ranging decades old belief that the brain is solely responsible for the
generation and subsequent emotional feelings. But, there is no currently available
system which can reliably sense and record EEG and electrocardiogram (ECG) to-
gether in a mobile environment. Furthermore, ECG complicates the experimental
setup since subjects have to wear a belt or place several sensors on their chest. A
workaround is to use photoplethsmogram (PPG) from commercially available devices
that derive PPG from the wrist. But, such devices usually use low sampling rates to
save battery power and hence can only measure heart rates, but not heart-rate variabil-
ity (HRV) that is typically only estimated by commercial devices.
Addressing the above key limitations of existing systems, we present an affordable,
wearable multi-modal bio-sensing platform that is capable of monitoring EEG, PPG,
eye-gaze, and limb dynamics (Fig. 1). The platform also supports the addition of other
biosensors including galvanic skin response (GSR) and lactate levels. Leveraging the
capabilities of this system, a new breadth of applications can be explored that allow
for better translations to impactful solutions.
 Fig. 1. Portable multi-modal bio-sensing platform paired with an Emotiv Epoc for EEG, PPG
behind ear and eye-gaze collection.

2 System Overview

We use modular design to increase the flexibility and efficiency of multiple meas-
urements of the multi-modal bio-sensing platform. Selecting a control board that is
well supported by the open-source community and had capable expansion was a pri-
ority. To this end, this study has explored different solutions including the Arduino,
Raspberry Pi, LeMaker Guitar, and other ARM-based embedded controllers. The
hardware evaluation metric that determined viability was the ability for the systems to
hit lower-bound frame-rates and collect data from multiple sensors in real-time using
the Lab-Streaming Layer (LSL [23]). The last but one of the most important evalua-
tion metrics was the expandability via general input/output or communication proto-
cols. After evaluation of the different platforms, the Raspberry Pi 3 (RPi3) was identi-
fied as being the system that best balances cost, support, and capabilities. The sensors
that were selected for preliminary use are explored in detail below.

2.1 Electroencephalogram (EEG)

Non-invasive EEG is used to collect neural signals from individuals. Any EEG
system that is supported by LSL can be used in the proposed multi-modal bio-sensing
framework. The Emotiv Epoc+ system is shown in Fig. 1 as it has a suitable tradeoff
between ease-of-use and performance. The Epoc allows for wireless collection of data
that can be time-stamped and synchronized in real-time by RPi 3. The sampling fre-
quency of the system is on the lower end of new commercial systems at 128Hz, but
has 14 channels (saline activated) and a gyroscope allowing for collection of cleaner
signals. Independent Component Analysis (ICA) [5, 6, 21] is used in real time using
ORICA [20, 24] toolbox in Matlab to separate the sources of EEG recordings in real-
time and plot them. For each of the independent components, the scalp map is plotted
in real-time to better depict the source localization. ICA is also used to remove EEG
artifacts due to eye blinks, muscles and other movements.

2.2 Photoplethsmogram (PPG)

Due to the uncomfortable nature of existing heart-rate and heart-rate variability
sensors, a new miniaturized PPG sensor (Fig. 2) was developed that magnetically
clipped to the ear. The miniaturization was achieved by integrating a high-precision
and high-sampling rate ADC to the sensor. Additionally, to eliminate noise, a third-
order filter (bandpass 0.8 – 4 Hz) was also integrated on the board such that only the
digitized and filtered signals are transmitted to the control board. To also account for
motion artifacts in the heart-rate signals, a 3-axis accelerometer was integrated into
the board. The two data streams, once collected by the core controller, are integrated
using an adaptive noise cancellation algorithm (ANC) [7, 18] (Fig. 3). Addressing the
discomfort and bulk associated with existing systems, the device was developed to be
mountable to the ear-lobe using magnets [7]. Because the system is low-profile and
capable of resting behind the ear [9, 16], more mobile studies can be conducted with-
out the constrained natures of existing systems.

Fig. 2. Miniaturized PPG sensor with scale reference. (A) 3-axis accelerometer, (B) 100 Hz 12-
bit ADC, (C) IR emitter and receiver, (D) third-order filter bank.
 Fig. 3. Schematic overview of adaptive noise cancellation integration with PPG.

2.3 Eye Gaze

The next sensor of the multi-modal system is a pair of cameras. One camera, an IR
emitting device, is capable of accurately capturing the pupil location. A pupil-
centering algorithm is also integrated into the platform and is capable of maintaining
the exact location even under perturbation. An algorithm developed by Pupil Labs [4]
for pupil detection and eye-gaze calibration is utilized. Refer to the results section for
quantification of tracking accuracy.
The second integrated camera in the system is a world-view camera. The camera
provides a wide-angle view of what the wearer is seeing. While being small and inte-
grated into the headset, the camera itself is a standard easily-accessible module. With
the information that is retrievable from both the pupil and the world cameras, it is
possible to retrospectively reconstruct the full-view that the user was observing. The
primary problem that stems from this type of mass video collection is that the amount
of data that must be manually labelled is enormous. There are machine-learning tools
that are capable of labelling video post-hoc, but limit the types of experiments that
can be performed. To create a truly portable system, the system’s video can be
streamed to a computer and processed using deep-learning libraries such as You Only
Look Once (YOLO) [19] that are capable of labelling 20 objects in real-time (trained
on Pascal VOC [20] dataset). By labelling exactly what the user is looking at and
allowing labelled data to be accessible during the experiment, the experimental rigidi-
ty can be relaxed allowing for more natural free-flowing behavior to be measured
with minimally intrusive cues (Fig. 4).
 Fig. 4. Pupil and world views from companion headset device (top-left). Deep-learning pack-
age used to classify objects in real-time (top-right). EEG with real-time ICA and PPG signals
capture (bottom panels).

2.4 Galvanic Skin Response (GSR)

The final sensor considered for addition to our multi-modal setup is a galvanic skin
response sensor. GSR specifically allows for the measurement of arousal through the
measurement of the skin’s impedance. This sensor is unique in that the efficacy of a
third party commercial product being integrated into this research platform needed to
be explored. The GSR sensor that was selected for use was the Microsoft Band 2 [14].

3 Evaluation

The proposed device addresses many of the limitations of existing systems while
providing the measurement capabilities in a form-factor that is convenient for both
researchers and subjects. To evaluate the efficacy of the system, the individual com-
ponents that were created in this study were evaluated. In particular, the evaluations
of the Emotiv Epoc and Microsoft Band are not explicitly evaluated in this review.
The novel PPG and eye-gaze tracking systems will be evaluated for effectiveness in
their respective areas.

3.1 PPG Evaluation

To quantify the performance of the miniaturized PPG sensor, different scenarios
are considered that are representative of real-world uses. The baseline system for
comparison is an EEG/ECG collection system from the Institute of Neural Engineer-
ing of Tsinghua University, Beijing, China. It is capable of measuring EEG/ECG at
1,000 Hz. Because the reference system takes measurements from electrodes placed
near the heart, the artifacts introduced from movements or other physiological re-
sponses are minimized. Simultaneously while taking measurements from the refer-
ence system, the PPG is collecting the ECG signal from the user’s ear at a rate of 100
Hz. As both systems can be connected in parallel, they are synchronized using the
lab-streaming layer [23] and analyzed post-hoc.
The first experiment was a resting scenario - the user remained seated for a fixed
period of two minutes. For the PPG sensor, the data were compared to the reference
with and without the adaptive noise cancellation filter. The second experiment was an
active scenario where the user was instructed to walk in-place at a normal pace to
simulate an active walking scenario. Again the data after using adaptive noise cancel-
lation was compared against the standard raw PPG signal.
A peak detection algorithm [25] using minimum distance to next peak as one of the
parameters to eliminate false peaks was used to calculate Heart Rate (HR) from ECG
and PPG Data. Fifteen-second trails were used to calculate the HR using the peak-
detection algorithm. Figures 5 & 6 show the normalized errors, the ratio of the differ-
ence in HR between PPG and ECG-based methods divided by the mean HR obtained
by PPG. A perfect HR estimation should result in 0%. Examining the results from the
reference signal, the ANC enabled, and ANC disabled signals, it is clear that the ANC
enabled signals have the least amount of noise and most closely match the reference
signal. For resting, the ANC-disabled signals were nearly undistinguishable from the
ANC-enabled signals (Fig. 5). It is in active environments that having the ANC filter-
ing provide a marked improvement in noise rejection (Fig. 6).
 Fig. 5. Bland-Altman plot comparing the measured PPG signal to a reference while at rest
(top). Similarly comparing the measured PPG signal using an adaptive noise cancellation filter
to reference while at rest (bottom).
Fig. 6. Bland-Altman plot comparing the measured PPG signal while walking (top). Similarly
comparing the measured PPG signal using an adaptive noise cancellation filter while walking
(bottom).
3.2 Eye Gaze Evaluation
The performance of the paired pupil- and world- view cameras was evaluated using
a structured visual task to measure precision and accuracy during use. The user sat 2-
2.5 feet away from a computer monitor such that the world camera was >90% of the
camera view was composed of the task screen. Both cameras were streamed at 30fps.
For the first task, the participant was instructed to fix their head movement and only
move their eyes to gaze at static targets that appeared on the screen. A calibration step
where 9 targets appeared in a regular fashion on the screen calibrated the user’s gaze
marker. Immediately following the calibration process, a series of 20 unique targets
are collected in distributed manner across the full screen accounting for the majority
of the field of view. This process was followed by a period of 30 seconds of rest
where the user was asked to move their head around without removing the headset.
This action was designed to simulate the active movement scenarios when wearing
the headset. Next, the participant is instructed to return to a preferred position and
maintain head position. Twenty new unique points are shown on the screen to meas-
ure the precision and accuracy of the eye-tracking system after active use. This pro-
cess was repeated for a total of three trials per subject.
Examining the results for the patients, we are able to observe that the accuracy and
precision of the eye gaze setup does not drift significantly from the expected output.
The accuracy is measured as the average angular offset (distance in degrees of the
visual angle) between fixation locations and the corresponding fixation targets (Fig.
7). The precision is measured as the root-mean-square of the angular distance (degree
of visual angle) between successive samples during a fixation (Fig 8.). Compared to
literature, the gaze accuracy drift of 0.42 degrees is significantly less than the 1-2
degree drift found in commercial systems [11, 12]. The precision, on the other hand,
experiences only a 0.2 degree shift post movement, indicating a minimal angular dis-
tance shift.
Fig. 7. Gaze accuracy analysis comparing the mean after calibration (red) and after 30 seconds
of dynamic head movement to simulate active conditions (blue).

Fig. 8. Angular precision analysis comparing the mean after calibration (red) and after 30 sec-
onds of dynamic head movement to simulate active conditions (blue).
4 Conclusion

There are numerous sensors capable of measuring useful metrics for human
behavior and interactions, however, limitations in the collection hardware and soft-
ware hinder their use in experiments spanning multiple modalities. By developing a
low-cost, portable, multi-modal bio-sensing platform that is capable of interfacing
with numerous different sensors, we are able to explore richer experimental questions
that have previously been unable to be accessed due to the constrained nature of the
measurement hardware. In particular, the modular nature of the control board, inter-
face software, and headset, time can be better spent looking for novel research in-
sights rather than wrangling devices and software packages from different manufac-
turers.

5 References
1. LaFleur, Karl, et al. "Quadcopter control in three-dimensional space using a
noninvasive motor imagery-based brain–computer interface." Journal of neu-
ral engineering 10.4 (2013): 046003.
2. Bell, Christian J., et al. "Control of a humanoid robot by a noninvasive
brain–computer interface in humans." Journal of neural engineering 5.2
(2008): 214.
3. Carlson, Tom, and Jose del R. Millan. "Brain-controlled wheelchairs: a ro-
botic architecture." IEEE Robotics & Automation Magazine 20.1 (2013): 65-
73.
4. Kassner, Moritz, William Patera, and Andreas Bulling. "Pupil: An Open
Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interac-
tion.(April 2014)." CoRR abs 1405.0006 (2014).
5. Makeig, Scott, et al. "Independent component analysis of electroencephalo-
graphic data." Advances in neural information processing systems (1996):
145-151.
6. Makeig, Scott, et al. "Blind separation of auditory event-related brain re-
sponses into independent components." Proceedings of the National Acade-
my of Sciences 94.20 (1997): 10979-10984.
7. Poh, Ming-Zher, Nicholas C. Swenson, and Rosalind W. Picard. "Motion-
tolerant magnetic earring sensor and wireless earpiece for wearable photo-
plethysmography." IEEE Transactions on Information Technology in Bio-
medicine 14.3 (2010): 786-794.
8. Van der Wall, E. E., and W. H. Van Gilst. "Neurocardiology: close interac-
tion between heart and brain." Neth Heart J 21.2 (2013): 51-52.
9. Patterson, James AC, Douglas C. McIlwraith, and Guang-Zhong Yang. "A
flexible, low noise reflective PPG sensor platform for ear-worn heart rate
monitoring." Wearable and Implantable Body Sensor Networks, 2009. BSN
2009. Sixth International Workshop on. IEEE, 2009.
10. Samuels, Martin A. "The brain–heart connection." Circulation 116.1 (2007).
11. Morgante, James D., Rahman Zolfaghari, and Scott P. Johnson. "A critical
test of temporal and spatial accuracy of the Tobii T60XL eye tracker." Infan-
cy 17.1 (2012): 9-32.
12. Hansen, Dan Witzner, and Qiang Ji. "In the eye of the beholder: A survey of
models for eyes and gaze." IEEE transactions on pattern analysis and ma-
chine intelligence 32.3 (2010): 478-500.
13. Notch Motion Tracking System (https://wearnotch.com/)
14. Microsoft Band (https://www.microsoft.com/microsoft-band/)
15. Da He, David, Eric S. Winokur, and Charles G. Sodini. "A continuous,
wearable, and wireless heart monitor using head ballistocardiogram (BCG)
and head electrocardiogram (ECG)." Engineering in Medicine and Biology
Society, EMBC, 2011 Annual International Conference of the IEEE, 2011.
16. He, David Da. A wearable heart monitor at the ear using ballistocardiogram
(BCG) and electrocardiogram (ECG) with a nanowatt ECG heartbeat detec-
tion circuit. Diss. Massachusetts Institute of Technology, 2013.
17. Vaughan, Theresa M., Jonathan R. Wolpaw, and Emanuel Donchin. "EEG-
based communication: Prospects and problems." IEEE transactions on reha-
bilitation engineering 4.4 (1996): 425-430.
18. Widrow, Bernard, et al. "Adaptive noise cancelling: Principles and applica-
tions." Proceedings of the IEEE 63.12 (1975): 1692-1716.
19. Redmon, Joseph, et al. "You only look once: Unified, real-time object detec-
tion." Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition. 2016.
20. Everingham, Mark, et al. "The pascal visual object classes (voc) challenge."
International journal of computer vision 88.2 (2010): 303-338.
21. Bell, Anthony J., and Terrence J. Sejnowski. "An information-maximization
approach to blind separation and blind deconvolution." Neural computation
7.6 (1995): 1129-1159.
22. Kothe, Christian Andreas, and Scott Makeig. "BCILAB: a platform for
brain–computer interface development." Journal of neural engineering 10.5
(2013): 056014.
23. Kothe, C. "Lab streaming layer (LSL)." https://github.
com/sccn/labstreaminglayer. Accessed in 2015.
24. Hsu, Sheng-Hsiou, et al. "Online recursive independent component analysis
for real-time source separation of high-density EEG." Engineering in Medi-
cine and Biology Society (EMBC), 2014 36th Annual International Confer-
ence of the IEEE. IEEE, 2014.
25. Matlab Signal Processing Toolbox (www.mathworks.com/help/sig nal/)
26. Devillez, Hélène, Nathalie Guyader, and Anne Guérin-Dugué. "An eye fixa-
tion–related potentials analysis of the P300 potential for fixations onto a tar-
get object when exploring natural scenes." Journal of vision 15.13 (2015).
27. Kamienkowski, Juan E., et al. "Fixation-related potentials in visual search: A
combined EEG and eye tracking study Fixation-related potentials in visual
search." Journal of vision 12.7 (2012): 4-4.
28. Acqualagna, Laura, and Benjamin Blankertz. "Gaze-independent BCI-
spelling using rapid serial visual presentation (RSVP)." Clinical Neurophysi-
ology 124.5 (2013): 901-908.