Proceedings of the 22dAnnual EMBS International Conference, July 23-28,2000, Chicago IL.

Environmental Control by a Brain-Computer Interface

Faisal Karmali**,Mark
Abstract - Locked-in patients, such as those with
amytrophic lateral sclerosis (ALS), lose control of their
bodies, leaving them unable to perform simple tasks such
as speech, locomotion, and the ability to effectively interact
with their environment. Brain-Computer Interfaces (BCI)
show promise in allowing these individuals to interact with
a computer using EEG. A system was created to allow
individuals, via a BCI, to control home appliances and to
have the computer articulate preselected words. It uses
ActiveHome from X10 [l], which can control of a wide
variety of home appliances via a computer, and allows
tasks such as light dimming, TV channel changing, and
turning appliances on and off. The system was evaluated
with a set of tasks to measure its ease-of-use, ease-oflearning, ratesferror, and amount of time the subject
required to complete the task.
I. INTRODUCI~~N
A. Environmental Control
A variety of work has been done for people with severe
disabilities to control their environment. Most require some
neuromuscular control. For example, home automation, often
via X10,has been demonstrated utilizing voice, switch [2] and
eye twitch [3] sensors. Aids such as Intellikey [5] allows
typing on an on-screen keyboard using a variety of switches.

Aleksandar Kostov ,4
electrodes at C3,C4, P3 and P4.The system has one- and twodimensional control, with success rates approaching 100%for
one-dimensional control. It is trained by showing the subject a
cursor and target. The subject thinks of thoughts that they wish
to use to make the cursor move towards the target, and the
system learns to recognize these thoughts and associate them
with directions. A driver has been developed to allow the BCI
to control the Windows mouse cursor.
11. METHODS
A. User Interjace
The user interface of ECBCI, developed under Windows,
consists of four vertical buttons, with the first button usually
labelled Up Menu to return to the previous menu (figure 1).
A scanning algorithm is employed, which uses two BCI states:
scan and stop. When the scan state is detected, each
button is highlighted for a certain period of time before the
system scans to the next button. The entire button is
highlighted to ensure subjects with poor vision are likely to
see it. When the BCI stop state is detected, the scanning
stops, and the current button remains highlighted. If the button
remains highlighted for a specified time, it is selected. A
selected button can either open a submenu, or perform an
action such as activating appliances or playing a sound. The
amount of time before a button is selected is configurable.

Work has also been undertaken for patients who have minimal ECBCI contains a lock-mode, which effectively locks the
or no neuromuscular control. Direct control of a single o d o f f system and prevents the user from accidentally selecting a
switch using EEG alpha rhythms has been achieved [4].
Control of an on-screen keyboard has been achieved using a
single electrode implanted in the motor cortex of the human
brain [6]. Implanted electrodes in primates have controlled
prosthetic devices [8]. One system allows subjects to parse a
binary menu tree, selecting between yes and no answers at
each level using a BCI [91.

menu item when they are not concentrating on the BCI. In

lock-mode, control of the system is blocked until a specific
sequence of four buttons is selected (figure 2). Once the four
buttons are correctly chosen in sequence, the top menu returns.
If an incorrect button is selected while unlocking, the subject
must start again from the first step. Lock-mode is activated by
selecting Up Menu from the top menu.

The system described in this paper, named Environmental

Control-BCI (ECBCI) is controlled by the BCI at the
University of Alberta. It is designed to perform a variety of
tasks in a reduced timeframe compared to other systems.
ECBCI provides four choices at each menu level, including
one choice to move to the previous menu. It is designed to
work with a wide variety of applications, and to add new
applications as they become available.

A sample menu tree was developed based on consultation with

rehabilitation professionals. It encompasses a cross-section of
the tasks an individual may wish to perform. It contains 10
submenus and 19 action buttons (figure 3).

B. Brain-ComputerInterjaces
A BCI is a system that records electrical activity from the
brain and classifies these signals into different states. A
common configuration is directional control, where different
brain states correspond to moving a cursor on the screen left,
right, up and down.
The BCI developed at the University of Alberta [7] utilizes
artificial neural networks to classify EEG signals from
0-7803-6465-1/00/$10.00 02000 IEEE

Figure 1: A submenu of ECBCI

2990

Proceedings of the 22dAnnual EMBS International Conference, July 23-28, 2000, Chicago IL.

Windows Media Player is used to articulate phrases, which are

recording as wave sound files, and to play MP3 songs. Games
developed for the BCI were successfully started from ECBCI.
There is an endless choice of other applications, since any
software that can be executed from the command-line can be
initiated by ECBCI. This results in a flexible architecture.
Sound
Sava

Figure 2: Lock-mode

APplianCeS

A sequence of actions was developed to evaluate the speed

and number of errors made by subjects using ECBCI. It
contains 12 actions requiring a total of 44 button presses. The
actions in the sequence are Hello , How Are You?, Tum
radio on, Set light to high, Tum radio ofi I am hungry, Set
light to low, Tumfan on, I am tired, Tum light ofi Turnfan
ofl and Lock ECBCI.
B. Environmental Control
ECBCI is designed to be flexible enough to accommodate a
wide variety of actions, from controlling home appliances and
prosthetics, to articulating pre-recorded phrases, playing music
and starting games. ECBCI is designed to be distributed across
multiple machines, so that processing of EEG is not hindered
by other tasks, such as sound playback. It is highly
configurable as external software can be executed. This allows
for the easy design of different menu structures. A few modes
of action have been developed and tested (figure 4) to control
home appliances, to articulate phrases by playing sounds and
to start games.
The ActiveHome XI0 Home Automation System [l] is used
to control applicances. In the evaluation configuration there is
a radio and a fan that can be switched on and off, and a lamp
that can be set to various levels of brightness. The X10 system
sends control signals through power-lines using a highfrequency modulated carrier. The transmitter (Radio Shack
#61-2417) connects to the computer serial port. The radio and
fan are controlled by a Universal Appliance Switch (#6l2684B) and the incandescent lamp by a Dimmer Switch (#612682B). The X10 software includes a command-line program
to transmit instructions to appliances, which is used by
ECBCI.

EnvlronmentalControl
Radio

off

on

Grwtlngs

Fan

111. RESULTS
The scanning algorithm used worked extremely well, and the
test subjects were able to carry out various sequences of tasks
quickly and with minimal error.
Preliminary trials were conducted on a healthy test subject
with a sequence containing 12 actions that required 44 button
presses to navigate the menu tree (table 1). None of the errors
made were serious, in that they did not communicate any
messages that would affect the user, nor did they change the
state of any devices the user did not want to affect. Qualitative
tests were also performed with a post-polio subject. He was
able to successfully complete the sequence.

Trial

Time
required
(min:sec)

Mean
Mean
Actions
time per time per erroneously
action
button
selected
5:OO
25.0sec
6.8 sec
2
1
6:05
30.4sec
8.3 sec
1
2
27.7sec
7.6sec
533
Mean
Table 1: Prelimary Results From Subject Trials
Lock-mode did not adequately meet its objective of locking
ECBCI. When a test was performed for accidental unlocking,
the subject, who could not see the screen, unlocked and then
relocked ECBCI in about five minutes. Calculations of the
probability that the sequence could accidentally be selected in
a certain amount of time was made, and confirmed the
observation that lock-mode is not adequate.

Computer Control

Physical Needs

011

off

Hello

Wedlcation

Low

-I

on

HowAreYoti?

Thlw

High

0-7803-6465-11001$10.00 02000 IEEE

- HowstheWeather?

comned via IC10

powa Line nehaolk

Figure 4: ECBCI System Configuration

Communlcatlons

Ugh1

IC10 Home
Autom&ior~S a v a

Hungv

2991

Feelings

*PPY
Tlred

Angry

Play Breakout

YP3s

Song1
song2

Proceedings of the 22dAnnual EMBS International Conference, July 23-28,2000, Chicago IL.

The X10 controller does not always function reliably.

Communication is not possible between certain combinations
of electrical sockets within the lab. This is likely due to the
sockets in the building using a three-phase system and thus
some combinations being out of phase. In addition, some
power bars impede signal transmission, while others do not.
Commands sent from the transmitter are not always received,
especially by a receiver that has just been plugged in and
which has not had time to warm-up. Devices that are told to
turn off sometimes turn off then immediately turn back on.
X10 attributes all of these problems to power line interference.
Despite these issues, the system works satisfactorily if the
correct configuration is found.
IV. DISCUSSION AND CONCLUSIONS
The scanning mode is the most effective method found to
control the user interface. The software has been written with
an open architecture that allows an endless number of existing
and new applications. It has been successfully tested with the
X10 system, with sound files, and to start computer
applications.
Initial subject evaluations show promising results. The subject
was able to select buttons in an average of less than eight
seconds, and actions in an average of less than 30 seconds.
This is comparable to the amount of time it takes to get up
from ones computer, go to the radio, turn it on and return to
the computer. The subject was able to navigate the menus with
few errors. The lock-mode works, but may not provide
adequate security; a longer sequence may be required.
Further testing with multiple subjects is required to confirm
the usefulness of ECBCI, and to find the optimal configuration
that minimizes the time required to perform actions and the
number of errors. Investigation is required into the effects of
cursor velocity, the number of buttons on the screen, the mode
of cursor movement used and the time required to select a
button. The lock-mode also seems inadequate in stopping the
user from unintentionally performing actions. A complete
study of the probability of unlocking the lock-mode needs to
be performed.

[2] Enabling Devices, a division of Toys for Special

Children,
Inc.
Product
Information
(www.enablingdevices.com)
[3] ACS Technologies, Inc. P.O. Box 889, Jackson, MI, USA
49204, phone (800)788-0889
[4] Kirkup, L. et al. (1997) EEG-Based system for rapid onoff switching without prior learning, Medical and
Biological Engineering and Computing, v35n5 Sep 1997,
p504-509.
[SI IntelliTools, Inc., 55 Leveroni Court, Suite 9, Novato,
CA, USA 94949 (www.intellitools.com)
[6] Kennedy, P.R., Adams, K., Bakay, R.A.E., Goldthwaite,
J., Montgomery, G., Moore, M. (1999) Direct Control of
A Computer From the Human Central Nervous System,
Brain-Computer Interface Technology: Theoly and
Practice, First Intemational Meeting. June 16-20, 1999,
Rensselaerville, New York.
[7] Polak, M. And Kostov, A. (1998), Feature Extraction in
Development of Brain-Computer Interface: A Case
Study, Proceedings of the 20h Annual International
Conference of the IEEE Engineering in Medicine and
Biology Society, Volume 20, 1998.
[8] Isaac, R.E. et al. (1999), Real-time Control of a Cortical
Neural
Prosthesis.
Brain-Computer
Interface
Technology: Theory and Practice, First International
Meeting. June 16-20, 1999, Rensselaerville, New York.
[9] Miner L.A., McFarland D.J., Wolpaw J.R. (1998).
Answering questions with an electroencephalogrambased brain-computer interface. Arch. Phys. Med.
Rehabil., vol. 79, pp. 1029-1033.
Author Information
Faculty of Rehabilitation Medicine, University of Alberta,
3-48 Corbett Hall, Edmonton, Alberta, Canada T6G 2G4
Department of Systems Design Engineering, University of
Waterloo, Canada. fkarmali @uwaterloo.ca
Department of Computing Science, University of Alberta,
Canada. mark.polak@ualberta.ca
Glenrose Rehabilitation Hospital, Edmonton, Canada

ECBCI can be improved by adding more applications. One

example is prosthetic devices and simulators. Another is
functionality to work with ones computer, such as navigating
the file system. Home automation using methods other than
X10 can be implemented. Communications is a key field. The
ability to access email would lead to an enormous
enhancement in the lives of locked-in individuals. Similarly,
access to ICQ and other text messaging software could lead to
a world where individuals can send messages to each other via
wireless modems using just their thoughts. The ability to use a
web browser opens up possibilities for entertainment,
communications, and work.

v.

REFERENCES

[l] X10 Ltd. Product Information (accessible through

www.xl0.com)

0-7803-6465-11001$10.00 02000 IEEE

2992

Proceedings of the 22dAnnual EMBS International Conference, July 23-28,2000, Chicago IL.

Using artificial intelligence to assist people with

dementia to be more indeDendent
Alex Mihailidis1.2,Geoff Fernie, Ph.D. , and Joseph C. Barbenel, Ph.D.2
Abstract - A computerized device, which uses
artificial intelligence, is under development to assist
people with severe dementia to complete washroom
activities more independently. Currently, the device
can watch the user, learn from hidher actions,
adapt to individual preferences, and issue varying
levels of cue detail. Further development is required
before clinical trials will begin.

activities [2, 3, 4, 51. These devices were proven to be

efficacious, but none of them were used with people
with dementia, nor was artificial intelligence (AI) used.
AI techniques are algorithms that could be used to
design a computer program that acts more like a human
when it completes various cognitive tasks such as
decision making, or task guidance [6]. Two specific AI
algorithms are applicable to this cueing device design.
An artificial neural network (ANN) learns
Keywords - Cueing, activities of daily living, artificial associations between input data patterns and output
data patterns. An ANN learns these associations by first
intelligence, dementia
learning the correct associations for a set of training data
and then applying what it has learned to inputs that it
may have never seen before. The trained ANN can be
I. INTRODUCTION
used to classify these new data into pre-learned
It is estimated that one out of three people in Canada categories [6]. A plan is a set of tasks that when
over the age of 85 has a moderate to severe dementia executed together correctly accomplish a particular goal.
[ 11. Dementia is an umbrella term that covers several Given a set of tasks, a plan recognition algorithm
different sub-types. The most common form of (PRA) will find a plan that explains the actions of the
dementia is Alzheimers disease [l]. Dementia is user [6]. This plan, and its associated tasks, can be used
characterized by a decline in cognitive function and to guide the user until the goal is met.
memory. This often results in a person not being able
111. DESIGN OBJECTIVE
to complete activities of daily living (ADL) such as
using the toilet, or washing hisher hands, because
The objective is to create a device that makes use of
helshe may become disoriented and confused. The
current solution is for a caregiver or family member, to AI programming techniques to intelligently and
constantly supervise and assist the person using verbal rationally guide a person with dementia through any
prompts. This loss of autonomy and independence often ADL task.
results in the person becoming embarrassed and
IV. DEVICE DESIGN
agitated, and may also result in a strain being placed on
the relationship with hisher caregiver, or family.
The device will consist of a hardware component that
It is hypothesized that independence can be improved
through the use of an intelligent computerized device tracks the actions of the user, and a software component
that provides the reminders needed by the user, and that analyzes these inputs and makes decisions.
Software has been under development for several
monitors hisher progress during an ADL. Thus,
assistance is provided but a caregiver is not required to months, while hardware development has just
be present. This paper describes the progress of the commenced.
There are three main modules in the software-the
development of such a device for people with dementia.
ANN, the PRA, and the Action module (AM). These
three modules were programmed using Matlab v.5.3 [7].
The modules are vector-based, which allows normally
11. BACKGROUND
complex operations to be completed with more ease and
Several computerized cueing devices and memory speed than a non-vectored algorithm. A vector, or a
aids have been developed previously by other particular element of one, defines all events, inputs, and
researchers for tasks which included assisting a person records. For example, all of the verbal cues were
while shaving and guiding a person through vocational assigned cue identification numbers when they were

0-7803-6465-11001$10.00 02000 IEEE

2993

Proceedings of the 22dAnnual EMBS International Conference, July 23-28,2000, Chicago IL.

recorded. When the device needed to determine if a

particular cue had already been played to the user, a
simple search of a vector, whose elements are the
already issued cues, was done.
The actions of the devices modules, progress of the
user, and other parameters are displayed using a
graphical user interface (GUI) that was programmed
using LabVIEW v.5.1. [8]. Figure 1, is an example of
the GUI.

Figure 1 - An example of the devices GUI

The GUI and the data from the elements programmed in

Matlab are lidced together using dynamic data exchange
functions in LabVIEW. These functions allow data to
be easily passed between Windows applications.
The primary objective of the hardware design is to
select, or design, a system that will allow the device to
know what task the user is completing. Various onperson tracking devices, environmental switches, and
optical scanning systems are being tested with the
developed software to determine their efficacy.
V. ARTIFICIAL NEURAL NETWORK (ANN)

After the hardware acquires input, the first module,

the ANN, is invoked. The ANN is a neural network,
which uses probability theory to learn which tasks
correspond with the various inputs from the
environment. The network classifies the inputs into task
identification numbers. A pre-existing algorithm for this
type of ANN with some modifications was used. An
ANN was used because it can accurately classify inputs
that may have never been seen before by the device, and
it allows for easy on-line training of the device for any
ADL.
This type of network is actually a statistical
algorithm. It is based on the Bayes optimal decision

0-7803-6465-1/001$10.00 02000 IEEE

rule, which states that X should be classified into

population i if
hici fi(X) > hj cj fj(X), [9]
where h is the prior probability of an unknown sample
being drawn from a population, c is the associated cost
of misclassification, and f(X) is the probability density
function (PDF) [9]. Normally, the prior probabilities are
treated as being equal, and can be ignored. The same is
true for the costs. This leaves the problem of
determining the PDF, which is typically unknown. The
PDFs of the populations can be estimated using a
training set of data, and Parzens method [lo]. This
method provides a tool for constructing an
asymptotically Bayes optimal classifier even when the
PDFs are unknown. The PDF estimator is a scaled
average of that function across the training set of data
[9]. For a population with sample size n, the estimated
density function is:

where the weighting factor, W, is a Gaussian function.

Using the now known density function, new inputs to
the ANN can be classified into their respective
categories using Bayesian confidence measures
represented by:

VI. PLAN RECOGNTION ALGORITHM (PFL4)

The next module is the PRA. Using the output from
the ANN it attempts to understand the actions of the user
by forming a model of the users actions and goals. The
PRA does this by trying to determine which plan the
user is completing by comparing a vector that keeps
track of the steps already completed, with a pre-existing
database of possible plans that could be executed. This
database contains several row vectors. Each vector
constitutes a plan, and is a different sequence of task
identification numbers. These sequences, if completed
correctly, would lead to the required goal. The first
element of each vector identifies the plan identification
number. This algorithm uses vector searches to find the
step vector within one of the databases plan vectors. If
a match can not be found, the algorithm will attempt to
predict which plan the user is attempting. This is done
by removing the most recently added step to the step
vector one at a time until a match can be found. Once a
plan has been found the module becomes a more typical
planning agent as defined in classical AI. A planning

2994

Proceedings of the 22ndAnnual EMBS international Conference, July 23-28,2000, Chicago IL.

agent comes up with a plan, or a sequence of steps, that

will lead the user to a particular goal [6]. This algorithm
can be better illustrated using the following example:
1. Step Vector = [l] (This represents the task
identification number of the task the user just
completed).
2. Plan Database contains row vectors =

which level of cue detail to issue. This history is

constantly updated to reflect the most recently
successful cueing strategy for the user. Once the cue
detail level is selected, the associated cue is played using
a soundcard that is connected to a speaker hidden in the
user's environment.
VIII. SUMMARY OF DEVICE FEATURES

[l 1 2 3 4 5 1
[22 13451
[3 1 2 3 5 4 1
[421 3541

3.
4.
5.
6.

7.

8.

9.

10.
11.
12.
13.

There are four possible plans that the user could

follow to complete the required five tasks. The first
element of each row vector indicates the plan
identification number.
Compare Step Vector with database row vectors
using a sequential search.
Step Vector found in row vectors 1 and 3.
Therefore, the user is completing either Plan 1 or
Plan 3.
Guide the user through either plan.
Step Vector = [ 1 31 -i.e. the user has now completed
task 3 in addition to task 1.
Compare new Step Vector with Plan 1 and Plan 3
vectors, and with remaining row vectors from the
Plan Database using a sequential search (in case
the user completed a task that changed the plan that
he/she is completing).
The new Step Vector is not contained in any of the
row vectors. The user has made a mistake and is not
completing any of the acceptable plans.
Last task identification number added to the Step
Vector is removed. Step Vector = [ 11.
Compare corrected Step Vector with Plan Database
vectors using a sequential search.
Corrected Step Vector found in row vectors 1 and 3.
Therefore the user must be trying to complete one
of these plans.
Cue the user to complete the next task in either of
these plans-i.e. task 2.
Continue until all tasks in the plan are completed.

The following is a summary of the current primary

device functions:
Monitors and reports the progress of a user
Issues a verbal prompt to a user only when
necessary
Issues varying levels of cue detail for each task
Adapts to individual cueing strategies
Adapts to user's preferences with respect to
varying plans
Easily trained on-line for any ADL.

IX. FUTURE DEVELOPMENT

The next steps in development are to:
Developkelect an accurate, reliable, and
generalizable monitoring system
Add speech recognition capabilities to the
software
Re-design the Ah4 to make it more intelligent
Improve the overall look of the GUI and the userfriendliness of the device.
Acknowledgement
This research is supported by a Ph.D. Training
Award from the Alzheimer Society of Canada.
References
1. McDowell, I(1994). Canadian Study of Health and Aging: Study

VII. ACTION MODULE (AM)

The final module is the AM, which is responsible for
selecting and playing a pre-recorded verbal cue only
when indicated as being necessary by the PRA. An
individualized cue history is used to select the required
level of detail for the issued cue. The cue history is a
file that contains a column vector. Each element of this
vector corresponds to a different task, and indicates

0-7803-6465-1/00/$10.0002000 IEEE

methods and prevalence of dementia. Canadian Medical

Association Journal, 150(6): 889-913.
2. Flannery, M. Rice, D (1997). Using Available Technology for
Reminding. RESNA R o c d i n e s Pittsburgh: 517-519.
3. Kirsch, N.L Levine, S.P., Lajiness, R. (1988). Improving
Functional Performance with Computerized Task Guidance
Systems. ICAART Montreal: 564-567.
4. Napper, S.A. Narayan, S. (1994). Cognitive Orthotic Shell.
Nashville: 423-425.
5. LoPresti, E.F. Friedman, M.B. Hages, D. (1997) Electronic
Vocational Aid for People with Cognitive Disabilities. RESNA
Proceedings Pittsburgh: 514-516.

2995

Proceedings of the 22ndAnnual EMBS International Conference, July 23-28,2000, Chicago IL.

6. Russell, S. Nowig. P (1995).

Approack New Jersey: Prentice Hall.
7. Matlab Technical Computing Software - The Mathworks, Inc:
Natick, MA,(508) 647-7000.
8. Labview v.5.1
National Instruments: Austin, TX, (512) 7940100.
9.Masters, T.(1993). h c t i c a l Neural Network Reciws in C++ New
York. Academic Press.
10. Panen. E. (1962). On Estimation of a Probability Density Function
and Mode. Annals of Mathematical Statistics, 33:. 1065-1076.

Centre for Studies in Aging

Sunnybrook & Women's College HSC
Toronto, Ontario, M4N 3M5, Canada
(416) 480-5858 - tel.
(416) 480-5856 -fax.

'Bioengineering Unit
University of Strathclyde
Glasgow, Scotland G4 O N W
+44 (0) 141 548 3780 -tel.
+44 (0) 141-552 6098 -fax.

0-7803-6465-1/00/$10.00 02000 IEEE

2996