Sree Narayana Gurukulam College of Engineering: Brain Computer Interface
Sree Narayana Gurukulam College of Engineering: Brain Computer Interface
Sree Narayana Gurukulam College of Engineering: Brain Computer Interface
KOLENCHERY
Seminar Report
JYOTHY S MENON
Reg.No. 56430
2005-2006
CERTIFICATE
This is to certifi that the Seminar Report entitled "BRAIN
Prof. Dr. Janahan La1 Head of Department Computer Science & Engg. SNGCE, Kolenchery.
P.S. Smijesh Staff in charge Computer Science & Engg. SNGCE, Kolenchery.
I express my sincere thanks to Dr. (Prof.) P.S Janahanlal Head of the Department for providing me the guidance and facilities for the seminar.
I extend my sincere gratitude to Lecturer P.S Smijesh for his co-
JYOTHY S MENON.
CONTENTS
SLNO
TOPIC
PAGE NO
1. 2.
Abstract.........................................................................................2 Introduction...................................................................................3 2.1. What is a Brain-Computer Interface (BCI)?.......................... 3 2.2. How does the brain work?.................................................... 6 2.3. How can we make it possible?............................................. 7
3.
BCI Architecture............................................................................9 3.1. EEG Acquisition Unit..........................................................0 I 3.2. EEG Signal Processing Unit.............................................. 15 3.2.1. Preprocessing Unit................................................. 16 3.2.2. Feature extraction & classification unit................... 16 3.3. Output Unit......................................................................... 17
4.
BCI Functioning...:........................................................................ 18 BCI Implementation Example....................................................... 25 Factors affecting BCI.................................................................... 27 Key issues in the development & application of BCI systems......28 Practical BCls............................................................................... 29 Conclusion................................................................................... -30 References.................................................................................. 31
5.
6.
7. 8. 9. 10.
SNGCE. Kadayiruppu
1. Abstract
A brain-computer interface (BCI) is literally a direct technological interface between a brain and a computer not requiring any motor output from the user. That is, neural impulses in the brain are intercepted and used to control an electronic device. The brain's electrical output is translated by a computer into physical outputs,.such as moving a cursor on a computer screen. To make the computer understand what the brain intends to communicate necessitates monitoring the brain activity. Among the possible brain monitoring methods, the scalp recorded electroencephalogram (EEG) constitutes an adequate alternative because of its good time resolution, relative simplicity and noninvasiveness. The EEG signals are analyzed and mapped into actions inside the computer rendered environment. A BCI allows a person to communicate with or control the external world without using the brain's normal output pathways of peripheral nerves and muscles. Messages.and commands are expressed not by muscle contractions, but rather by electrophysiological signals from the brain. BCls provide an alternative communication and control option for the severely disabled.There has been great success in using cochlear implants in humans as a treatment for non-congenital deafness. There is also promising research in vision science indicating retinal implants may some day prove to be similarly successful in treating non-congenital blindness.
SNGCE; Kadayiruppu
2. Introduction
This technology offers creating a completely new way of communication offering lots of possibilities. For example, for persons with movements' disabilities, a BCI could help them to command their electric wheelchair without needing somebody else help, giving them more independence. As another example, we can image substitute some damaged nerves by computers intercepting the brain messages and redirecting them to the muscle. For people having to deal with epilepsy crisis, it would be interesting to improve their knowledge of the different processes that stimulate a crisis and try to avoid them, controlling their own brain as another muscle.
Department of Computer Science & Engineering SNGCE, Kadayiruppu
The idea of using our brains to directly control a machine isn't particularly new. As far back as 1967, Edmond Dewan described experiments using subjects wired to an electroencephalograph (EEG), which records and graphs the electrical activity of the brain. With practice, the subjects were able to reduce the amplitude of their brain's alpha rhythms, to transmit Morse code to a teleprinter. Research into the Brain Computer Interface, or BCI, began in earnest in the early 70's, when the United States Department of Defense saw the promise of fighter pilots using their minds to directly control their planes. Given the technology of the time, there was limited success, and the program was cancelled. But the groundwork was laid for a field of research now growing rapidly. A major motivation has been to help patients suffering from conditions such as cerebral palsy, or spinal injuries, which inhibit physical control, but which leave intellectual faculties intact. Over the last decade, great advances have been made. Automatic systems capable of understanding different facets of human communication will be at the heart of human-computer interfaces (HCI) in the near future. An HCI which is built on the guiding principle: "think and make it happen without any physical effort" is called a brain-computer interface (BCI). Indeed, the "think" part of this principle involves the human brain, "make it happen" implies that an executor is needed (here: a computer) and "without any physical effort" means that a direct interface between the brain and the computer is required. To make the computer understand what the brain intends to communicate necessitates monitoring the brain activity. Among the possible brain monitoring methods, the scalp recorded electroencephalogram (EEG) constitutes an
SNGCE, Kadayiruppu
adequate alternative because of its good time resolution, relative simplicity and noninvasiveness when compared to other methods such as: functional magnetic resonance imaging, positron emission tomography, magnetoencephalography and electrocorticogram. Furthermore, there is clear evidence that observable changes in EEG result from performing given mental activities.
The success of this exciting work depends on close and productive interaction of scientists, engineers, and clinicians from many different disciplines and requires recognition and attention to a number of crucial issues. This meeting is designed to foster such interdisciplinary interactions and address these crucial issues, and thereby promote the development of BCI systems of practical value to people with disabilities.
During a particular mental activity (MA), we can observe electric potential variations of the brain active regions. Neurons are generating small electrical variation that summed over a region give a potential variation in the space. These variations can be decomposed in series of electrical maps. This means that a mental activity can be seen as a sequential continuation of brain electrical states.
SNGCE, Kadayiruppu
Based on recordings of the brain electrical activity, we would like to reflect as well as possible the dynamic of the functional states of the brain and identify the states representing a mental task as well has possible in order to be able to recognize later on the same mental activity.
Why not using another more sophisticated way to measure the brain electrical activity? We could for example use the Magnetic Image Resonance
Department of Computer Science & Engineering SNGCE, Kadayiruppu
method (IRM)? Let's remember our first objective. All we would like to do is to identify some pattern in the mental activity of the subject in order to give him an immediate feed back. The IRM are useful to describe the brain form and to study the blood fluctuations but they don't measure the electrical activity of the patient. It is easy to imagine that there is a correlation between the electrical activity and the blood fluctuations in the brain, but the latency (variation in time) is fundamentally different! Another more interesting idea is to use microchips introduced directly to the contact of the brain. It poses some new problems like difficulties of avoiding infections in the human body, or transmitting the data measured at the contact of the brain to the computer (through wireless communication?). Moreover, this kind of solutions will just measure the brain activity in a very close region of the brain. Studying the ecological states of a few trees randomly selected in a forest, is not enough to conclude about the ecological state of the entire forest. So, it would be necessary to increase the number of introduced microchips augmenting at the same time the discomfort of the user. Moreover, integrating chips in the human corps is not yet very well defined in the medicine world and neither in the ethic of the humanity. In conclusion, the EEG systems based on 10120 international positioning, is efficient enough for our objective: detecting some special mental activities.
SNGCE, Kadayiruppu
4
-3.3001 0.5917 -6.0412 -6.1430 -1.1795 -4.1720 -1.0386 electrodes -8.7089 -1.0109 0.4586 -2.0777 -2,1974 1.3035 -1.5028
n u m b at samples t
-1.2996
-2.2864
Fig: Matrix of data To record the electrical activity, we sample the mental activity using 16 electrodes and store them in a matrix (16xNumber of measurements, Figure
4). Each column contains the 16 values collected in the electrodes at a certain
time t. For the analysis of the values, Matlab will be employed, having an important strength in mathematical analysis.
SNGCE, Kadayiruppu
10
3. BCI Architecture
To make the computer understand what the brain intends to communicate necessitates monitoring the brain activity. Among the'possible brain monitoring methods, the scalp recorded electroencephalogram (EEG) constitutes an adequate alternative because of its good time resolution, relative simplicity and noninvasiveness when compared to other methods such as: functional magnetic resonance imaging, positron emission tomography, magnetoencephalography and electrocorticogram. Furthermore, there is clear evidence that observable changes in EEG result from performing given mental activities. In the following, an EEG based BCI will be simply called a BCI. Current BCls use the following EEG signals:
Event Related Potentials (ERPs)
ERPs are transient signals which are characterized by a voltage deviation in the EEG and are caused by external stimuli or cognitive processes triggered by external events. When the user pays attention to a particular stimulus, presented by the BCI an ERP that is time locked with that stimulus appears in her EEG. The changes induced by the ERP in the EEG can be detected by the BCI. Therefore, by focusing her attention to the adequate stimuli, the user can command the BCI. The advantage of an ERP based BCI resides in the fact that little training is necessary for a new user to gain control of the BCI. Nonetheless, the communication is slow since the user must wait for the relevant stimulus presentation.
SNGCE, Kadayiruppu
11
SSVERs are elicited by a visual stimulus that is modulated at a fixed frequency. SSVERs are characterized by an increase in the EEG activity at the stimulus frequency. Through feedback, users learn to voluntarily control their SSVER amplitude, whose variations can be detected by the BCI.
Slow Cortical Potential Shifts (SCPSs)
SCPSs are shifts of cortical voltage, lasting from a few hundred milliseconds up to several seconds. Users can learn to produce slow cortical amplitude shifts in an electrically positive or negative direction for binary control. This skill can be acquired if the users are provided with a feedback on the course of their SCPS production and if they are positively reinforced for correct responses.
Oscillatory sensorimotor activity
The 8-12 Hz and 18-26Hz activities recorded over the motor cortex exhibit noticeable changes during movement, preparation for movement and imagined movement. Indeed, such activities decrease in the hemisphere that is opposite (contralateral) to the movement and increase in the other hemisphere (ipsilateral). The frequency ranges and the magnitude of the changes are user dependent; if trained, a BCI can detect these changes and react according to a previously established protocol.
Spontaneous EEG signals
These signals are recorded during the performance of mental activities other than imagined motor tasks and are not elicited by external stimuli (e.g. mental counting, mental rotation of an object, etc.). The BCI can function with spontaneous signals if the patterns characterizing the corresponding mental activities are learned by the BCI in a training phase.
SNGCE, Kadayiruppu
12
A BCI can be defined as a communication system in which the messages or commands that the user sends to the external world do not pass through the brain's normal output pathways of peripheral nerves and muscles. The user communicates with the BCI by performing mental activities (MAS) that the BCI is able to recognize because of previous training. An unknown MA triggers a neutral answer from the BCI; the nature of this answer depends on the context in which the BCI is used. By convention we denote by MAo an unknown MA. BCI consist of three units, namely EEG acquisition, processing and output.
Digitized signals
I
Acquisition
EEGSrgnai Processing
extraction
n
output
User
Feedback
SNGCE, Kadayiruppu
13
While the user is performing the MAS that control the BCI, hislher EEG signals are acquired, digitized and sent to a computer where the signals are processed and translated into commands which generate actions in the output unit. These actions that can be noticed by the user constitute a feedback that the user exploits so as to modulate her mental activity to make the BCI accomplish her intents. The correspondence between the patterns present in the EEG, resulting from a mental activity, and the actions generated by the output unit is determined using machine learning techniques. Therefore a training phase, in which the user is asked to perform the MASthat will be used to command the BCI, is carried out. The training phase is composed of several sessions and results in models associated with each MA that serve as reference for their respective MA. The models are built through several training sessions in order to take into account different external (environmental) and internal (user related) conditions which can induce variations in the EEG. The MA models resulting from the training phase are validated in a training with feedback (TRWF) phase in which the user is asked to perform the MAS and a feedback, telling her if the MA she just performed was successfully recognized (positive feedback) or not (negative feedback) is provided. This implies simultaneous learning of both, the user (who adapts her EEG depending on the feedback) and the BCI (who updates its MA models). When the TRWF phase is completed the user can start controlling the BCI
by performing the MAS for which the BCI was trained (application phase). The
SNGCE, Kadayiruppu
14
EEG tfWS
The amount of EEG that is analyzed by the BCI to generate an action, the EEG trial duration and the interval between two consecutive actions, the BCI period depend on the type of MAS and on the user. These parameters are determined during the training phase, in which some typical values for these parameters are tested and those values that provide the best results in terms of recognition error are finally selected. The above mentioned process, namely training, training with feedback and application is carried out several times throughout the use of the BCI.
SNGCE, Kadayiruppu
15
EEG signals are measured at the scalp by affixing an array of electrodes positioned according to the 10-20 international system and with reference to digitally linked ears (DLE). DLE referenced voltages are obtained by using the average of voltages at both ear lobes as reference. The ear lobes are selected because they constitute an almost quiet reference, As a matter of fact, they present small influences due to temporal activity.
Fig: International 10-20 system of electrodes-placement. The electrode labels correspond to their position with respect to the brain zones, i.e. Fronto-polar (Fp), Frontal (F), Central (C), Temporal (T), Parietal (P) and Occipital (0). Odd indexes are located in the left hemisphere and even ones in the right hemisphere .If Ve is the voltage at electrode e, VA1 and VA2 the voltages at left and right ear lobes respectively and if we arbitrarily choose VA1 as the physical reference, then the DLE referenced voltage of electrode e is
SNGCE, Kadayiruppu
16
frequency band above 25 Hz, in order to detect such artifacts. Ocular artifacts are identified by tracking the abrupt power changes in the signals recorded at prefrontal electrodes.
3.2.2 Feature extraction and classification unit In the context of BCI, the features used for EEG trial classification mainly result from the time, frequency, and time-frequency analysis. Most of the research groups work in the frequency domain and extract the information characterizing mental activities from the nonparametric and parametric spectral representations of EEG. Also, the joint spectral properties of the EEG components are analyzed in for detecting particular emotional states. The
SNGCE, Kadayiruppu
17
relationship between the time evolutions of the signals coming from different electrodes serves as an indication of motor activities in. Useful information can also be extracted from particular brain configurations that can be interpreted in terms of brain states. The feature extraction and classification unit determines the commands where Ne is the sent by the user to the BCI. An EEG trial X belongs to2 R~~~~~~~~ number of electrodes and Ttrial is the number of samples in the EEG trial. The rows of X X (1)... X (Ne) contain the signals recorded at different electrodes. The goal of feature extraction is to map the EEG trial space, RNe xTtrial into a feature
' space, R ~(Df is the dimensionality of the feature space) which is suitable for the
discrimination of EEG trials resulting from the performance of different MAS. The mapping function is usually determined using the physiological knowledge of the MAS that are used to control the BCI. Common mappings result from timespace, frequency (parametric and nonparametric) and timefrequency analysis of EEG trials.
3.3 Output Unit The commands from the feature extraction unit are sent to the output subsystem. It then generates a "system answer" that constitutes a feedback to the user who can modulate his mental activities so as to produce those EEG patterns that make the BCI accomplish his intents.
SNGCE, Kadayiruppu
18
4.
BCI Functioning
The BCI period is the average time between two consecutive answers and
the EEG trial duration is the duration of EEG that the BCI needs to analyze in order to generate an answer.
We call "neutral state" when nothing happens (the BCI provides a neutral answer), the "active state" when the BCI executes something, the "neutral EEG
-
SNGCE, Kadayiruppu
19
set" as composed of those EEG trials that elicit the neutral answer and the "active EEG setnthe complement of the neutral EEG set. The ideal BCI is a two-state machine whose state changes occur at a rate defined by the BCI period and are determined by a Boolean variable 61 (activation) which becomes true when the BCI detects an element of the active EEG set and false otherwise.
51=o
Neutral
SNGCE, Kadayiruppu
20
We propose a less ideal BCI by introducing a transition state so that the BCI cannot switch from the neutral to the active state immediately. The BCI remains in the transition state as long as a second Boolean variable B2 (confirmation) is false.
We consider a BCI that is commanded by NM MAS, each of them associated with an action. The set of actions that the BCI is able to perform is 'noted as {Ao, All ..., ANMd where A. is the neutral action that corresponds to an unknown MA.
21
MA generator
*xlm)l
b"'
Fig: BCI Model BCI functioning is modeled as depicted in the figure. Each BCI period the MA generator (which is composed of the user's brain, the acquisition device and the feature extractor) produces a vectorex (m) drawn from a probability density function (PDF) p(x (m)) which depends on the action at time (m - I), A(m conditions) and internal (user's state of mind). The vector x(m) is then processed by the recognition algorithms whose result is the vector L(m) containing NMA scores that measure the likelihood that x(m) corresponds to each MA. The taken action can be noticed by the MA generator who adapts its behavior consequently (feedback). Additionally, the current action A(m) depends on L(m) and the last T actions A(m-1), ...,A(m-T), this dependency is presented in section 4.2. In order to produce the desired actions, the MA generator should adapt its PDF through reinforcement learning. While external factors can be controlled, internal ones are harder to control even by the user. It becomes then necessary
- 1)
and on extra-system factors (Ext) that can be external (noise and environmental
SNGCE, Kadayiruppu
22
to introduce a mechanism that allows the recognition algorithms to adapt to the user's current PDF; this mechanism is presented in Section 4.1.
The parameters of recognition algorithms need to be flexible in order to dynamically characterize the PDF associated with each MA under different extrasystem conditions. The characterization process is carried out in several training sessions and it is continuously updated under the assumption that the extrasystem conditions remain constant during each session.
Feature vectar belonging to the MAq class
MAq Cue
,--n\H kmv
0
MA generator
Fig: Training without feedback (open loop) The first characterization of MAq that is denoted by OMAq (0) is obtained after training without feedback session (open loop) in which the user is asked, by a visual cue, to perform the MAq while the produced feature vectors are stored to compute OMAq (0). Subsequent characterizations, OMAq (k) for k > 0, are obtained after training with feedback sessions (closed loop), in which the user is asked to perform MAq and the system uses the last characterization to tell the user if the current MA was successfully identified.
SNGCE, Kadayiruppu
23
*ns(
k)
dwq!ILLl j
Fig: Training with feedback (closed loop) An effective way to integrate the current characterization with the precedent one consists in considering OMAq as a set composed of those vectors that define the boundaries of the PDFq.
4.2 Action generation
The action generator is a state machine whose (NMA + 1) states correspond to the NMA actions and the neutral one. In the ideal scheme the action at time m, depends only on L (m) .Since the recognition of MAS is not perfect; the action generator is not allowed to abruptly change its state. Instead, a dynamic transition mechanism is introduced that depends on the recognition error. As the user performance increases, i.e. the recognition error rate decreases, the mechanism converges to the ideal scheme. Consider that at time m the BCI is executing the action Aql and, according to the ideal scheme criteria A (m + I) A@. The transition towards As = is allowed if, under the ideal scheme criteria, A (m+l) = A (m+2) = ... = A (m+~q2ql) A@ ,otherwise the BCI keeps executing Aql. =
SNGCE, Kadayiruppu
24
log E
1%
~q2ql
where pq2ql is the probability that a vector x, generated during the performance of MAql was recognized as belonging to the MAQ class and is a constant that characterizes the degree of confidence on the transition from A to A@,e.g. if ,, 99% confidence is required then, the value of is set to 0.01.
-q2ql
t
~n4l-Q
1 1
of confusion between MA@and MAq1the shorter the transition duration. Therefore, as the user performance improves, the transition scheme evolves towards the ideal scheme.
SNGCE, Kadayiruppu
25
Three types of MAS were used: imagination of left (MAI) or right (MA2) index finger movements and a baseline (MA3) where the subject can imagine anything except MA1 or MA2. The goal was to allow the user to control the movement, to the left or to the right, of a cursor on a computer screen. Our BCI implementation can be explained in terms of three states: the
neutral state in which the BCI whether recognizes MA3 or cannot recognize any
known MA, the active state in which the BCI recognizes MA1 or MA2 and answers with an action and a transition state between the above mentioned states. State changes occur at a rate defined by the BCI period, and are determined by the activation of two Boolean variables: detection (BI) and confirmation (82). 61 is true when MA1 or MA2 are recognized and B2 is true when B1 is true and if the M previously recognized MAS are equal to the currently recognized MA.
SNGCE, Kadayiruppu
26
The parameter M (latency time) depends on the rate of false positive recognitions, although it would not be larger than two seconds. The set of BCI parameters are summarized in the following table. BCI period EEG trial duration Classification parameters Latency time
The optimal values for these parameters are determined during several training sessions and continuously updated.
- -
SNGCE, Kadayiruppu
27
First, when a new user accesses the BCI, it adapts to that user's signal characteristics.
Second, periodic adjustments are necessary to reduce the impact of EEG variations that results from different user's mind states.
Third, the adaptive capacity of the brain is engaged in the sense that, through feedback the brain activity will modify so as to produce those EEG patterns that best control the BCI.
SNGCE, Kadayiruppu
28
29
8. Practical BCls
.There has been great success in using cochlear implants (Cochlear implants are hearing devices that can help people with certain kinds of hearing impairment or who are entirely deaf.)in humans as a treatment for non-congenital deafness. There is also promising research in vision science indicating retinal implants may some day prove to be similarly successful in treating noncongenital blindness. Simple brain-computer interfaces already exist in the form of
neuroprosthetics, with a great deal of neuroscience, robotics, and computer science research currently dedicated to furthering these technologies. Recent achievements demonstrate that it is currently possible to implement brain-computer interfaces that allow neuronal clusters to directly control computers. Current BCls are mainly designed for people with severe motor disabilities in order to provide them with new ways of communication which do not depend on the brain's normal output channels of peripheral nerves and muscles. Present-day BCls provide maximum information transfer rates up to 25bitslmin. Furthermore, BCls can be used as a complement to other HCI devices to enrich the interaction between humans and computers.
Department of Computer Science & Engineering SNGCE, Kadayiruppu
30
9. Conclusion
Over the past decade, productive BCI research programs have arisen. Facilitated and encouraged by new understanding of brain function, by the advent of powerful low-cost computer equipment, and by growing recognition of the needs and potentials of people with disabilities, these programs concentrate on developing new communication & control technology for those with severe neuro - muscular disorders. The immediate goal is to provide these users, who may be completely paralyzed or "locked in," with basic communication capabilities so that they can express their wishes to caregivers, operate simple word processing programs etc. Future progress depends on attention to the key issues listed previously. With adequate recognition and effective engagement of these issues, BCI systems could provide an important new communication and control option for those with motor disabilities. They might also give to those without disabilities, a supplementary control channel or a control channel useful in special circumstances.
SNGCE, Kadayiruppu
31
9. References
http://www.cs.man.ac.uk/-toby/writing/PCW/bci.html http://en.wikipedia.orglwikilBrain-computer interface
http://ltsl pcl9.epfl.ch/repositorv/Garcia2004 1464.pdf http://ltsl pcl9.epfl.ch/re~ositorv/Garcia2003 438.pdf http://ltsl pcl9.epfl.ch/repositon//Garcia2003 436.pdf Many more on the web ................................................
SNGCE, Kadayinrppu