Us 9495684

USOO9495684B2
(12) United States Patent (10) Patent No.: US 9,495.684 B2

Jung et al. (45) Date of Patent: *Nov. 15, 2016
(54) METHODS AND SYSTEMS FOR A6IB5/0484 (2006.01)
INDICATING BEHAVOR IN A POPULATION A6IB 5/16 (2006.01)
COHORT A61 B 5/04 (2006.01)
A61 B 5/053 (2006.01)
(75) Inventors: Edward K. Y. Jung, Bellevue, WA A61 B 5/05.5 (2006.01)
(US); Eric C. Leuthardt, St. Louis, A61 B 5/OO (2006.01)
MO (US); Royce A. Levien, Lexington, A61 B 5/145.5 (2006.01)
MA (US); Robert W. Lord, Seattle, (52) U.S. Cl.
WA (US); Mark A. Malamud, Seattle, CPC ........... G06Q 30/02 (2013.01); A61 B 5/04842
WA (US); WA
Bell John(US):
D. Rinaldo,
Lowell L.Jr.,Wood no.386".52%.S.
.UT), .UT),
"...N. W se 000, A61 B 5/055 (2013.01); A61 B 5/0533
es s (2013.01); A61 B 5/14553 (2013.01); A61 B
(73) Assignee: The Invention Science Fund I, LLC, 5/167 (2013.01); A61 B 5/441 (29.99
Bellevue, WA (US) ( .01)
(58) Field of Classification Search
(c ) Notice: Subject tO any disclaimer, the term of this USPC - - - - - -. . . . . . . . . . . . . . . . . . . . . . . . . . . . . - - - 7O6/46
patent is extended or adjusted under 35 See application file for complete search history.
U.S.C. 154(b) by 974 days. (56) References Cited
This patent is Subject to a terminal dis- U.S. PATENT DOCUMENTS
claimer.
5,243,517 A 9, 1993 Schmidt et al.
(21) Appl. No.: 13/439,733 5,339,826 A 8, 1994 Schmidt et al.
Continued
(22) Filed: Apr. 4, 2012 ( )
FOREIGN PATENT DOCUMENTS
(65) Prior Publication Data
US 2012/0316793 A1 Dec. 13, 2012 WO WO 2008/063527 A2 5, 2008
OTHER PUBLICATIONS
Related U.S. Application Data
U.S. Appl. No. 12/215,192. J tal.
(63) Continuation of application No. 12/217.288, filed on ppl. No Contin ald
Jul. 1, 2008, now Pat. No. 8,615,479, which is a (Continued)
continuation-in-part of application No. 12/002.289, Pri Examiner — Michael B Hol
filed on Dec. 13, 2007, and a continuation-in-part of rimary Examiner 1CaC OS
application No. 12/002,778, filed on Dec. 17, 2007, (57) ABSTRACT
and a continuation-in-part of application No. Avatars, methods, apparatuses, computer program products,
Continued devices and systems are described that carry out identifying
(Continued) a member of a population cohort; and indicating at least one
(51) Int. Cl behavior in the member of the population cohort based on an
Goor iz/00 (2006.01) association between the population cohort and at least one
G06N 5/02 (2006.015 cohort-linked avatar.
G06O 30/02 (2012.01) 30 Claims, 31 Drawing Sheets
US 9,495.684 B2
Page 2
Related U.S. Application Data 2003/0177187 A1 9, 2003 Levine et al.

2004.0049490 A1 3, 2004 Milov
12/005,046, filed on Dec. 20, 2007, now Pat. No. 2004/0075677 A1 4/2004 Loyall et al.
8,195.593, and a continuation-in-part of application 2004/013 1998 A1 7/2004 Marom et al.
No. 12/005,063, filed on Dec. 20, 2007, now aban- 2004/O199923 A1 10, 2004 Russek
doned, and a continuation-in-part of application No. 2004/0210634 A1 10, 2004 Ferrer et al.
2004/0221224 A1 11/2004 Blattner et al.
12005,114, filed on Dec. 20, 2007, now abandoned, 2005/OO 10637 A1 1/2005 Dempski et al.
and a continuation-in-part of application No. 12/005, 2005/O137015 A1 6/2005 Rogers et al.
115, filed on Dec. 20, 2007, now Pat. No. 8,069,125, 2005/0171955 A1 8/2005 Hullet al.
and a continuation-in-part of application No. 12/006, 2005/020661.0 A1 9, 2005 Cordelli
234, filed on Dec. 31, 2007, and a continuation-in- 2005/0216243 A1 9, 2005 Graham et al.
2005/0223328 A1 10, 2005 Ashtekar et al.
part of application No. 12/006,235, filed on Dec. 31, 2005/0273017 A1 12/2005 Gordon
2007, now Pat. No. 8,356,004, and a continuation- 2005/0283054 A1 12/2005 Reiman
in-part of application No. 12/006,239, filed on Dec. 2006.0089.543 A1 4, 2006 Kim et al.
31, 2007, now abandoned. 2006/01292.77 A1 6, 2006 Wu et al.
2006.0143647 A1 6, 2006 Bill
(56) References Cited 2.83. A 3: First al. a C a.
U.S. PATENT DOCUMENTS 2.83. A 1939. Riyet al. CCaC ( a.

5.447,166 A 9, 1995 Gevins 2006, O294,084 A1 12/2006 Patel et al.
5,724,987 A 3, 1998 Gevins et al. 2006/02944.65 A1 12/2006 Ronen et al.
6,001,065. A 12/1999 DeVito 2007/0002057 A1 1/2007 Danzig et al.
6,009,458. A 12/1999 Hawkins et al. 2007/0050715 A1 3, 2007 Behar
6,081,660 A 6, 2000 Macleod et al. 2007/006691.6 A1 3, 2007 Lemos
6,097.981 A 8, 2000 Freer 2007/0082738 A1 4, 2007 Fickie et al.
6,099,319 A 8/2000 Zaltman et al. 2007/0101368 A1 5/2007 Jacoby et al.
6,102,846 A 8, 2000 Patton et al. 2007/O168357 A1 T/2007 Mo
6.251,017 B1 6/2001 Leason et al. 2007/0176921 A1 8/2007 Iwasaki et al.
2.68
I-
R 658) R tal
ering et al.
2007/0197274 A1
2007/0207846 A1
8/2007 Dugan
9, 2007 Burak et al.
s: R: Hi8 solo et al. 2007/02141 06 A1 9, 2007 Johnston et al.
6904.40s B1 6/2005 N Alth tal 2007/0218.987 A1 9, 2007 Van Luchene et al.
; : 5. E." 2007/0233579 A1 10, 2007 Saarinen et al.
7,120,880 Bf 10/2006 Dryeretal 2007,0260984 A1 11/2007 Marks et al.
7,478,047 B2 1/2009 Loyall et al. 2007,0265507 A1 11/2007 de Lemos
7,764,311 B2 T/2010 B11 2007/0282177 A1 12, 2007 P1z
7,788,188 B2 8, 2010 Kramer 2007/0282912 A1 12/2007 Reiner
7,930, 199 B1 4/2011 Hill 2008/0030496 A1 2/2008 Lee et al.
7.979,574 B2 7/2011 Gillo et al. 2008/0039204 A1 2/2008 Ackley et al.
8,069,125 B2 * 11/2011 Jung .................. A61B 5,04842 2008, OO39737 A1 2/2008 Breiter et al.
TO6/11 2008/0065468 A1 3/2008 Berg et al.
8, 150,796 B2 * 4/2012 Jung ...................... G06Q 30/02 2008/008 1692 A1 4/2008 Pope et al.
TO6/45 2008/009 1692 A1 4/2008 Keith et al.
8, 195,593 B2 * 6/2012 Jung ........................ GO6N 5/02 2008/01201 13 A1 5/2008 Loyall et al.
TO6/12 2008/O120558 A1 5.2008 Nathan et al.
8,356,004 B2 * 1/2013 Jung ...................... A61B: 2008/O120588 A1 5.2008 Becker
2008. O158222 A1 7, 2008 Li et al.
E.
s- ww.
E. 1338 Myde .................... G06F 19,326
2008/0163054 A1
2008/O163379 A1
7/2008 Pieper et al.
7/2008 Robinson et all
128.200.16
8,615.407 B2 * 12/2013 Hyde .................... Goo3,
128,200.15
2008/01724.12 A1
2008, 0214902 A1
7/2008 Gruhl et al.
9, 2008 Lee et al.
8,615.479 B2 * 12/2013 Jung .................. A61B 5,04842 2008, 0215971 A1 9, 2008 Gillo et al.
TO6/46 2008/0222295 A1 9, 2008 Robinson et al.
8,706,518 B2 4/2014 Hyde ................... A61B 5,0484 2008, 0235581 A1 9/2008 Caporale et al.
128,200.15 2008/0262911 A1 10, 2008 Altberg et al.
8,930.208 B2 * 1/2015 Hyde .................. G06F 1934 2008/0263.459 A1 10, 2008 N. et al.
2008/0275340 A1 11/2008 Beach et al.
9,026,369 B2 * 5/2015 Hyde .................. G06F 1955 2008/0309675 A1 12/2008 Fleury et al.
9,064.036 B2 * 6/2015 Hyde .................... G06F 19,326 2009,0002479 A1 1/2009 Sangberg et al.
9,211,077 B2 * 12/2015 Jung ... ... A61B 5,0476 2009 OO 13002 A1 1/2009 Eggink et al.
9.282,927 B2 * 3/2016 Hyde ..................... A61B5/024 2009/0044113 A1 2/2009 Jones et al.
2001/0034661 Al 10/2001 Ferreira 2009 OO63249 A1 3, 2009 Tomlin et al.
2001/0056225 Al 12/2001 DeVito 2009/01092 13 Al 4/2009 Hamilton, II et al.
2002fOO58867 A1 5, 2002 Breiter et al. 2009.0143695 A1 6, 2009 Mullen et al.
2002/0090985 Al
2002/0113820 A1 72002
8, 2002 Tochner
Robinsonetetal.al. 2009/0156907 A1 6/2009 Jung et al.
2003, OO69728 A1 4, 2003 Tato et al. 2009/0156955 A1 6/2009 Jung et al.
2003/0098.954 A1 5, 2003 Amir et al. 2009/0157323 A1 6/2009 Jung et al.
2003/0120140 A1 6/2003 Bango, Jr. 2009.0161856 A1 6, 2009 Lurie
2003/0142041 A1 7/2003 Barlow et al. 2009,0177976 A1 7, 2009 Bokor et al.
2003.0156134 A1 8, 2003 Kim 2009/0253982 A1 10/2009 Wang
US 9,495.684 B2
Page 3
(56) References Cited Contreras, Marco, et al., “Inactivation of the Interoceptive Insula
Disrupts Drug Craving and Malaise Induced by Lithium”: Science;
U.S. PATENT DOCUMENTS bearing a date of Oct. 26, 2007: pp. 655-658; vol. 318
Gur, Raquel E. et al., “An fMRI Study of Facial Emotion Processing
2009/0300.525 A1 12/2009 Jolliff et al. in Patients with Schizophrenia'; Am. J. Psych., bearing a date of
2010, OO 10366 A1 1/2010 Silberstein
Dec. 2002: pp. 1992-1999; vol. 159; No. 12.
Henig, Robin Marantz: “Looking for the Lie.” New York Times;
OTHER PUBLICATIONS bearing a date of Feb. 5, 2006; pp. 1-13; located at http://www.
nytimes.com/2006/02/05/magazine/05lying.
. Appl. No. 12/215,042, Jung et al. html?pagewanted print.
. Appl. No. 12/214,788, Jung et al. Holroyd, Clay B., et al., “Dorsal anterior cingulate cortex shows
. Appl. No. 12/214.539, Jung et al. fMRI response to internal and external error signals'; Nature
. Appl. No. 12/157.929, Jung et al. Neuroscience; bearing a date of May 2004; pp. 497-498; vol. 7: No.
. Appl. No. 12/006,239, Jung et al. 5.
. Appl. No. 12/006,235, Jung et al.
. Appl. No. 12/006,234, Jung et al. Illner et al.; "Functional brain imaging: Evaluation of the effects of
. Appl. No. 12/006,233, Jung et al. violent media exposure': Paediatr Child Health; May/Jun. 2003; pp.
. Appl. No. 12/006.232, Jung et al. 283-284; vol. 8, No. 5.
. Appl. No. 12/005,168, Jung et al. Kenning, Peter et al.; “NeuroEconomics: An overview from an
. Appl. No. 12/005,151, Jung et al. economic perspective,” Brain Research Bulletin; bearing a date of
. Appl. No. 12/005,115, Jung et al. 2005; pp. 343-354; vol. 67; Elseiver, Inc.
. Appl. No. 12/005,114, Jung et al. Kim, Sonho et al.: “Usability Evaluation of Humanoid-Animation
. Appl. No. 12/005,067, Jung et al. Avatar with Physiological Signals'; Frontiers in the Convergence of
. Appl. No. 12/005,063, Jung et al. Bioscience and Information Technologies; IEEE; bearing a date of
. Appl. No. 12/005,046, Jung et al. Oct. 11-13, 2007: pp. 628-629 & 636-639, 1 supplemental page.
. Appl. No. 12/002.778, Jung et al. King-Casas, Brooks et al., “Getting to Know You: Reputation and
.S. Appl. No. 12/002,289, Jung et al. Trust in a Two-Person Economic Exchange' Science, bearing a date
Men Motivated by 'Superior Wage'; BBC News; bearing a date of Apr. 1, 2005; pp. 78-83; vol. 308; No. 5718; AAAS.
of Nov. 23, 2007; p. 1; located at http://newsvote.bbc.co.uk/mpapps/ Knoch, Daria, et al.; “Diminishing Reciprocal Fairness by Disrupt
pagetools/print/news.bbc.co.uk/2/h; printed on Nov. 29, 2007. ing the Right Prefrontal Cortex'; Science; bearing a date of Nov. 3,
“Mind-reading machine knows what you see.” NewScientist.com; 2006; pp. 829-832; vol. 314; No. 5800; AAAS.
bearing a date of Apr. 25, 2005; pp. 1-3; located at http://www. Koechlin, Etienne and Hyafil. Alexandre; “Review: Anterior
newscientist.com/article.ns?id=dn7304&feed Id=online-news Prefrontal Function and the Limits of Human Decision-Making';
rss20; printed on Nov. 28, 2007. Science; bearing a date of Oct. 26, 2007: pp. 594-598; vol. 318.
“New mini-sensor may have biomedical and security applications'; Kording: Konrad; "Decision Theory: What "Should” the Nervous
Physorg.com; bearing a date of Nov. 1, 2007: pp. 1-4; located at System Do?”: Science; bearing a date of Oct. 26, 2007: pp. 606-610;
http://www.physorg.com/news113151078.html; printed on Nov. 8, vol. 318.
2007. Krepki et al.; "The Berlin Brain-Computer Interface (BBCI)
“Technology could enable computers to read the minds of users' towards a new communication channel for online control of mul
Physorg.com; pp. 1-2, located at http://www.physorg.com/ timedia applications and computer games"; 9" International Con
news110463755.html; printed on Nov. 28, 2007; Tufts University. ference on Distributed Multimedia Systems (DMS ’03); 2003; pp.
“The SIMS The People Simulator from the Creator of SimCity': 1-8.
User Manual: 2000; 56 pages; Electronic Arts. Lee, Nick: “What is Neuromarketing? A Discussion and Agenda
Ambler, Tim, et al.; “Salience and Choice: Neural Correlates of for Future Research”; International Journal of Psychophysiology;
Shopping Decisions'; Psychology & Marketing; bearing a date of bearing a date of 2007: pp. 199-204; vol. 63; Elseiver B.V.
Apr. 2004; pp. 247-261; vol. 21; Wiley Periodicals, Inc. Leuthardt et al.; “Electrocorticography-Based Brain Computer
Axelrod, Lesley, et al., “Smoke and Mirrors: Gathering User Interfac- The Seattle Experience'; IEEE Transactions on Neural
Requirements for Emerging Affective Systems'; 26th Int. Conf. Systems and Rehabilitation Engineering; Jun. 2006; pp. 194-198;
Information Technology Interfaces ITI 2004; bearing a date of Jun. vol. 14, No. 2: IEEE.
7-10, 2004; pp. 323-328, Cavtat, Croatia. Li, Hairong et al.; "The Impact of Perceived Channel Utilities,
Ayanwale et al.; "The Influence of Advertising on Consumer Brand Shopping Orientations, and Demographics on the Consumer's
Preference'; J. Soc. Sci., 2005; pp. 9-16; vol. 10(1): Kamla-Raj. Online Buying Behavior”; JCMC; Dec. 1999: pp. 1-20; vol. 5, No.
Bellman, Steven et al.; “Predictors of Online Buying Behavior”; 2.
Communications of the ACM; Dec. 1999: pp. 32-38; vol. 42, No. Martin, Sarah B., et al.; “Human Experience Seeking Correlates
12; ACM. with Hippocampus Volume: Convergent Evidence from Manual
Bloom, Niels; “Parallel Brain-Computer Interface signal process Tracing and Voxel-Based Morphometry; Neuropsychologia, bear
ing'; Oct. 23, 2007: pp. 1-69; located at http:janus.cs.utwente. ing a date of 2007: pp. 2874-2881 (abstract p. 1); vol. 45, Issue 12;
nl:8000/twiki/pub/brainmedia/intermediateandfinalreports/ Elseiver B.V.
afstudeerverslagneilsbloomfinal.pdf. Montgomery; Alan L. et al.: “Learning About Customers Without
Burke, Raymond R.; "Technology and the Customer Interface: Asking'; Jan. 2002: pp. 1-33; Alan L. Montgomery and Kannan
What Consumers Want in the Physical and Virtual Store'; Journal Srinivasan.
of the Academy of Marketing Science; 2002: pp. 411-432; vol. 30. Nakai et al.; "Scalable Collaborative Virtual Environment Consid
No. 4: Academy of Marketing Science. ering User's Interests Based on P2POverlay Network'; Proceedings
Cabeza, Roberto, et al., “Imaging Cognition II: An Empirical of the First International Conference on Complex, Intelligent and
Review of 275 PET and fMRI Studies”: Journal of Cognitive Software Intensive Systems (CISIS07); 2007; 5 pages; IEEE.
Neuroscience; bearing a date of 2000, pp. 1-47, vol. 12, No. 1. Parc Research; "Content-Centric Networking: PARC's Strategy for
Case, Thomas et al., “Drivers of On-Line purchasing Among U.S. Pioneering a Self-Organizing Network That Meets Information
University Students' AMCIS 2001 Proceedings; Dec. 31, 2001; pp. Needs'; pp. 1-4; Xerox Corporation; located at: http://www.parc.
873-878; Paper 169; AIS Electronic Library. Xerox.com/research/projects/networking/contentcentric/
Clarke, Peter; "IMEC has a brain wave: feed EEG emotion back into mediabackgrounder.html; printed on Mar. 2, 2009.
games'; EE Times online, bearing a date of Nov. 1, 2007: pp. 1-2; Paulus, Martin P: “Review: Decision-Making Dysfunctions in
located at http://www.eetimes.eu/design/202801.063; printed on Psychiatry-Altered Homeostatic Processing?"; Science, bearing a
Nov. 1, 2007. date of Oct. 26, 2007: pp. 602-606; vol. 318.
US 9,495.684 B2
Page 4
(56) References Cited Stratonnikov, Alexander A., et al.; “Evaluation of Blood Oxygen
Saturation in vivo from Diffuse Reflectance Spectra'; abstract;
OTHER PUBLICATIONS Journal of Biomedical Optics; bearing a date of Oct. 2001; pp.
457-467 (Abstract p. 1-2); vol. 6; printed on Nov. 28, 2007.
Picard, Rosalind W. et al.; "Toward Machine Emotional Intelli Van Den Poel, Dirk et al.; “Predicting online-purchasing behavior”;
gence: Analysis of Affective Physiological State'; IEEE Transac European Journal of Operational Research; 2005 pp. 557-575; vol.
tions on Pattern Analysis and Machine Intelligence; bearing a date 166
of Oct. 2001; vol. 23, No. 10; pp. 1175-1191. Vasalou, Asimina, et al.; "Constructing My Online Self: Avatars that
Potenza, Marc N. et al; “Gambling Urges in Pathological Gam Increase Self-Focused Attention'; CHI 2007; bearing dates of Apr.
bling'; Arch Gen Psychiatry; bearing a date of Aug. 2003; pp. 28-May 3, 2007: pp. 1-4; San Jose, California; ACM.
828-836; vol. 60; downloaded from www.archgenpsychiatry.com Wang, Gene-Jack, et al., “Gastric stimulation in obese subject
on Oct. 6, 2007: printed on Nov. 28, 2007. activates the hippocampus and other regions involved in brain
Rossiter et al.; “Brain-Imaging Detection of Visual Scene Encoding reward circuitry': PNAS, bearing a date of Oct. 17, 2006; pp.
in Long-term Memory for TV Commercials'; Journal of Advertis 15641-15645; vol. 103, No. 42.
ing Research; Mar/Apr. 2001; pp. 13-21. Wise, Jeff, “Thought Police. How Brain Scans Could Invade Your
Schaefer, Michael, et al.: “Neural Correlates of Culturally Familiar Private Life'. Popular Mechanics; bearing a date of Nov. 2007: pp.
Brands of Car Manufacturers'; NeuroImage; bearing dates of Oct. 1-4; located at: www.popularmechanics.com/science/research/
8, 2005, and Feb. 17, 2006; pp. 861-865; vol. 31. 4226614.html; printed on Nov. 8, 2007.
Stern, Peter; "Decisions, Decisions'; Science; bearing a date of Oct.
26, 2007; p. 593; vol. 318 * cited by examiner
U.S. Patent Nov. 15, 2016 Sheet 1 of 31 US 9,495.684 B2
·
*
$ $
·
*
•.*
•
N«*%g?ow.?eºn
&š?{};**.
3
33
:).-~~~~---3.---~-. ~~;~~~~~~;-~ ~ ~~
~)
~
3,8×3 %
~~~ ~ }
rig. 4
: ixiet 88.888 (f
38 x 3: ;
exist:88's 8888
six xxxii: sixx: {xf

8 gai: wit:
8x338's 8888
U.S. Patent Sheet 6 of 31 US 9,495.684 B2
{{}
U.S. Patent US 9,495.684 B2
***
OffG
-
0,29 099
Ionsuelo ?
(ŒuDoe^s~)
9
||
0
079
-
019
DOveoi~s —s
019 029
(OueogDrs~)
assa ra--
--------------------.
w- -–!- >|-w?:
NJ
L|| || ||
pueoalqji!d|
|e
|umwaxe?10 |90ual jaudi
|:puelq
(D
uers
O
eilslnplnoe u1
;
r.
~)og –|
–
!
|fÁilumeaionseopau?u?!|eleme?os
} } { { } } } }
U.S. Patent Nov. 15, 2016 Sheet 19 Of 31 US 9,495.684 B2
presenting at least one characteristic to at least one member of a

population Cohort
measuring at least one physiologic activity of the at least one member 520
N
of the population Cohort, the at least one physiologic activity proximate
to the at least one presented characteristic
associating the at least one physiologic activity with

at east one mental state
specifying at least one avatar attribute based on the at least one

mental state
1950
performing at least two successive iterations of the presenting at least
one characteristic to at least one member of a population cohort, the
measuring at least one physiologic activity of the at least one member
of the population Cohort, the at least one physiologic activity proximate
to the at least one presented characteristic, and the associating the at
least one physiologic activity with at least one mental state
End
·:
·
*·
i
'''''''''''''''''.
w
wa'www.w. awasawww.www.
3
•
92s~9o0?°a,
zzº~g9l0?%
&
traw-M- awasaww.www.waw88.www.sawaraw
} { }
* 'w--------------------'-------------'-------------'--------'-------
*.
szgya(~80%
*
%
#
~
8-8
{ {} ?
'
*;p~
*· .
^* * * '' ... ** ** ***
··:·
w.s.. . . . .
3§§§§
US 9,495,684 B2
1. 2
METHODS AND SYSTEMIS FOR ING A COHORT-LINKED AVATAR ATTRIBUTE,
INDICATING BEHAVOR IN A POPULATION naming Edward K.Y. Jung, Eric C. Leuthardt, Royce A.
COHORT Levien, Robert W. Lord, Mark A. Malamud, John D.
Rinaldo, Jr., and Lowell L. Wood, Jr. as inventors, filed
CROSS-REFERENCE TO RELATED 20 Dec. 2007 which is currently co-pending, or is an
APPLICATIONS application of which a currently co-pending application
is entitled to the benefit of the filing date.
The present application is related to and claims the benefit For purposes of the USPTO extra-statutory requirements,
of the earliest available effective filing date(s) from the the present application constitutes a continuation-in
following listed application(s) (the “Related Applications') 10
(e.g., claims earliest available priority dates for other than part of U.S. patent application Ser. No. 12/005,114,
provisional patent applications or claims benefits under 35 entitled METHODS AND SYSTEMS EMPLOYINGA
USC S 119(e) for provisional patent applications, for any and COHORT-LINKED AVATAR, naming Edward K.Y.
all parent, grandparent, great-grandparent, etc. applications Jung, Eric C. Leuthardt, Royce A. Levien, Robert W.
of the Related Application(s)). 15 Lord, Mark A. Malamud, John D. Rinaldo, Jr., and
Lowell L. Wood, Jr. as inventors, filed 20 Dec. 2007
RELATED APPLICATIONS which is currently co-pending, or is an application of
which a currently co-pending application is entitled to
For purposes of the USPTO extra-statutory requirements, the benefit of the filing date.
the present application constitutes a continuation of For purposes of the USPTO extra-statutory requirements,
U.S. patent application Ser. No. 12/217.288, entitled the present application constitutes a continuation-in
METHODS AND SYSTEMS FOR INDICATING
BEHAVIOR IN A POPULATION COHORT, naming part of U.S. patent application Ser. No. 12/005,115,
entitled METHODS AND SYSTEMS FOR COMPAR
Edward K.Y. Jung, Eric C. Leuthardt, Royce A. Levien,
Robert W. Lord, Mark A. Malamud, John D. Rinaldo, 25 ING MEDIA CONTENT, naming Edward K.Y. Jung,
Jr., and Lowell L. Wood, Jr. as inventors, filed Jul. 1, Eric C. Leuthardt, Royce A. Levien, Robert W. Lord,
2008, which is currently co-pending or is an application Mark A. Malamud, John D. Rinaldo, Jr., and Lowell L.
of which a currently co-pending application is entitled Wood, Jr. as inventors, filed 20 Dec. 2007 which is
to the benefit of the filing date. currently co-pending, or is an application of which a
For purposes of the USPTO extra-statutory requirements, 30 currently co-pending application is entitled to the ben
the present application constitutes a continuation-in efit of the filing date.
part of U.S. patent application Ser. No. 12/002.289 For purposes of the USPTO extra-statutory requirements,
entitled METHODS AND SYSTEMS FOR SPECIFY
the present application constitutes a continuation-in
ING ANAVATAR naming Edward K.Y. Jung, Eric C. part of U.S. patent application Ser. No. 12/006,234
Leuthardt, Royce A. Levien, Robert W. Lord, Mark A. 35
entitled METHODS AND SYSTEMS FOR IDENTI
Malamud, John D. Rinaldo, Jr., and Lowell L. Wood, Jr. FYING AN AVATAR-LINKED POPULATION
as inventors, filed 13 Dec. 2007, which is currently COHORT, naming Edward K.Y. Jung, Eric C.
co-pending, or is an application of which a currently
co-pending application is entitled to the benefit of the Leuthardt, Royce A. Levien, Robert W. Lord, Mark A.
filing date. 40 Malamud, John D. Rinaldo, Jr., and Lowell L. Wood, Jr.
For purposes of the USPTO extra-statutory requirements, as inventors, filed 31 Dec. 2007 which is currently
the present application constitutes a continuation-in co-pending, or is an application of which a currently
part of U.S. patent application Ser. No. 12/002.778, co-pending application is entitled to the benefit of the
entitled METHODS AND SYSTEMS FOR IDENTI filing date.
FYING AN AVATAR-LINKED POPULATION 45 For purposes of the USPTO extra-statutory requirements,
COHORT, naming Edward K.Y. Jung, Eric C. the present application constitutes a continuation-in
Leuthardt, Royce A. Levien, Robert W. Lord, Mark A. part of U.S. patent application Ser. No. 12/006,235
Malamud, John D. Rinaldo, Jr., and Lowell L. Wood, Jr. entitled METHODS AND SYSTEMS FOR COMPAR
as inventors, filed 17 Dec. 2007 which is currently ING MEDIA CONTENT, naming Edward K.Y. Jung,
co-pending, or is an application of which a currently 50
Eric C. Leuthardt, Royce A. Levien, Robert W. Lord,
co-pending application is entitled to the benefit of the Mark A. Malamud, John D. Rinaldo, Jr., and Lowell L.
filing date.
For purposes of the USPTO extra-statutory requirements, Wood, Jr. as inventors, filed 31 Dec. 2007 which is
the present application constitutes a continuation-in currently co-pending, or is an application of which a
part of U.S. patent application Ser. No. 12/005,046, 55 currently co-pending application is entitled to the ben
entitled METHODS AND SYSTEMS FOR INDICAT efit of the filing date.
ING BEHAVIOR IN A POPULATION COHORT, For purposes of the USPTO extra-statutory requirements,
naming Edward K.Y. Jung, Eric C. Leuthardt, Royce A. the present application constitutes a continuation-in
Levien, Robert W. Lord, Mark A. Malamud, John D. part of U.S. patent application Ser. No. 12/006,239
Rinaldo, Jr., and Lowell L. Wood, Jr. as inventors, filed 60 entitled METHODS AND SYSTEMS FOR SPECIFY
20 Dec. 2007 which is currently co-pending, or is an ING ANAVATAR, naming Edward K.Y. Jung, Eric C.
application of which a currently co-pending application Leuthardt, Royce A. Levien, Robert W. Lord, Mark A.
is entitled to the benefit of the filing date. Malamud, John D. Rinaldo, Jr., and Lowell L. Wood, Jr.
For purposes of the USPTO extra-statutory requirements, as inventors, filed 31 Dec. 2007 which is currently
the present application constitutes a continuation-in 65 co-pending, or is an application of which a currently
part of U.S. patent application Ser. No. 12/005,063, co-pending application is entitled to the benefit of the
entitled METHODS AND SYSTEMS FOR SPECIFY filing date.
US 9,495,684 B2
3 4
TECHNICAL FIELD summary is illustrative only and is NOT intended to be in
any way limiting. Other aspects, features, and advantages of
This description relates to data capture and data handling the devices and/or processes and/or other subject matter
techniques. described herein will become apparent in the teachings set
forth herein.
SUMMARY
BRIEF DESCRIPTION OF THE DRAWINGS
An embodiment provides a method. In one implementa
tion, the method includes but is not limited to identifying a With reference now to FIG. 1, shown is an example of an
member of a population cohort; and indicating at least one 10 avatar attribute specification system in which embodiments
behavior in the member of the population cohort based on an may be implemented, perhaps in a device and/or through a
association between the population cohort and at least one network, which may serve as a context for introducing one
cohort-linked avatar. In addition to the foregoing, other or more processes and/or devices described herein. Also
method aspects are described in the claims, drawings, and depicted on FIG. 1 is the example operational flow of FIG.
text forming a part of the present disclosure. 15 24 described below.
In one or more various aspects, related systems include FIG. 2 illustrates certain alternative embodiments of the
but are not limited to circuitry and/or programming for system of FIG. 1.
effecting the herein-referenced method aspects; the circuitry FIG. 3 illustrates certain alternative embodiments of the
and/or programming can be virtually any combination of system of FIG. 1.
hardware, software, and/or firmware configured to effect the FIG. 4 shows diagrammatic views of the surface of the
herein-referenced method aspects depending upon the human brain.
design choices of the system designer. With reference now to FIG. 5, shown is an example of an
An embodiment provides a system. In one implementa operational flow representing example operations related to
tion, the system includes but is not limited to circuitry for specifying an avatar attribute, which may serve as a context
identifying a member of a population cohort; and circuitry 25 for introducing one or more processes and/or devices
for indicating at least one behavior in the member of the described herein.
population cohort based on an association between the FIG. 6 illustrates an alternative embodiment of the
population cohort and at least one cohort-linked avatar. In example operational flow of FIG. 5.
addition to the foregoing, other system aspects are described FIG. 7 illustrates an alternative embodiment of the
in the claims, drawings, and text forming a part of the 30 example operational flow of FIG. 5.
present disclosure. FIG. 8 illustrates an alternative embodiment of the
An embodiment provides a computer program product. In example operational flow of FIG. 5.
one implementation, the computer program product includes FIG. 9 illustrates an alternative embodiment of the
but is not limited to a signal-bearing medium bearing (a) one example operational flow of FIG. 5.
or more instructions for identifying a member of a popula 35 FIG. 10 illustrates an alternative embodiment of the
tion cohort; and (b) one or more instructions for indicating example operational flow of FIG. 5.
at least one behavior in the member of the population cohort FIG. 11 illustrates an alternative embodiment of the
based on an association between the population cohort and example operational flow of FIG. 5.
at least one cohort-linked avatar. In addition to the forego FIG. 12 illustrates an alternative embodiment of the
ing, other computer program product aspects are described 40 example operational flow of FIG. 5.
in the claims, drawings, and text forming a part of the FIG. 13 illustrates an alternative embodiment of the
present disclosure. example operational flow of FIG. 5.
An embodiment provides a system. In one implementa FIG. 14 illustrates an alternative embodiment of the
tion, the system includes but is not limited to a computing example operational flow of FIG. 5.
device and instructions. The instructions when executed on 45 FIG. 15 illustrates an alternative embodiment of the
the computing device cause the computing device to (a) example operational flow of FIG. 5.
identify a member of a population cohort; and (b) indicate at FIG. 16 illustrates an alternative embodiment of the
least one behavior in the member of the population cohort example operational flow of FIG. 5.
based on an association between the population cohort and FIG. 17 illustrates an alternative embodiment of the
at least one cohort-linked avatar. In addition to the forego 50 example operational flow of FIG. 5.
ing, other system aspects are described in the claims, draw FIG. 18 illustrates an alternative embodiment of the
ings, and text forming a part of the present disclosure. example operational flow of FIG. 5.
In one or more various aspects, related systems include FIG. 19 illustrates an alternative embodiment of the
but are not limited to computing means and/or programming example operational flow of FIG. 5.
for effecting the herein referenced method aspects; the 55 With reference now to FIG. 20, shown is a partial view of
computing means and/or programming may be virtually any an example computer program product that includes a
combination of hardware, software, and/or firmware config computer program for executing a computer process on a
ured to effect the herein referenced method aspects depend computing device related to specifying an avatar attribute,
ing upon the design choices of the system designer. which may serve as a context for introducing one or more
In addition to the foregoing, various other method and/or 60 processes and/or devices described herein.
system and/or program product aspects are set forth and With reference now to FIG. 21, shown is an example
described in the teachings such as text (e.g., claims and/or device in which embodiments may be implemented related
detailed description) and/or drawings of the present disclo to specifying an avatar attribute, which may serve as a
SUC. context for introducing one or more processes and/or
The foregoing is a Summary and thus contains, by neces 65 devices described herein.
sity, simplifications, generalizations and omissions of detail; With reference now to FIG. 22, shown is an example of
consequently, those skilled in the art will appreciate that the a system for indicating behavior in a population cohort in
US 9,495,684 B2
5 6
which embodiments may be implemented, perhaps in a association module 244, and/or cognition association mod
device and/or through a network, which may serve as a ule 246. Device 206 may also include population cohort
context for introducing one or more processes and/or identification unit 266.
devices described herein. Also depicted on FIG. 22 is the Device 206 may also include attribute specification unit
example operational flow of FIG. 24 described below. 250, which may in turn include voice specification unit 252,
FIG. 23 illustrates certain alternative embodiments of the speech specification unit 254, non-verbal attribute specifi
system of FIG. 22. cation unit 256, facial attribute specification unit 258, cloth
With reference now to FIG. 24 shown is an example of an ing specification unit 260, and/or body attribute specification
operational flow representing example operations related to unit 262. Device 206 may also include presentation unit 270,
indicating behavior in a population cohort, which may serve 10 which may in turn include display 272, which may in turn
as a context for introducing one or more processes and/or include desktop display 274 and/or mobile display 276,
devices described herein. which may in turn include pico projector 278 and/or wear
FIG. 25 illustrates an alternative embodiment of the able display 280. Member of population cohort 102, mul
example operational flow of FIG. 24. tiple members of population cohort 104, and/or one or more
FIG. 26 illustrates an alternative embodiment of the
15 members of population 105 may interact with device 206
including presentation unit 270, and/or be monitored by
example operational flow of FIG. 24. device 206 including physiologic activity measurement unit
FIG. 27 illustrates an alternative embodiment of the 210.
example operational flow of FIG. 24. In FIG. 2, the device 206 is illustrated as possibly being
FIG. 28 illustrates an alternative embodiment of the included within a system 100. Of course, virtually any kind
example operational flow of FIG. 24. of computing device may be used to implement the physi
With reference now to FIG. 29, shown is a partial view of ologic activity measurement unit 210, Surrogate marker
an example computer program product that includes a measurement unit 230, association unit 340, attribute speci
computer program for executing a computer process on a fication unit 350, and/or presentation unit 264, such as, for
computing device related to indicating behavior in a popu 25 example, a workstation, a desktop computer, a networked
lation cohort, which may serve as a context for introducing computer, a server, a collection of servers and/or databases,
one or more processes and/or devices described herein. a virtual machine running inside a computing device, a
With reference now to FIG. 30, shown is an example mobile computing device, or a tablet PC.
device in which embodiments may be implemented related Additionally, not all of the physiologic activity measure
to indicating behavior in a population cohort, which may 30 ment unit 210, surrogate marker measurement unit 230,
serve as a context for introducing one or more processes association unit 240, population cohort identification unit
and/or devices described herein. 266, attribute specification unit 250, and/or presentation unit
With reference now to FIG. 31, shown is an example 264 need be implemented on a single computing device. For
system in which embodiments may be implemented related example, one or more of the physiologic activity measure
to indicating behavior in a population cohort, which may 35 ment unit 210, surrogate marker measurement unit 230,
serve as a context for introducing one or more processes association unit 340, population cohort identification unit
and/or devices described herein. 366, attribute specification unit 350, and/or presentation unit
The use of the same symbols in different drawings typi 264 may be implemented and/or operable on a remote
cally indicates similar or identical items. computer, while one or more of these functions are imple
40 mented and/or occur on a local computer. Further, aspects of
DETAILED DESCRIPTION the physiologic activity measurement unit 210 may be
implemented in different combinations and implementations
FIG. 1 illustrates an example system 100 in which than that shown in FIG. 1. For example, functionality of a
embodiments may be implemented. The system 100 physiologic activity measurement unit 210 may be incorpo
includes a device 106. The device 106 may contain, for 45 rated into the Surrogate marker measurement unit 230,
example, a presentation unit 170, a physiologic activity association unit 240, population cohort identification unit
measurement unit 110, an association unit 140, an attribute 266, attribute specification unit 250, and/or presentation unit
specification unit 150, and/or a population cohort identifi 264. The association unit 240, population cohort identifica
cation unit 166. The device 106 may interact with one or tion unit 366, and/or attribute specification unit 250 may
more members of population cohort 104 and/or population 50 perform simple data relay functions and/or complex data
105. Population cohort 104 may be a part of a population analysis, including, for example, fuzzy logic and/or tradi
105. tional logic steps. Further, many methods of searching
FIG. 2 illustrates the example system 100 in which functional brain mapping and/or Surrogate marker activity
embodiments may be implemented. The system 100 databases known in the art may be used, including, for
includes a device 206. Device 206 may include physiologic 55 example, unsupervised pattern discovery methods, coinci
activity measurement unit 210, which may in turn include dence detection methods, and/or entity relationship model
brain activity measurement unit 212, which may in turn ing. In some embodiments, the association unit 240 may
include functional near-infrared imaging fNIR module 214, process physiologic activity measurements according to
functional magnetic resonance imaging fMRI module 216, activity profiles available as updates through a network.
magnetoencephalography MEG module 218, electroen 60 Similarly, in some embodiments, the population cohort
cephalography EEG module 220, and/or positron emission identification unit 366 and/or attribute specification unit 250
topography PET module 222. Device 206 may also include may process association unit output according to population
Surrogate marker measurement unit 230 including iris cohort, media content, avatar attribute, and/or avatar profiles
response module 232, gaze tracking module 234, skin available as updates through a network.
response module 236, and/or voice response module 238. 65 Outputs of physiologic activity measurement unit 210,
Device 206 may also include association unit 240, which in Surrogate marker measurement unit 230, association unit
turn may include emotion association module 242, attention 240, population cohort identification unit 266, attribute
US 9,495,684 B2
7 8
specification unit 250, and/or presentation unit 264 may be population cohort based on a mental state associated with a
stored in virtually any type of memory that is able to store physiological activity of member of population cohort 102
and/or provide access to information in, for example, a or member of population 105. For example, where the
one-to-many, many-to-one, and/or many-to-many relation output of physiologic activity measurement unit 310 is
ship. Such a memory may include, for example, a relational 5 organized, keyed to, and/or otherwise accessible using one
database and/or an object-oriented database, examples of or more reference physiologic activity profiles, association
which are provided in more detail herein. unit 340 may employ various Boolean, statistical, and/or
FIG. 3 illustrates the example system 100 in which semi-boolean searching techniques to match physiologic
embodiments may be implemented. The system 100 activity measurement output with one or more appropriate
includes a device 306. The device 306 may communicate 10 mental states. Similarly, for example, where association unit
over a network 374 with a presentation device 364. Device 340 output is organized, keyed to, and/or otherwise acces
306 may include physiologic activity measurement unit 310, sible using one or more reference population cohort profiles,
which may in turn include brain activity measurement unit various Boolean, statistical, and/or semi-boolean searching
312, which may in turn include functional near-infrared techniques may be performed by population cohort identi
imaging fNIR module 314, functional magnetic resonance 15 fication unit 366 to match the mental state of the member of
imaging fMRI module 316, magnetoencephalography MEG population 105 with one or more appropriate population
module 318, electroencephalography EEG module 320, and/ cohorts.
or positron emission topography PET module 322. Device Many examples of databases and database structures may
306 may also include Surrogate marker measurement unit be used in connection with the device 306, association unit
330 including iris response module 332, gaze tracking 340, and/or attribute specification unit 350. Such examples
module 334, skin response module 336, and/or voice include hierarchical models (in which data is organized in a
response module 338. Device 306 may also include asso tree and/or parent-child node structure), network models
ciation unit 340, which in turn may include emotion asso (based on set theory, and in which multi-parent structures
ciation module 342, attention association module 344, and/ per child node are Supported), or object/relational models
or cognition association module 346. Device 306 may also 25 (combining the relational model with the object-oriented
include population cohort identification unit 366. model).
Device 306 may also include attribute specification unit Still other examples include various types of eXtensible
350, which may in turn include voice specification unit 352, Mark-up Language (XML) databases. For example, a data
which may in turn include speech specification unit 354. base may be included that holds data in some format other
Attribute specification unit 350 may include non-verbal 30 than XML, but that is associated with an XML interface for
attribute specification unit 356, which may in turn include accessing the database using XML. As another example, a
facial attribute specification unit 358, clothing specification database may store XML data directly. Additionally, or
unit 360, and/or body attribute specification unit 362. alternatively, virtually any semi-structured database may be
Presentation device 364 may include presentation unit used, so that context may be provided to/associated with
370, which may in turn include display 372, which may in 35 stored data elements (either encoded with the data elements,
turn include desktop display 374 and/or mobile display 376, or encoded externally to the data elements), so that data
which may in turn include pico projector 378 and/or wear storage and/or access may be facilitated.
able display 380. Member of population cohort 102 and or Such databases, and/or other memory storage techniques,
multiple members of population cohort 104 may be moni may be written and/or implemented using various program
tored by device 306 including physiologic activity measure 40 ming or coding languages. For example, object-oriented
ment unit 310. Member of population cohort 102, multiple database management systems may be written in program
members of population cohort 104, and/or one or more ming languages such as, for example, C++ or Java. Rela
members of population 105 may interact with presentation tional and/or object/relational models may make use of
device 364 including presentation unit 370. database languages, such as, for example, the structured
In this way, a member of population cohort 102 or a 45 query language (SQL), which may be used, for example, for
member of population 105, who may be interacting with a interactive queries for information and/or for gathering
presentation device 364 that is connected through a network and/or compiling data from the relational database(s).
374 with a device 306 (e.g., in a home, an office, outdoors For example, SQL or SQL-like operations over one or
and/or in a public environment), may interact with the more reference physiologic activity measurement and/or
system 100 as if the member of population cohort 102 or 50 reference mental state may be performed, or Boolean opera
member of population 105 were interacting locally with the tions using a reference physiologic activity measurement
device 306 on which the association unit 340, population and/or reference mental state may be performed. For
cohort identification unit 366, and/or attribute specification example, weighted Boolean operations may be performed in
unit 350 is operable. In such an embodiment, the physiologic which different weights or priorities are assigned to one or
activity measurement unit 310 and/or Surrogate marker 55 more of the reference physiologic activity measurements
measurement unit 330 may also be located locally with the and/or reference mental states, including reference physi
member of population cohort 102 or member of population ologic activity measurements and/or reference mental states
105, transmitting output via network to a remote association associated with various reference avatar attributes, perhaps
unit 340, population cohort identification unit 366, and/or relative to one another. For example, a number-weighted,
attribute specification unit 350. 60 exclusive-OR operation may be performed to request spe
As referenced herein, the device 306, association unit cific weightings of desired (or undesired) physiologic activ
340, population cohort identification unit 366, and/or attri ity reference data to be included or excluded. Reference
bute specification unit 350 may be used to perform various physiologic activity measurements may include normal
data querying and/or recall techniques with respect to output physiological values for an individual in a demographic
of physiologic activity measurement unit 310, and/or output 65 group responding to a given stimulus. Such normal physi
of association unit 340, respectively, in order to for example, ological activity values may be “normal relative to the
obtain, identify, and/or transmit an avatar attribute and/or member of population cohort 102, to a group of population
US 9,495,684 B2
10
cohort 104, to the entire population cohort 104, to a member Physorg.com, “New mini-sensor may have biomedical and
of population 105, and/or the entire population 105. Simi security applications.” Nov. 1, 2007, http://www.physorg.
larly, reference demographic characteristics may be associ com/news 113151078.html.
ated with a general population or a Subpopulation defined by Electroencephalography
Such things as age, gender, ethnicity, or other demographic Another method of measuring at least one physiologic
measure known to those of ordinary skill in the art. activity may include measuring the electrical activity of the
FIG. 4 shows diagrammatic views of the surface of the brain by recording from electrodes placed on the scalp or, in
human brain. Lateral surface of the brain with Brodmann's special cases, subdurally, or in the cerebral cortex. The
areas 400 shows various numbered areas of a lateral aspect resulting traces are known as an electroencephalogram
of the human brain. Medial surface of the brain with 10 (EEG) and represent a Summation of post-synaptic poten
Brodmann's areas 402 shows various numbered areas of the tials from a large number of neurons. EEG is most sensitive
to a particular set of post-synaptic potentials: those which
medial aspect of the human brain. are generated in Superficial layers of the cortex, on the crests
Measuring at least one physiologic activity of a member of gyri directly abutting the skull and radial to the skull.
of population cohort 102 may include measuring magnetic, 15 Dendrites that are deeper in the cortex, inside sulci, are in
electrical, hemodynamic, and/or metabolic activity in the midline or deep structures (such as the cingulate gyms or
brain. hippocampus) or that produce currents that are tangential to
Magnetoencephalography the skull make a smaller contribution to the EEG signal.
One method of measuring at least one physiologic activity One application of EEG is event-related potential (ERP)
may include measuring the magnetic fields produced by analysis. An ERP is any measured brain response that is
electrical activity in the brain via magnetoencephalography directly the result of a thought or perception. ERPs can be
(MEG) using magnetometers such as Superconducting quan reliably measured using electroencephalography (EEG), a
tum interference devices (SQUIDS) or other devices. Such procedure that measures electrical activity of the brain,
measurements are commonly used in both research and typically through the skull and scalp. As the EEG reflects
clinical settings to, e.g., assist researchers in determining the 25 thousands of simultaneously ongoing brain processes, the
function of various parts of the brain. Synchronized neu brain response to a certain stimulus or event of interest is
ronal currents indicate very weak magnetic fields that can be usually not visible in the EEG. One of the most robust
measured by magnetoencephalography. However, the mag features of the ERP response is a response to unpredictable
netic field of the brain is considerably smaller at 10 fem stimuli. This response is known as the P300 (P3) and
totesla (fT) for cortical activity and 103 ft for the human 30 manifests as a positive deflection in Voltage approximately
alpha rhythm than the ambient magnetic noise in an urban 300 milliseconds after the stimulus is presented.
environment, which is on the order of 108 ft. Two essential The most robust ERPs are seen after many dozens or
problems of biomagnetism arise: weakness of the signal and hundreds of individual presentations are averaged together.
strength of the competing environmental noise. The devel This technique cancels out noise in the data allowing only
opment of extremely sensitive measurement devices Such as 35 the Voltage response to the stimulus to stand out clearly.
SQUIDS facilitates analysis of the brains magnetic field in While evoked potentials reflect the processing of the physi
spite of the relatively low signal versus ambient magnetic cal stimulus, event-related potentials are caused by higher
signal noise. Magnetoencephalography (and EEG) signals processes, such as memory, expectation, attention, or other
derive from the net effect of ionic currents flowing in the changes in mental state.
dendrites of neurons during synaptic transmission. In accor 40 A two-channel wireless brain wave monitoring system
dance with Maxwell's equations, any electrical current will powered by a thermo-electric generator has been developed
produce an orthogonally oriented magnetic field. It is this by IMEC (Interuniversity Microelectronics Centre, Leuven,
field that is measured with MEG. The net currents can be Belgium). This device uses the body heat dissipated natu
thought of as current dipoles, which are currents having an rally from the forehead as a means to generate its electrical
associated position, orientation, and magnitude, but no spa 45 power. The wearable EEG system operates autonomously
tial extent. According to the right-hand rule, a current dipole with no need to change or recharge batteries. The EEG
gives rise to a magnetic field that flows around the axis of its monitor prototype is wearable and integrated into a head
vector component. band where it consumes 0.8 milliwatts. A digital signal
In order to generate a detectable signal, approximately processing block encodes extracted EEG data, which is sent
50,000 active neurons are needed. Because current dipoles 50 to a PC via a 2.4-GHz wireless radio link. The thermoelec
must have similar orientations to generate magnetic fields tric generator is mounted on the forehead and converts the
that reinforce each other, it is often the layer of pyramidal heat flow between the skin and air into electrical power. The
cells in the cortex, which are generally perpendicular to its generator is composed of 10 thermoelectric units intercon
Surface, that give rise to measurable magnetic fields. Further, nected in a flexible way. At room temperature, the generated
it is often bundles of these neurons located in the Sulci of the 55 power is about 2 to 2.5-mW or 0.03-mW per square centi
cortex with orientations parallel to the surface of the head meter, which is the theoretical limit of power generation
that project measurable portions of their magnetic fields from the human skin. Such a device is proposed to associate
outside of the head. emotion with EEG signals. See Clarke, “IMEC has a brain
Smaller magnetometers are in development, including a wave: feed EEG emotion back into games.” EE Times
mini-magnetometer that uses a single milliwatt infrared laser 60 online, http://www.eetimes.eu/design/202801063 (Nov. 1,
to excite rubidium in the context of an applied perpendicular 2007).
magnetic field. The amount of laser light absorbed by the EEG can be recorded at the same time as MEG so that
rubidium atoms varies predictably with the magnetic field, data from these complimentary high-time-resolution tech
providing a reference scale for measuring the field. The niques can be combined.
stronger the magnetic field, the more light is absorbed. Such 65 Measuring at least one physiologic activity of a member
a system is currently sensitive to the 70 ft range, and is of population cohort 102 may also include measuring meta
expected to increase in sensitivity to the 10 ft range. See bolic or hemodynamic responses to neural activity. For
US 9,495,684 B2
11 12
example, in positron emission tomography (PET), positrons, lated sinusoid at Some frequency and detected output signal
the antiparticles of electrons, are emitted by certain radio has changes in amplitude and phase; (3) TR-time
nuclides that have the same chemical properties as their resolved—Intime resolve spectroscopy, a very short pulse is
non-radioactive isotopes and that can replace the latter in introduced to be measured and the pulse length is usually on
biologically-relevant molecules. After injection or inhala the order of picoseconds. The detected signal is usually a
tion of tiny amounts of these modified molecules, e.g., longer signal and has a decay time.
modified glucose (FDG) or neurotransmitters, their spatial In one approach, an infrared imager captures an image of
distribution can be detected by a PET-scanner. This device a portion of the user. For example, the imager may capture
is sensitive to radiation resulting from the annihilation of a portion of the user's forehead. Infrared imaging may
emitted positrons when they collide with ubiquitously-pres 10 provide an indication of blood oxygen levels which in turn
ent electrons. Detected distribution information concerning may be indicative of brain activity. With such imaging, the
metabolism or brain perfusion can be derived and visualized infrared imager may produce a signal indicative of brain
in tomograms. Spatial resolution is on the order of about 3-6 activity. According to one method, hemoglobin oxygen
mm, and temporal resolution is on the order of several saturation and relative hemoglobin concentration in a tissue
minutes to fractions of an hour. 15 may be ascertained from diffuse reflectance spectra in the
Functional Near-Infrared Imaging visible wavelength range. This method notes that while
Another method for measuring physiologic activity is oxygenated and deoxygenated hemoglobin contributions to
functional near-infrared imaging (fNIR). fNIR is a spectro light attenuation are strongly variable functions of wave
scopic neuro-imaging method for measuring the level of length, all other contributions to the attenuation including
neuronal activity in the brain. The method is based on scattering are Smooth wavelength functions and can be
neuro-vascular coupling, i.e., the relationship between neu approximated by Taylor series expansion. Based on this
ronal metabolic activity and oxygen level (oxygenated assumption, a simple, robust algorithm Suitable for real time
hemoglobin) in blood vessels in proximity to the neurons. monitoring of the hemoglobin oxygen Saturation in the
Time-resolved frequency-domain spectroscopy (the fre tissue was derived. This algorithm can be used with different
quency-domain signal is the Fourier transform of the origi 25 fiber probe configurations for delivering and collecting light
nal, time-domain signal) may be used in fNIR to provide passed through tissue. See Stratonnikov et al., “Evaluation
quantitation of optical characteristics of the tissue and of blood oxygen saturation in vivo from diffuse reflectance
therefore offer robust information about oxygenation. Dif spectra,” J. Biomed. Optics, vol. 6, pp. 457-467 (2001).
fuse optical tomography (DOT) in fNIR enables researchers Functional Magnetic Resonance Imaging
to produce images of absorption by dividing the region of 30 Another method of measuring at least one physiologic
interest into thousands of Volume units, called Voxels, cal activity may include measuring blood oxygen level depen
culating the amount of absorption in each (the forward dent effects by, for example, functional magnetic resonance
model) and then putting the Voxels back together (the imaging (fMRI). fMRI involves the use of magnetic reso
inverse problem). fNIR systems commonly have multiple nance scanners to produce sets of cross sections—tomo
Sources and detectors, signifying broad coverage of areas of 35 grams—of the brain, detecting weak but measurable reso
interest, and high sensitivity and specificity. fMIR systems nance signals that are emitted by tissue water Subjected to a
today often consist of little more than a probe with fiber very strong magnetic field after excitation with a high
optic sources and detectors, a piece of dedicated hardware frequency electromagnetic pulse. Acquired resonance sig
no larger than a small Suitcase and a laptop computer. Thus, nals can be attributed to their respective spatial origins, and
fNIR systems can be portable; indeed battery operated, 40 cross sectional images can be calculated. The signal inten
wireless continuous wave fNIR devices have been devel sity, often coded as a gray value of a picture element,
oped at the Optical Brain Imaging Lab of Drexel University. depends on water content and certain magnetic properties of
fNIR employs no ionizing radiation and allows for a wide the local tissue. In general, structural MR imaging is used to
range of movement; its possible, for example, for a subject depict brain morphology with good contrast and high reso
to walk around a room while wearing a fNIR probe. fNIR 45 lution. Visualizing brain function by MRI relies on the
studies have examined cerebral responses to visual, auditory relationship between increased neural activity of a brain
and somatosensory stimuli, as well as the motor system and region and increased hemodynamic response or blood flow
language, and Subsequently begun to construct maps of to that brain region. The increased perfusion of activated
functional activation showing the areas of the brain associ brain tissue is the basis of the so-called Blood Oxygenation
ated with particular stimuli and activities. 50 Level Dependent (BOLD)-effect: hemoglobin, the oxygen
For example, a fNIR spectroscopy device (fNIRS) has carrying molecule in blood, has different magnetic proper
been developed that looks like a headband and uses laser ties depending on its oxygenation state. While oxyhemoglo
diodes to send near-infrared light through the forehead at a bin is diamagnetic, deoxyhemoglobin is paramagnetic,
relatively shallow depth e.g., (two to three centimeters) to which means that it locally distorts the magnetic field,
interact with the brains frontal lobe. Light ordinarily passes 55 leading to a local signal loss. In activated brain tissue the
through the body's tissues, except when it encounters oxy increased oxygen consumption is accompanied by a blood
genated or deoxygenated hemoglobin in the blood. Light flow response. Thus, during activation of a brain region,
waves are absorbed by the active, blood-filled areas of the deoxyhemoglobin is partly replaced by oxyhemoglobin,
brain and any remaining light is diffusely reflected to fNIRS leading to less distortion of the local magnetic field and
detectors. See “Technology could enable computers to read 60 increased signal intensity. Color-coded Statistical parametric
the minds of users.” Physorg.com http://www.physorg.com/ activation maps (SPMs) are typically generated from statis
news110463755.html (1 Oct. 2007). tical analyses of fMRI time series comparing signal intensity
There are three types of fNIR: (1) CW—continuous during different activation states.
wave-In this method, infrared light shines at the same Temporal and spatial resolution of fMRI depends on both
intensity level during the measurement period. The detected 65 scanning technology and the underlying physiology of the
signal is lower intensity static signal (dc valued); (2) FD— detected signal intensity changes. Structural images are
frequency domain. In this method, input signal is a modu usually obtained with a resolution of at least 1 mmx1 mmx1
US 9,495,684 B2
13 14
mm Voxels (the equivalent of a pixel in a Volume), while the anterior commissure (AC) and posterior commissure
fMRI voxels typically have edge lengths of about 3-5 mm. (PC)—two relatively invariant fiber bundles connecting the
Temporal resolution of fMRI is on the order of between 1 two hemispheres of the brain. The AC-PC line defines the
and 3 seconds. The cerebral blood flow (CBF) response to a y-axis of the brain coordinate system. The origin is set at the
brain activation is delayed by about 3-6 seconds. There is a AC. The Z-axis is orthogonal to the AC-PC-line in the
balance between temporal and spatial resolution, allowing foot-head direction and passes through the interhemispheric
whole brain scans in less than 3 seconds, and non-invasive fissure. The x-axis is orthogonal to both the other axes and
ness, permitting repeated measurements without adverse points from AC to the right. Any point in the brain can be
events. In addition, the choice of scanning parameters allows identified relative to these axes.
increasing one parameter at the expense of the other. Recent 10
Accordingly, anatomical regions may be identified using
fMRI approaches show that for some neural systems the the Talairach coordinate system or the Talairach daemon
temporal resolution can be improved down to milliseconds
and spatial resolution can be increased to the level of cortical (TD) and the nomenclature of Brodmann. The Talairach
columns as basic functional units of the cortex. daemon is a high-speed database server for querying and
In one embodiment, an fMRI protocol may include fMRI 15 retrieving data about human brain structure over the internet.
data may be acquired with an MRI scanner such as a 3 T The core components of this server are a unique memory
Magnetom Trio Siemens scanner. T2-weighted functional resident application and memory-resident databases. The
MR images may be obtained using axially oriented echo memory-resident design of the TD server provides high
planar imaging. For each Subject, data may be acquired in speed access to its data. This is supported by using TCP/IP
three scanning sessions or functional runs. The first four Sockets for communications and by minimizing the amount
volumes of each session may be discarded to allow for T1 of data transferred during transactions. A TD server data
equilibration effects. For anatomical reference, a high-reso may be searched using X-y-Z coordinates resolved to 1x1x1
lution T1-weighted anatomical image may be obtained. mm Volume elements within a standardized Stereotaxic
Foam cushioning may be placed tightly around the side of space. An array, indexed by X-y-Z coordinates, that spans
the subjects head to minimize artifacts from head motion. 25 170 mm (x), 210 mm (y) and 200 min (Z), provides high
Data preprocessing and statistical analysis may be carried speed access to data. Array dimensions are approximately
out using a statistical parametric mapping function, Such as 25% larger than those of the Co-planar Stereotaxic Atlas of
SPM99 (Statistical Parametric Mapping, Welcome Institute the Human Brain (Talairach and Tournoux, 1988). Coordi
of Cognitive Neurology, London, UK). Individual functional nates tracked by a TD server are spatially consistent with the
images may be realigned, slice-time corrected, normalized 30 Talairach Atlas. Each array location stores a pointer to a
into a standard anatomical space (resulting in isotropic 3 mm relation record that holds data describing what is present at
voxels) and smoothed with a Gaussian kernel of 6 mm. In the corresponding coordinate. Data in relation records are
one embodiment, a standard anatomical space may be based either Structure Probability Maps (SP Maps) or Talairach
on the ICBM 152 brain template (MNI, Montreal Neuro Atlas Labels, though others can be easily added. The relation
logical Institute). A block-design model with a boxcar 35 records are implemented as linked lists to names and values
regressor convoluted with the hemodynamic response func for brain structures. The TD server may be any computing
tion may be used as the predictor to compare activity related device, such as a Sun Sparcstation 20 with 200 Mbytes of
to a stimulus versus a control object. High frequency noise memory. Such a system provides 24-hour access to the data
may be removed using a low pass filter (e.g., Gaussian using a variety of client applications.
kernel with 4.0 s FWHM) and low frequency drifts may be 40 Some commercially available analysis Software such as
removed via a high pass filter. Effects of the conditions for SPM5 (available for download from http://www.fillion.u-
each subject may be compared using linear contrast, result cl.ac.uk/spn/software/spn5/) uses brain templates created
ing in a t-statistic for each voxel. A group analysis may be by the Montreal Neurological Institute (MNI), based on the
carried out on a second level using a whole brain random average of many normal MR brain scans. Although similar,
effect analysis (one-sample t-test). Regions that contain a 45 the Talairach and the MNI templates are not identical, and
minimum of five contiguous voxels thresholded at PK0.001 care should be given to assigning localizations given in MNI
(uncorrected for multiple comparisons) may be considered coordinates correctly to, for example, cytoarchitectonically
to be active. See Schaefer et al., “Neural correlates of defined brain areas like the Brodmann areas (BAs), which
culturally familiar brands of car manufacturers.” NeuroIm are regions in the brain cortex defined in many different
age vol. 31, pp. 861-865 (2006). 50 species based on its cytoarchitecture. Cytoarchitecture is the
Mapping Brain Activity organization of the cortex as observed when a tissue is
When brain activity data are collected from groups of stained for nerve cells. Brodmann areas were originally
individuals, data analysis across individuals may take into referred to by numbers from 1 to 52. Some of the original
account variation in brain anatomy between and among areas have been subdivided further and referred to, e.g., as
individuals. To compare brain activations between individu 55 “23a and “23b.” The Brodmann areas for the human brain
als, the brains are usually spatially normalized to a template include the following:
or control brain. In one approach they are transformed so Areas 1, 2 & 3—Primary Somatosensory Cortex (fre
that they are similar in overall size and spatial orientation. quently referred to as Areas 3, 1, 2 by convention)
Generally, the goal of this transformation is to bring homolo Area 4 Primary Motor Cortex
gous brain areas into the closest possible alignment. In this 60 Area 5–Somatosensory Association Cortex
context the Talairach stereotactic coordinate system is often Area 6 Pre-Motor and Supplementary Motor Cortex
used. The Talairach system involves a coordinate system to (Secondary Motor Cortex)
identify a particular brain location relative to anatomical Area 7 Somatosensory Association Cortex
landmarks; a spatial transformation to match one brain to Area 8 Includes Frontal eye fields
another, and an atlas describing a standard brain, with 65 Area 9 Dorsolateral prefrontal cortex
anatomical and cytoarchitectonic labels. The coordinate Area 10 Frontopolar area (most rostral part of superior
system is based on the identification of the line connecting and middle frontal gyri)
US 9,495,684 B2
15 16
Area 11—Orbitofrontal area (orbital and rectus gyri, plus associated with episodic memory encoding and retrieval.
part of the rostral part of the Superior frontal gyms) The occipital temporal regions of the brain are associated
Area 12 Orbitofrontal area (used to be part of BA11. with vision and visual-spatial processing.
refers to the area between the superior frontal gyms and the Attention
inferior rostral sulcus) Attention can be divided into five categories: Sustained
Area 13–Insular cortex attention, selective attention, Stimulus-Response compat
Area 17 Primary Visual Cortex (V1) ibility, orientation of attention, and division of attention. The
Area 18- Visual Association Cortex (V2) tasks included in the Sustained attention section involved
Area 19 V3 continuous monitoring of different kinds of stimuli (e.g.,
Area 20 Inferior Temporal gyms 10 Somatosensory stimulation). The selective attention section
Area 21—Middle Temporal gyms includes studies in which subjects selectively attended to
Area 22 Superior Temporal Gyms, of which the rostral different attributes of the same set of stimuli (e.g., attend to
part participates to Wernicke's area color only for stimuli varying with respect to both color and
Area 23 Ventral Posterior cingulate cortex shape). The stimulus-response (SR) compatibility section
Area 24 Ventral Anterior cingulate cortex 15 also includes studies examining selective attention, with the
Area 25—Subgenual cortex important difference that they involve a “conflict compo
Area 26-Ectosplenial area nent.” In all cases, this is implemented by employing the
Area 28 Posterior Entorhinal Cortex Stroop task.
Area 29 Retrosplenial cingular cortex Prefrontal and parietal areas, preferentially in the right
Area 30 Part of cingular cortex hemisphere, are frequently engaged during tasks requiring
Area 31—Dorsal Posterior cingular cortex attention. An fMRI study involving a visual vigilance task
Area 32—Dorsal anterior cingulate cortex was in close agreement with the results of a PET study
Area 34—Anterior Entorhinal Cortex (on the Parahip showing predominantly right-sided prefrontal and parietal
pocampal gyms) activation. Observed data is consistent with a right fronto
Area 35—Perirhinal cortex (on the Parahippocampal 25 parietal network for sustained attention. Selective attention
gyms) to one sensory modality is correlated with Suppressed activ
Area 36-Parahippocampal cortex (on the Parahippocam ity in regions associated with other modalities. For example,
pal gyms) studies have found deactivations in the auditory cortex
Area 37—Fusiform gyms during attention area activations. Taken together, the results
Area 38 Temporopolar area (most rostral part of the 30 Suggest the existence of a fronto-parietal network underlying
Superior and middle temporal gyri Sustained attention. Direct Support for fronto-parietal inter
Area 39 Angular gyms, part of Wernicke's area actions during sustained attention has been provided by
Area 40 Supramarginal gyms part of Wernicke's area structural equation modeling of fMRI data. Studies on the
Areas 41 & 42 Primary and Auditory Association Cor effects of attention on thalamic (intralaminar nuclei) and
teX 35 brain stem (midbrain tegmentum) activity have shown that
Area 43—Subcentral area (between insula and post/pre these areas may control the transition from relaxed wake
central gyms) fulness to high general attention.
Area 44 pars opercularis, part of Broca's area Selective attention is characterized by increased activity
Area 45 pars triangularis Broca's area in posterior regions involved in stimulus processing. Dif
Area 46—Dorsolateral prefrontal cortex 40 ferent regions seem to be involved depending on the specific
Area 47—Inferior prefrontal gyms attribute that is attended to. Studies have shown attentional
Area 48 RetroSubicular area (a small part of the medial modulation of auditory regions, and modulation of activity
surface of the temporal lobe) in the lingual and fusiform gyri during a color attention task
Area 52—Parainsular area (at the junction of the temporal has also been demonstrated. Attending to motion activates a
lobe and the insula) 45 region in occipito-temporal cortex, and it has also been
Associating Brain Activity with Brain Function or Mental shown that, in addition to extrastriate regions, attention to
State motion increased activity in several higher-order areas as
The brain performs a multitude of functions. It is the well. It may be that activity in extrastriate regions may be
location of memory, including working memory, semantic modulated by prefrontal, parietal and thalamic regions.
memory, and episodic memory. Attention is controlled by 50 Similarly, modulation of activity in specific posterior
the brain, as is language, cognitive abilities, and visual regions is mediated by regions in parietal and anterior
spatial functions. The brain also receives sensory signals and cingulate cortices, as well as the pulvinar. A role of parietal
generates motor impulses. The frontal lobes of the brain are cortex, especially the inferior parietal lobe, in control of
involved in most higher-level cognitive tasks as well as selective attention has also been suggested. The prefrontal
episodic and semantic memory. There is some degree of 55 cortex may also play a role in attentional modulation. As
lateralization of the frontal lobes, e.g., the right frontal lobe long as attentional load is low, task-irrelevant stimuli are
is a locus for Sustained attention and episodic memory perceived and elicit neural activity, however, when the
retrieval, and the left frontal lobe is a locus for language, attentional load is increased, irrelevant perception and its
semantic memory retrieval, and episodic memory encoding. associated activity is strongly reduced.
The cingulated regions of the brain are associated with 60 The stimulus-response compatibility panel includes selec
memory, initiation and inhibition of behavior, and emotion. tive attention studies on the Stroop test. The Stroop test is
The parietal regions of the brain are associated with atten associated with activations in the anterior cingulate cortex.
tion, spatial perception and imagery, thinking involving time SR compatibility studies point to a role of both the anterior
and numbers, working memory, skill learning, and Success cingulate and the left prefrontal cortex. See Cabeza et al.
ful episodic memory retrieval. The lateral temporal lobe of 65 “Imaging Cognition II: An Empirical Review of 275 PET
the brain is associated with language and semantic memory and fMRI Studies,” J. Cognitive Neurosci., vol. 12, pp. 1-47
encoding and retrieval, while the medial temporal lobe is (2000).
US 9,495,684 B2
17 18
Activation of the thalamic reticular nucleus is also asso involves both pathways to some extent (a similar conclusion
ciated with selective attention. See Contreras et al., “Inac has been drawn based on network analysis of PET data).
tivation of the Interoceptive Insula Disrupts Drug Craving Face perception involves the same ventral pathway as
and Malaise Indicated by Lithium. Science, vol. 318, pp. object perception, but there is a tendency for right-lateral
655-658 (26 Oct. 2007). ization of activations for faces, but not for objects. For
The category "orientation of attention' includes studies example, bilateral fusiform gyms activation is seen for faces,
associating shifts of spatial attention to parietal and prefron but with more extensive activation in the right hemisphere.
tal regions. Another study found activations in Superior Faces are perceived, at least in part, by a separate processing
parietal regions during a visual search for conjunction of stream within the ventral object pathway. In an fMRI study,
features. Based on the similarities in activation patterns, it 10
a region was identified that is more responsive to faces than
appears that serial shifts of attention took place during the to objects, termed the “fusiform face area’’ or FF area.
search task. There is also evidence for a large-scale neural Whereas perception of objects and faces tends to prefer
system for visuospatial attention that includes the right entially activate regions in the ventral visual pathway, per
posterior parietal cortex. PET and fMRI have been
employed to study attentional orienting to spatial locations 15 ception of spatial location tends to selectively activate more
(left VS. right) and to time intervals (short vs. long stimulus dorsal regions located in parietal cortex. Greater activity in
onset times). Both spatial and temporal orienting were found the superior parietal lobe (area 7) as well as in the premotor
to activate a number of brain regions, including prefrontal cortex is seen during location judgments than during object
and parietal brain regions. Other analyses revealed that judgments. The dorsal pathway is not only associated with
activations in the intraparietal Sulcus were right-lateralized space perception, but also with action. For example, per
for spatial attention and left lateralized for temporal atten ception of Scripts of goal-directed hand action engage parts
tion. Moreover, simultaneous spatial and temporal attention of the parietal cortex. Comparison have been done of
activate mainly parietal regions, suggesting that the parietal meaningful actions (e.g., pantomime of opening a bottle)
cortex, especially in the right hemisphere, is a site for and meaningless actions (e.g., signs from the American Sign
interactions between different attentional processes. Parietal 25 Language that were unknown to Subjects). Whereas mean
activation has also been demonstrated in an fMRI study of ingless actions activated the dorsal pathway, meaningful
nonspatial attention shifting. In addition, the cerebellum has actions activated the Ventral pathway. Meaningless actions
been implicated in attention shifting, and this is consistent appear to be decoded in terms of spatiotemporal layout,
with other findings of attentional activation of the cerebel while meaningful actions are processed by areas that allow
lum. It has also been shown that spatial direction of attention 30 semantic processing and memory storage. Thus, as object
can influence the response of the extrastriate cortex. Spe perception, location/action perception may involve both
cifically, it was demonstrated that while multiple stimuli in dorsal and ventral pathways to some extent.
the visual field interact with each other in a suppressive way, Activations in the orbitofrontal cortex (where the second
spatially directed attention partially cancels out the Suppres ary olfactory cortex is located), particularly in the right
sive effects. 35 hemisphere, and the cerebellum are associated with smell
With respect to division of attention, activity in the left ing, as well as increased activity in the primary olfactory
prefrontal cortex increases under divided-attention condi cortex (piriform cortex). Odorants (regardless of Sniffing)
tions. In this context, it is also relevant to mention that if two activate the posterior lateral cerebellum, whereas Sniffing
tasks activate overlapping brain areas, there may be signifi (nonodorized air) activate anterior parts of the cerebellum.
cant interference effects when the tasks are performed simul 40 Thus the cerebellum receives olfactory information for
taneously. See Cabeza et al., “Imaging Cognition II: An modulating Sniffing. Odorants (regardless of Sniffing) acti
Empirical Review of 275 PET and fMRI Studies,” J. Cog vate the anterior and lateral orbitofrontal cortex whereas
nitive Neurosci., vol. 12, pp. 1-47 (2000). Sniffing (even in the absence of odorants) activates the
Perception piriform and medial/posterior orbitofrontal cortices. In sum,
Perception processes can be divided into object, face, 45 smell perception involves primarily the orbitofrontal cortex
space/motion, Smell and “other categories. Object percep and parts of the cerebellum and its neural correlates can be
tion is associated with activations in the Ventral pathway dissociated from those of Sniffing.
(ventral brain areas 18, 19, and 37). The ventral occipito With respect to the “other category, fMRI has been
temporal pathway is associated with object information, employed to define a “parahippocampal place area’’ (PPA)
whereas the dorsal occipito-parietal pathway is associated 50 that responds selectively to passively viewed scenes. A
with spatial information. For example, it has been shown region probably overlapping with PPA responds selectively
that viewing novel, as well as familiar, line drawings, to buildings, and this brain region may respond to stimuli
relative to scrambled drawings, activated a bilateral extras that have orienting value (e.g., isolated landmarks as well as
triate area near the border between the occipital and tem scenes). The neural correlates of music perception have been
poral lobes. Based on these findings, it appears that this area 55 localized to specialized neural systems in the right Superior
is concerned with bottom-up construction of shape descrip temporal cortex, which participate in perceptual analysis of
tions from simple visual features. It has also been shown that melodies. Attention to changes in rhythm activate Broca's/
a region termed the “lateral occipital complex’ (LO) is insular regions in the left hemisphere, pointing to a role of
selectively activated by different kinds of shapes (e.g., this area in the sequencing of auditory input. Further, studies
shapes defined by motion, texture, and luminance contours). 60 of “emotional perception’ Suggest that perception of differ
Greater activity in lingual gyms (Area 19) and/or inferior ent kinds of emotion are based on separate neural systems,
fusiform gyms (Area 37) is seen when subjects make with a possible convergence in prefrontal regions (area 47).
judgments about appearance than when they make judg Consistent with the role of the amygdala in fear condition
ments about locations, providing confirmation that object ing, the amygdala is more activated for fearful faces relative
identity preferentially activates regions in the ventral path 65 to happy faces. See Cabeza et al., “Imaging Cognition II: An
way. Both ventral and dorsal activations during shape-based Empirical Review of 275 PET and fMRI Studies,” J. Cog
object recognition Suggests that visual object processing nitive Neurosci., vol. 12, pp. 1-47 (2000).
US 9,495,684 B2
19 20
Imagery Thus, visual mental imagery is a function of the visual
Imagery can be defined as manipulating sensory informa association cortex, although different association areas seem
tion that comes not from the senses, but from memory. The to be involved depending on the task demands. In addition,
memory representations manipulated can be in working prefrontal areas have been activated in many of the reported
memory (e.g., holding three spatial locations for 3 seconds), comparisons. Partly, these effects may be driven by eye
episodic memory (e.g., retrieving the location of an object in movements (especially for areas 6 and 8), but other factors,
the study phase), or semantic memory (e.g., retrieving the Such as image generation and combination of parts into a
shape of a bicycle). Thus, imagery-related contrasts could be whole, may account for some activations as well.
classified within working memory, episodic retrieval, and Neuroanatomical correlates of motor imagery via a men
semantic retrieval sections. Imagery contrasts can be 10 tal writing task implicate a left parietal region in motor
described as visuospatial retrieval contrasts, and vice versa. imagery, and, more generally, show similarities between
A central issue in the field of imagery has been whether mental writing and actual writing. Similarities between
perception and imagery are seen in both musical imagery
those visual areas that are involved when an object is and perception. For example, relative to a visual baseline
perceived are also involved when an object is imagined. In 15 condition, an imagery task is associated with increased
its strictest form, this idea would imply activation of the activity in the bilateral secondary auditory cortex. This was
primary visual cortex in the absence of any visual input. A so despite the fact that the contrast included two entirely
series of PET experiments provides support for similarities silent conditions. Similarly, a comparison of a task involving
between visual perception and visual imagery by showing imaging a sentence being spoken in another person’s voice
increased blood flow in Area 17 during imagery. In particu with a visual control task reveals left temporal activation.
lar, by comparing tasks involving image formation for Small Activation of the Supplementary motor area was also seen,
and large letters, respectively, these studies provide evidence Suggesting that both input and output speech mechanisms
that imagery activates the topographically mapped primary are engaged in auditory mental imagery. See Cabeza et al.
visual cortex. A subsequent PET study, involving objects of “Imaging Cognition II: An Empirical Review of 275 PET
three different sizes, provides additional support that visual 25 and fMRI Studies,” J. Cognitive Neurosci., vol. 12, pp. 1-47
imagery activates the primary visual cortex. (2000).
Increased activation in extrastriate visual regions is also Language
associated with imaging tasks. The left inferior temporal Language mapping studies are commonly divided into
lobe (area 37) is most reliably activated across subjects (for four categories: spoken and written word recognition
some subjects the activation extended into area 19 of the 30 crossed with spoken or no-spoken response. Word recogni
occipital lobe). Compared with a resting state, a left poste tion, regardless of input modality and whether or not a
rior-inferior temporal region was also activated. Moreover, spoken response is required, has consistently been found to
mental imagery of spoken, concrete words has been shown activate areas 21 and 22 in the temporal cortex. In general,
to activate the inferior-temporal gyrus/fusiform gyms bilat this activation tends to be bilateral, although in the category
erally. Thus, right temporal activation may be related to 35 of written word recognition all activations are left-lateral
more complex visual imagery. ized. The cortical surface covered by these areas is most
Color imagery and color perception engage overlapping likely made up by several distinct regions that can be
networks anterior to region V4 (an area specialized for color functionally dissociated. Involvement of left superior tem
perception), whereas areas V1-V4 were selectively activated poral gyrus/Wernicke's area in word recognition is in agree
by color perception. There is an increase in primary visual 40 ment with the traditional view implicating this area in
cortex activity during negative imagery, as compared to comprehension.
neutral imagery. The primary visual cortex therefore appears Whereas left temporal brain regions have been associated
to have a role in visual imagery, and emotion appears to with word comprehension, left inferior prefrontal cortex/
affect the quality of the image representations. Broca's area has traditionally been linked to word produc
Mental rotation of visual stimuli involves lateral parietal 45 tion. However, comparing conditions involving spoken
areas (BA47 and BA40). The bulk of the computation for response with conditions involving no spoken response do
this kind of mental rotation is performed in the Superior not suggest that (left) prefrontal involvement is greater when
parietal lobe. PET has been employed to study a mental spoken responses are required. Instead, the major difference
rotation task in which subjects were asked to decide whether between these two classes is that conditions involving
letters and digits, tilted in 120°, 180°, or 240°, were in 50 spoken responses tend to activate the cerebellum to a higher
normal or mirror image form. The left parietal cortex is extent. Broca's area is involved in word perception, as well
activated in this task. as in word production, and in addition to having an output
Mental "exploration of maps or routes has been studied function, the left prefrontal areas may participate in recep
using PET, revealing that this task is associated with tive language processing in the uninjured State. An fMRI
increased activity in the right Superior occipital cortex, the 55 study has shown that cerebellar activation is related to the
supplementary motor area (SMA) and the cerebellar vermis. articulatory level of speech production.
The latter two activations are related to eye movements, and Visual areas are more frequently involved in the case of
it appears that the Superior occipital cortex has a specific role written word recognition, and regardless of output (spoken/
in generation and maintenance of visual mental images. In no spoken), written word recognition tends to differentially
a Subsequent PET study, occipital activation was again 60 activate left prefrontal and anterior cingulate regions. More
observed, although this time the peak was in left middle over, left inferior prefrontal activation has been associated
occipital gyms. This activation was specific to a task involv with semantic processing.
ing mental navigation—static visual imagery was not asso A posterior left temporal region (BA37) is a multimodal
ciated with occipital activation. Mental navigation tasks language region. Both blind and sighted Subjects activate
appears to tap visual memory to a high extent, and feedback 65 this area during tactile vs. Visual reading (compared to
influences from areas involved in visual memory may acti non-word letter Strings). This area may not contain linguistic
vate visual (occipital) areas during certain imagery tasks. codes per se, but may promote activity in other areas that
US 9,495,684 B2
21 22
jointly lead to lexical or conceptual access. Area 37 has been nance, such as delayed response tasks. Ventrolateral pre
activated in several studies of written word recognition but frontal regions are involved in simple short-term operations,
not in studies of spoken word recognition. Lip-reading whereas mid-dorsal prefrontal regions perform higher-level
activates the auditory cortex in the absence of auditory executive operations, such as monitoring. Object working
speech sounds. The activation was observed for silent memory may be left-lateralized while spatial-working
speech as well as pseudo-speech, but not for nonlinguistic memory is right-lateralized.
facial movements, suggesting that lip-reading modulates the In addition to prefrontal activations, working memory
perception of auditory speech at a prelexical level. studies normally show activations in parietal regions, par
There are few differences between sign language and ticularly areas 7 and 40. In the case of verbal/numeric tasks,
spoken language, and sign language in bilingual persons 10 these activations tend to be left-lateralized, Suggesting that
activates a similar network as that underlying spoken lan they are related to linguistic operations. The phonological
guage. The difference in activation in Ventral temporal loop consists of a phonological store, where information is
cortex (area 37) related to sign language appears to relate to briefly stored, and a rehearsal process, which refreshes the
an attention mechanism that assigns importance to signing contents of this store. Left parietal activations may reflect
hands and facial expressions. With respect to the processing 15 the phonological store, whereas left prefrontal activations in
of native and foreign languages, native-language processing. area 44 (Broca's area) may reflect the rehearsal process.
relative to processing of a foreign language, selectively When nonverbal materials are employed, parietal activa
activates several brain regions leading to the conclusion that tions, particularly those in area 7, tend to be bilateral, and to
Some brain areas are shaped by early exposure to the occur for spatial but not for object working memory. Thus
maternal language, and that these regions may not be the distinction between a ventral pathway for object pro
activated when people process a language that they have cessing and a dorsal pathway for spatial processing may also
learned later in life. In Broca's area, second languages apply to working memory.
acquired in adulthood are spatially separated from native Working memory tasks are also associated with anterior
languages, whereas second languages acquired at an early cingulate, occipital, and cerebellar activations. Anterior cin
age tend to activate overlapping regions within Broca's area. 25 gulate activations are often found in Area 32, but they may
In Wernicke's area, no separation based on age of language not reflect working memory operations per se. Activity in
acquisition is observed. Further, fMRI has been used to dorsolateral prefrontal regions (areas 9 and 46) varies as a
determine brain activity related to aspects of language function of delay, but not of readability of a cue, and activity
processing. During phonological tasks, brain activation in in the anterior cingulate (and in Some right ventrolateral
males was lateralized to the left inferior frontal gyms, 30 prefrontal regions) varies as a function of readability but not
whereas the pattern was more diffuse for females. of delay of a cue. Thus, the anterior cingulate activation
Activation patterns related to the processing of particular seems to be related to task difficulty, rather than to working
aspects of information show that a set of brain regions in the memory per se. Occipital activations are usually found for
right hemisphere is selectively activated when subjects try to visuospatial tasks, and may reflect increased visual attention
appreciate the moral of a story as opposed to semantic 35 under working memory conditions. Cerebellar activations
aspects of the story. Brain activation associated with Syn are common during verbal working memory tasks, particu
tactic complexity of sentences indicates that parts of Broca's larly for tasks involving phonological processing (e.g., hold
area increase their activity when sentences increase in ing letters) and tasks that engage Broca's area (left area 44).
Syntactic complexity. See Cabeza et al., “Imaging Cognition Consistent with the idea that mid-dorsal areas 9/46 are
II: An Empirical Review of 275 PET and fMRI Studies,” J. 40 involved in higher-level working memory operations, acti
Cognitive Neurosci., vol. 12, pp. 1-47 (2000). Vations in these areas are prominent in the reasoning and
Working Memory planning tasks. Area 10 activations are also quite prevalent,
Working memory consists of three main components: a and may be related to episodic memory aspects of problem
phonological loop for the maintenance of Verbal informa Solving tasks (see episodic memory retrieval section above).
tion, a visuospatial sketchpad for the maintenance of visu 45 Tasks involving sequential decisions, such as conceptual
ospatial information, and a central executive for attentional reasoning and card Sorting consistently engage the basal
control. Dozens of functional neuroimaging studies of work ganglia, thalamic, and cerebellar regions. These regions are
ing memory have been carried out. Working memory is typical skill learning regions and may reflect the skill
associated with activations in prefrontal, parietal, and cin learning aspects of sequential problem-solving tasks. Also,
gulate regions. There also may be involvement of occipital 50 the basal ganglia, thalamus, and prefrontal cortex are inti
and cerebellar regions discriminations between different mately linked and dysfunction of this circuitry could under
Brodmann’s areas. lie planning deficits in Parkinson disease. See Cabeza et al.
Working memory is almost always associated with “Imaging Cognition II: An Empirical Review of 275 PET
increased activity in the prefrontal cortex. This activity is and fMRI Studies,” J. Cognitive Neurosci., vol. 12, pp. 1-47
typically found in areas 6, 44, 9 and 46. Area 44 activations 55 (2000).
are more prevalent for verbal/numeric tasks than for visu Semantic Memory Retrieval
ospatial tasks, and tend to be lateralized to the left hemi Semantic memory refers to knowledge we share with
sphere (i.e., Broca's area), Suggesting that they reflect pho other members of our culture, such as knowledge about the
nological processing. Area 6 activations are common for meaning of words (e.g., a banana is a fruit), the properties of
Verbal, spatial, and problem-solving tasks, and, hence, they 60 objects (e.g., bananas are yellow), and facts (e.g., bananas
are likely related to general working memory operations grow in tropical climates). Semantic memory may be
(i.e., they are not material or task-specific). In contrast, divided into two testing categories, categorization tasks and
activations in areas 9 and 46 seem to occur for certain kinds generation tasks. In categorization tasks, Subjects classify
of working memory tasks but not others. Activations in these words into different categories (e.g., living vs. nonliving),
two areas tend to be more prevalent for tasks that require 65 whereas in generation tasks, they produce one (e.g., word
manipulation of working memory contents, such as N-back stem completion) or several (for example, fluency tasks)
tasks, than for tasks that require only uninterrupted mainte words in response to a cue. Semantic memory retrieval is
US 9,495,684 B2
23 24
associated with activations in prefrontal, temporal, anterior left prefrontal regions. Occipital activations could reflect the
cingulate, and cerebellar regions. processing of the subtle differences in physical features that
Prefrontal activity during semantic memory tasks fre distinguish animals, whereas prefrontal activations could be
quently found in the left hemisphere but not in the right. This related to linguistic or motor aspects of tool utilization.
is so even when the stimuli are nonverbal materials. Such as Animal knowledge activates a more anterior region (area 21)
objects and faces. This striking left-lateralization is in sharp of the inferior temporal lobe than the one associated with
contrast with the right-lateralization of prefrontal activity tool knowledge (area 37). Whereas generating color words
typically observed during episodic memory retrieval. This activates fusiform areas close to color perception regions,
asymmetric pattern has been conceptualized in terms of a generating action words activates a left temporo-occipital
hemispheric encoding/retrieval asymmetry (HERA) model. 10 area close to motion perception regions. Thus knowledge
This model consists of three hypotheses: (1) the left pre about object attributes is stored close to the regions involved
frontal cortex is differentially more involved in semantic in perceiving these attributes. See Cabeza et al., “Imaging
memory retrieval than is the right prefrontal cortex; (2) the Cognition II: An Empirical Review of 275 PET and fMRI
left prefrontal cortex is differentially more involved in Studies. J. Cognitive Neurosci., vol. 12, pp. 1-47 (2000).
encoding information into episodic memory than is the right 15 Episodic Memory Encoding
prefrontal cortex; and (3) the right prefrontal cortex is Episodic memory refers to memory for personally expe
differentially more involved in episodic memory retrieval rienced past events, and it involves three Successive stages:
than is the left prefrontal cortex. Thus, the left-lateralization encoding, storage, and retrieval. Encoding refers to pro
of prefrontal activations supports the first hypothesis of the cesses that lead to the formation of new memory traces.
model. The second and third hypotheses are addressed by Storage designates the maintenance of memory traces over
episodic memory encoding and episodic memory retrieval time, including consolidation operations that make memory
testing, respectively, as discussed above. traces more permanent. Retrieval refers to the process of
Within the frontal lobes, activations are found in most accessing stored memory traces. Encoding and retrieval
prefrontal regions, including Ventrolateral (areas 45 and 47), processes are amenable to functional neuroimaging
Ventromedial (area 11), posterior (areas 44 and 6), and 25 research, because they occur at specific points in time,
mid-dorsal (areas 9 and 46) regions. Activations in Ventro whereas storage/consolidation processes are not, because
lateral regions occur during both classification and genera they are temporally distributed. It is very difficult to differ
tion tasks and under a variety of conditions, Suggesting that entiate the neural correlates of encoding and retrieval on the
they are related to generic semantic retrieval operations. In basis of the lesion data, because impaired memory perfor
contrast, area 11 activations are more common for classifi 30 mance after brain damage may reflect encoding deficits,
cation than for generation tasks, and could be related to a retrieval deficits, or both. In contrast, functional neuroim
component of classification tasks, such as decision-making aging allows separate measures of brain activity during
Conversely, activations in posterior and dorsal regions are encoding and retrieval.
more typical for generation tasks than for classification Episodic encoding can be intentional, when Subjects are
tasks. Many posterior activations (areas 44 and 6) occur at 35 informed about a Subsequent memory test, or incidental,
or near Broca's area, thus they may reflect overt or covert when they are not. Incidental learning occurs, for example,
articulatory processes during word generation. Activations when Subjects learn information while performing a seman
in dorsal regions (areas 9 and 46) are particularly frequent tic retrieval task, such as making living/nonliving decisions.
for fluency tasks. Because fluency tasks require the moni Semantic memory retrieval and incidental episodic memory
toring of several items in working memory, these activations 40 encoding are closely associated. Semantic processing of
may reflect working memory, rather than semantic memory, information (semantic retrieval) usually leads to Successful
per se. Accordingly, when Subjects complete word stems, storage of new information. Further, when subjects are
areas 9/10 are more active for stems with many completions instructed to learn information for a Subsequent memory test
than for stems with few completions. These areas may (intentional encoding), they tend to elaborate the meaning of
therefore be involved in selecting among competing candi 45 the information and make associations on the basis of their
date responses. knowledge (semantic retrieval). Thus, most of the regions
Semantic retrieval tasks are also commonly associated (for example, left prefrontal cortex) associated with seman
with temporal, anterior cingulate, and cerebellar regions. tic retrieval tasks are also associated with episodic memory
Temporal activations occur mainly in the left middle tem encoding.
poral gyms (area 21) and in bilateral occipito-temporal 50 Episodic encoding is associated primarily with prefrontal,
regions (area 37). Left area 21 is activated not only for words cerebellar, and medial temporal brain regions. In the case of
but also pictures and faces, Suggesting it is involved in verbal materials, prefrontal activations are always left lat
higher-level semantic processes that are independent of eralized. This pattern contrasts with the right lateralization
input modality. In contrast, area 37 activations are more of prefrontal activity during episodic retrieval for the same
common for objects and faces, so they could be related to the 55 kind of materials. In contrast, encoding conditions involving
retrieval of visual properties of these stimuli. Anterior cin nonverbal stimuli sometimes yield bilateral and right-later
gulate activations are typical for generation tasks. The alized activations during encoding. Right-lateralized encod
anterior cingulate—like the dorsal prefrontal cortex —is ing activations may reflect the use of non-nameable stimuli,
more active for stems with many than with few completions, Such as unfamiliar faces and textures, but encoding of
whereas the cerebellum shows the opposite pattern. The 60 non-nameable stimuli has been also associated with left
anterior cingulate may therefore be involved in selecting lateralized activations with unfamiliar faces and locations.
among candidate responses, while the cerebellum may be Contrasting encoding of Verbal materials with encoding of
involved in memory search processes. Accordingly cerebel nonverbal materials may speak to the neural correlates of
lar activations are found during single-word generation, but different materials rather than to the neural correlates of
not during fluency tasks. 65 encoding per se.
The retrieval of animal information is associated with left The prefrontal areas most commonly activated for verbal
occipital regions and the retrieval of tool information with materials are areas 44, 45, and 9/46. Encoding activations in
US 9,495,684 B2
25 26
left area 45 reflects semantic processing while those in left activation with task-related rather than item-related activity
area 44 reflects rote rehearsal. Areas 9/46 may reflect during episodic retrieval. Activations associated with
higher-order working memory processes during encoding. retrieval effort show a tendency to be left lateralized, spe
Activation in left area 9 increases as a function of organi cifically in left areas 47 and 10. Bilateral Areas 10,9, and 46
Zational processes during encoding, and is attenuated by are sometimes associated with retrieval success. Prefrontal
distraction during highly organizational tasks. Cerebellar activity is also seen to increase with Success activations
activations occur only for verbal materials and show a when subjects are warned about the proportion of old and
tendency for right lateralization. The left-prefrontal/right new items during the scan (biasing).
cerebellum pattern during language, Verbal-semantic Medial-temporal activations have been seen in the typical
memory, and Verbal-episodic encoding tasks is consistent 10 pattern of episodic retrieval in PET and fMRI studies, for
with the fact that fronto-cerebellar connections are crossed. both verbal and nonverbal materials. In contrast with
Medial-temporal activations are seen with episodic medial-temporal activations during episodic encoding, those
memory encoding and can predict not only what items will during episodic retrieval tend to occur in both hemispheres,
be remembered, but also how well they will be remembered. regardless of the materials employed. That they are some
Medial-temporal activations show a clear lateralization pat 15 times found in association with retrieval Success, but never
tern: they are left-lateralized for verbal materials and bilat in association with retrieval effort or retrieval mode, Suggest
eral for nonverbal materials. Under similar conditions, that they are related to the level of retrieval performance.
medial-temporal activity is stronger during the encoding of Medial-temporal activity increases as linear function of
pictures than during the encoding of words, perhaps explain correct old word recognition, and this activity may reflect
ing why pictures are often remembered better than words. In Successful access to stored-memory representations. Further,
the case of nonverbal materials, medial-temporal activity hippocampal activity has been associated with conscious
seems to be more pronounced for spatial than for nonspatial recollection. Hippocampal activity is also sensitive to the
information, consistent with the link between the hippocam match between study and test conditions, such as the orien
pus and spatial mapping shown by animal research. See tation of study and test objects. However, recollection need
Cabeza et al., “Imaging Cognition II: An Empirical Review 25 not be accurate; for example in the case of significant
of 275 PET and fMRI Studies,” J. Cognitive Neurosci., vol. hippocampal activations during the recognition of false
12, pp. 1-47 (2000). targets. Accurate recognition yields additional activations in
Episodic Memory Retrieval a left temporoparietal region, possibly reflecting the retrieval
Episodic memory retrieval refers to the search, access, of sensory properties of auditorily studied words. Further,
and monitoring of stored information about personally expe 30 intentional retrieval is not a precondition for hippocampal
rienced past events, as well as to the Sustained mental set activity; activations in this area are found for old informa
underlying these processes. Episodic memory retrieval is tion encountered during a non-episodic task, Suggesting that
associated with seven main regions: prefrontal, medial tem they can also reflect spontaneous reminding of past events.
poral, medial parieto-occipital, lateral parietal, anterior cin After the right prefrontal cortex, the most typical region
gulate, occipital, and cerebellar regions. 35 in PET/fMRI studies of episodic retrieval is the medial
Prefrontal activations during episodic memory retrieval parieto-occipital area that includes retrosplenial (primarily
are sometimes bilateral, but they show a clear tendency for areas 29 and 30), precuneus (primarily medial area 7 and
right-lateralization. The right lateralization of prefrontal area 31), and cuneus (primarily medial areas 19, 18, and 17)
activity during episodic memory retrieval contrasts with the regions. The critical role of the retrosplenial cortex in
left lateralization of prefrontal activity during semantic 40 memory retrieval is Supported by evidence that lesions in
memory retrieval and episodic memory encoding. Left pre this region can cause severe memory deficits (e.g., retro
frontal activations during episodic retrieval tend to occur for splenial amnesia. The role of the precuneus has been attrib
tasks that require more reflectively complex processing. uted to imagery and to retrieval Success. Retrieval-related
These activations may be related to semantic retrieval pro activations in the precuneus are more pronounced for image
cesses during episodic retrieval. Semantic retrieval can aid 45 able than for nonimageable words. However, the precuneus
episodic retrieval particularly during recall, and bilateral region was not more activated for object recall than for word
activations tend to be more frequent during recall than recall. Imagery-related activations are more anterior than
during recognition. Moreover, left prefrontal activity during activations typically associated with episodic retrieval. The
episodic retrieval is associated with retrieval effort, and is precuneus is activated for both imageable and abstract
more common in older adults than in young adults. 50 words, and for both visual and auditory study presentations.
Prefrontal activity changes as a function of the amount of Thus this region appears to be involved in episodic retrieval
information retrieved during the scan have been measured irrespective of imagery content. The precuneus cortex is
by varying encoding conditions (e.g., deep vs. shallow), or more active in a high-target than in low-target recognition
by altering the proportion of old items (e.g., targets) during condition.
the Scan. As more information is retrieved during the Scan, 55 Episodic memory retrieval is also associated with activa
prefrontal activity may increase (retrieval Success), decrease tions in lateral parietal, anterior cingulate, occipital, and
(retrieval effort), or remain constant (retrieval mode). These cerebellar regions. Lateral parietal regions have been asso
three outcomes are not necessarily contradictory; they may ciated with the processing of spatial information during
correspond to three different aspects of retrieval: maintain episodic memory retrieval and with the perceptual compo
ing an attentional focus on a particular past episode (re 60 nent of recognition. Anterior cingulate activations (areas 32
trieval mode), performing a demanding memory search and 24) have been associated with response selection and
(retrieval effort), and monitoring retrieved information (re initiation of action. Anterior cingulate activations may be
trieval Success). related to language processes because they are more fre
These different aspects of retrieval may map to distinct quent for verbal than for nonverbal materials. As expected,
prefrontal regions. The region most strongly associated to 65 occipital activations are more common during nonverbal
retrieval mode is the right anterior prefrontal cortex (area retrieval, possibly reflecting not only more extensive pro
10). A combined PET/ERP study associated a right area 10 cessing of test stimuli but also memory-related imagery
US 9,495,684 B2
27 28
operations. Cerebellar activations have been associated with regions, including fusiform gyms and cuneus. Thus, prim
self-initiated retrieval operations. This idea of initiation is ing-related reductions in activity in visual areas occur even
consistent with the association of cerebellar activations with after Subliminal presentation.
retrieval mode and effort, rather than with retrieval success. Priming cannot only facilitate perceptual processes, but
With respect to context memory, a fusiform region is more may also influence conceptual processes. The primed con
active for object identity than for location retrieval, whereas dition is associated with decreased activity in several
an inferior parietal region shows the opposite pattern. Thus regions, including the left inferior prefrontal cortex. Simi
the ventral/dorsal distinction applies also to episodic larly, several fMRI studies that have included repeated
retrieval. In the time domain, recognition memory (what) 10
semantic processing of the same items have found reduced
has been contrasted with recency memory (when). Medial left prefrontal activation associated with the primed condi
temporal regions are more active during item memory than tion. Left prefrontal reduction of activation is not seen when
during temporal-order memory, whereas dorsal prefrontal words are non-semantically reprocessed, Suggesting that the
and parietal regions are more active during temporal-order effect reflects a process-specific change (not a consequence
memory than during item memory. Parietal activations dur 15 of mere repeated exposure). This process-specific effect can
ing temporal-order memory Suggest that the dorsal pathway be obtained regardless of the perceptual format of the stimuli
may be associated not only with “where' but also with (e.g., pictures or words). Many memory tests rely upon a
“when. mixture of processes, and even the stem-completion task,
Prefrontal regions were similarly activated in both recall which has been used in several studies of perceptual prim
and recognition tests. This may signify the use of associative ing, has been associated with priming-related left prefrontal
recognition—a form of recognition with a strong recollec reductions. This may be taken as evidence that this task, too,
tion component, or to the careful matching of task difficulty taps both perceptual and conceptual processes.
in the two tests. A comparison of free and cued recall found With respect to a neural correlate of priming, repeating
a dissociation in the right prefrontal cortex between dorsal items during performance of the same task, or even during
cortex (areas 9 and 46), which is more active during free 25 performance of different tasks, can lead to decreases in the
recall, and the ventrolateral cortex (area 47/frontal insula), amount of activation present in specific brain areas. This
which is more active during cued recall. Thus some of the effect may reflect enhanced processing of the involved
activations observed during episodic-memory retrieval tasks neurons or/and a specification of the involved neuronal
may reflect the working-memory components of these tasks. population, resulting in a spatially less diffuse response. See
Autobiographic retrieval is associated with activations along 30
Cabeza et al., “Imaging Cognition II: An Empirical Review
a right fronto-temporal network. of 275 PET and fMRI Studies,” J. Cognitive Neurosci., vol.
Episodic memory retrieval is associated with activations 12, pp. 1-47 (2000).
in prefrontal, medial temporal, posterior midline, parietal, Procedural Memory
anterior cingulate, occipital, and cerebellar regions. Prefron
tal activations tend to be right-lateralized, and have been 35 Procedural memory processes can be divided into three
associated with retrieval mode, retrieval effort, and retrieval Subcategories: conditioning, motor-skill learning, and non
Success. The engagement of medial temporal regions has motor skill learning. With respect to conditioning, studies on
been linked to retrieval success and recollection. Posterior eye-blink conditioning point to a consistent role of the
midline activations also seem related to retrieval Success. cerebellum in this form of learning (e.g., decreased activity
Parietal activations may reflect processing of spatial context, 40 in the cerebellum following conditioning). Conditioning is
and anterior cingulate activations may reflect selection/ also associated with increased activity in the auditory cortex.
initiation processes. Cerebellar involvement has been attrib Motor-skill learning is associated with activation of motor
uted to self-initiated retrieval. Spatial retrieval engaged regions. Area 6 is involved, and learning-related changes
parietal regions, and object retrieval activated temporal have also repeatedly been demonstrated in the primary
regions. Parietal regions are also activated during temporal 45 motor cortex (area 4). The size of the activated area in the
order retrieval, Suggesting a general role in context memory. primary motor cortex increases as a function of training.
See Cabeza et al., “Imaging Cognition II: An Empirical There is also parietal involvement in motor skill learning;
Review of 275 PET and fMRI Studies,” J. Cognitive Neu fronto-parietal interactions may underlie task performance.
rosci., vol. 12, pp. 1-47 (2000). With respect to nonmotor skill learning, cerebellar activation
Priming 50 is observed across tasks, as is consistent involvement of
Priming can be divided into perceptual and conceptual parietal brain regions. This is in line with the pattern
priming. In several studies, perceptual priming has been observed for motor-skill learning, and the overlap in acti
explored by studying completion of word-stems. In the Vation patterns may reflect common processes underlying
primed condition, it is possible to complete the stems with these two forms of procedural memory. See Cabeza et al.
previously presented words, whereas this is not possible in 55 “Imaging Cognition II: An Empirical Review of 275 PET
the unprimed condition. Visual perceptual priming is asso and fMRI Studies,” J. Cognitive Neurosci., vol. 12, pp. 1-47
ciated with decreased activity in the occipital cortex. PET (2000).
and fMRI studies on non-verbal visual perceptual priming Preference
have revealed priming-related reduction in activation of Neural correlates of preference can be detected through
regions in the occipital and inferior temporal brain regions. 60 neuroimaging studies. For example, in a simulated buying
Priming effects can persist over days; repetition priming decision task between similar fast moving consumer goods,
(item-specific learning) as measured by fMRI shows that only a subject’s preferred brand elicited a reduced activation
learning-related neural changes that accompany these forms in the dorsolateral prefrontal, posterior parietal and occipital
of learning partly involve the same regions. cortices and the left premotor area (Brodmann areas 9, 46.
Comparisons of blood flow responses associated with 65 7/19 and 6), and only when the target brand was the subjects
novel VS. familiar stimuli (across memory tasks) show that favorite one. Simultaneously, activity was increased in the
novel stimuli are associated with higher activity in several inferior precuneus and posterior cingulated (BA 7), right
US 9,495,684 B2
29 30
Superior frontal gyms (BA 10), right Supramarginal gyms Emotion
(BA40) and most pronounced in the ventromedial prefrontal Various emotions may be identified through detection of
cortex (“VMPFC, BA 10). brain activity. As discussed below, activation of the anterior
In fMRI analyses, activation of the nucleus accumbens is insula has been associated with pain, distress, and other
associated with product preference, and the medial prefron negative emotional States. Conversely, as discussed below,
tal cortex is associated with evaluation of gains and losses. positive emotional processes are reliably associated with a
When these areas of the brain are activated, subjects bought series of structures representing a reward center, including
a product at an accuracy rate of 60%. In other fMRI the striatum and caudate, and areas of the midbrain and
analyses, early stage romantic love has been associated with 10
cortex to which they project, Such as the Ventromedial
activation of Subcortical reward regions such as the right prefrontal cortex, orbitofrontal cortex, and anterior cingu
Ventral tegmental area and the dorsal caudate area. Subjects lated cortex, as well as other areas Such as the amygdala and
in more extended romantic love showed more activity in the the insula.
ventral pallidum. In still another fMRI analysis, in subjects In addition, approval and/or disapproval may be deter
experiencing a mistake, activation of the rostral anterior 15 mined based on brain activity. For example, in an fMRI
cingulated cortex increased in proportion to a financial study, blood-oxygen-level-dependent signal changes were
penalty linked to the mistake. See Wise, “Thought Police: measured in Subjects viewing facial displays of happiness,
How Brain Scans Could Invade Your Private Life.” Popular sadness, anger, fear, and disgust, as well as neutral faces.
Mechanics, (November 2007). Subjects were tasked with discriminating emotional valence
With respect to brand discrimination, brain activations in (positive versus negative) and age (over 30 versus under 30)
product choice differ from those for height discrimination, of the faces. During the task, normal Subjects showed
and there is a positive relationship between brand familiarity activation in the fusiform gyms, the occipital lobe, and the
and choice time. Neural activation during choice tasks inferior frontal cortex relative to the resting baseline condi
involves brain areas responsible for silent vocalization. tion. The increase was greater in the amygdala and hip
Decision processes take approximately 1 second as mea 25 pocampus during the emotional Valence discrimination task
Sured by magnetoencephalography and can be seen as two than during the age discrimination task. See Gur et al., “An
halves. The first period involves gender-specific problem fMRI study of Facial Emotion Processing in Patients with
recognition processes, and the second half concerns the Schizophrenia.” Am. J. Psych., vol. 159, pp. 1992-1999
choice itself (no gender differences). MEG measurements 30
(2002).
can be categorized in four stages: Frustration is associated with decreased activation in the
Stage 1 V (visual): Activation of the primary visual Ventral striatum, and increased activation in the anterior
cortices at around 90 ms after stimulus onset. insula and the right medial prefrontal cortex by fMRI. See
Stage 2 T (temporal): Neuronal activity predominantly Kenning et al., “Neuroeconomics: an overview from an
over left anterior-temporal and middle-temporal cortices at 35 economic perspective.” Brain Res. Bull., vol. 67, pp. 343
approximately 325 ms after stimulus onset. Some specific 354 (2005).
activity was also found over the left frontal and right Fairness, Altruism and Trust
extra-striate cortical areas. fMRI has been used to show that perceived unfairness
Stage 3—F (frontal): Activation of the left inferior frontal correlates with activations in the anterior insula and the
cortices at about 510 ms after stimulus onset. These signals 40 dorsolateral, prefrontal cortex (DLPFC). Anterior insula
are consistent with activation of Broca's speech area. activation is consistently seen in neuroimaging studies
Stage 4 P (parietal): Activation of the right posterior focusing on pain and distress, hunger and thirst, and auto
parietal cortices (P) at around 885 ms after stimulus onset. nomic arousal. Activation of the insula has also been asso
Male brain activity differed from female in the second ciated with negative emotional States, and activation in the
stage (T) but not in the other three stages (V. F and P). Left 45 anterior insula has been linked to a negative emotional
anterior temporal activity is present in both groups, but response to an unfair offer, indicating an important role for
males seem to activate right hemispherical regions much emotions in decision-making.
more strongly during memory recall than females do. As In contrast to the insula region, the DLPFC has been
noted above, response times also differed for male and linked to cognitive processes such as goal maintenance and
female subjects. See Amber et al., “Salience and Choice: 50 executive control. Thus, DLPFC activation may indicate
Neural Correlates of Shopping Decisions.” Psychology & objective recognition of benefit despite an emotional per
Marketing, Vol. 21(4), pp. 247-261 (April 2004). ception of unfairness.
In an fMRI study, a consistent neural response in the Event-related hyperscan-fMRI (“hfMRI” which means
ventromedial prefrontal cortex was associated with subjects that two Volunteers are measured parallel in two scanners)
behavioral preferences for sampled anonymized beverages. 55 has been used to measure the neural correlates of trust. By
In a brand-cued experiment, brand knowledge of one of the this method, the caudate nucleus has been shown to be
beverages had a dramatic influence on expressed behavioral involved in trust-building and reciprocity in economic
preferences and on the measured brain responses. See Ken exchange. The caudate nucleus is commonly active when
ning et al., “Neuroeconomics: an overview from an eco learning about relations between stimuli and responses. See
nomic perspective.” Brain Res. Bull., vol. 67, pp. 343-354 60 Kenning et al., “Neuroeconomics: an overview from an
(2005). economic perspective.” Brain Res. Bull., vol. 67, pp. 343
In an fMRI study, only the presence of a subject's favorite 354 (2005).
brand indicating a distinctive mode of decision-making was In a PET study, sanctions against defectors were associ
associated with activation of regions responsible for inte ated with activity in reward-processing brain regions. See
grating emotions. See Kenning et al., “Neuroeconomics: an 65 Kenning et al., “Neuroeconomics: an overview from an
overview from an economic perspective.” Brain Res. Bull. economic perspective.” Brain Res. Bull., vol. 67, pp. 343
vol. 67, pp. 343-354 (2005). 354 (2005).
US 9,495,684 B2
31 32
Reward Regarding lateralization, prefrontal activations during
In an fMRI study, activation changes in the sublenticular language, semantic memory retrieval, and episodic memory
extended amygdala (SLEA) and orbital gyms were associ encoding are usually left-lateralized, those during Sustained
ated with expected values of financial gain. Responses to attention and episodic retrieval are mostly right-lateralized,
actual experience of rewards increased monotonically with 5 and those during working memory are typically bilateral.
monetary value in the nucleus accumbens, SLEA, and With respect to distinctions between different prefrontal
thalamus. Responses to prospective rewards and outcomes areas, Ventrolateral regions (areas 45 and 47) are involved in
were generally, but not always, seen in the same regions. selecting, comparing, or deciding on information held in
Overlaps with activation changes seen previously in short-term and long-term memory, whereas mid-dorsal
response to tactile stimuli, gustatory stimuli, and euphoria 10 regions (areas 9 and 46) are involved when several pieces of
indicating drugs were found. See Kenning et al., “Neuro information in working memory need to be monitored and
economics: an overview from an economic perspective.” manipulated. Area 45/47 activations were found even in
Brain Res. Bull., vol. 67, pp. 343-354 (2005). simple language tasks, while activations in areas 9/46 were
In another fMRI study, within a group of cooperative associated with working memory and episodic encoding and
Subjects the prefrontal cortex showed activation changes 15 retrieval. However, areas 9/46 were also activated during
when Subjects playing a human compared to playing a Sustained attention tasks, which do not involve the simulta
computer. Within a group of non-cooperators, no significant neous consideration of several pieces of information. See
activation changes in the prefrontal cortex were seen Cabeza et al., “Imaging Cognition II: An Empirical Review
between computer and human conditions. See Kenning et of 275 PET and fMRI Studies,” J. Cognitive Neurosci., vol.
al., “Neuroeconomics: an overview from an economic per 12, pp. 1-47 (2000).
spective.” Brain Res. Bull., vol. 67, pp. 343-354 (2005). Humans restrain self-interest with moral and social val
In an fMRI study, products symbolizing wealth and status ues. They are the only species known to exhibit reciprocal
were associated with increased activity in reward-related fairness, which implies the punishment of other individuals
brain areas. See Kenning et al., “Neuroeconomics: an over unfair behaviors, even if it hurts the punisher's economic
view from an economic perspective.” Brain Res. Bull., vol. 25 self-interest. Reciprocal fairness has been demonstrated in
67, pp. 343-354 (2005). the Ultimatum Game, where players often reject their bar
In a PET study, participants were risk averse in gains and gaining partner's unfair offers. It has been shown that
risk-seeking in losses; and ambiguity-seeking in neither disruption of the right, but not the left, dorsolateral prefron
gains nor losses. Interactions between attitudes and beliefs tal cortex (DLPFC) by low-frequency repetitive transcranial
were associated with neural activation changes in dorsome 30 magnetic stimulation Substantially reduces Subjects will
dial and Ventromedial brain areas. See Kenning et al., ingness to reject their partners’ intentionally unfair offers,
“Neuroeconomics: an overview from an economic perspec which suggests that subjects are less able to resist the
tive.” Brain Res. Bull., vol. 67, pp. 343-354 (2005). economic temptation to accept these offers. Importantly,
In an fMRI study, increasing monetary gains were asso however, Subjects still judge Such offers as very unfair,
ciated with increased activity in a Subcortical region of the 35 which indicates that the right DLPFC plays a key role in the
Ventral striatum in a magnitude-proportional manner. This implementation of fairness-related behaviors. See Knoch et
Ventral striatal activation was not evident during anticipation al., “Diminishing Reciprocal Fairness by Disrupting the
of losses. Actual gain outcomes were associated with acti Right Prefrontal Cortex,” Science, vol. 314, pp. 829-832 (3
Vation of a region of the medial prefrontal cortex. During Nov. 2006).
anticipation of gain, Ventral striatal activation was associ 40 Differences across tasks can be found in frontopolar (area
ated with feelings characterized by increasing arousal and 10), opercular (area 44), and dorsal (areas 6 and 8) prefrontal
positive valence. See Kenning et al., “Neuroeconomics: an regions. Frontopolar activations were typical for episodic
overview from an economic perspective.” Brain Res. Bull. memory retrieval and problem-solving tasks. In the case of
vol. 67, pp. 343-354 (2005). episodic retrieval, they are found for both retrieval success
In an fMRI study, activation of parts of the limbic system 45 and retrieval mode, Suggesting they are probably not related
were associated with decisions involving immediate to performance level or task difficulty. Area 10 is involved
rewards. Activity changes in the lateral prefrontal cortex and in maintaining the mental set of episodic retrieval, but also
posterior parietal cortex were associated with inter-temporal has an involvement in problem-solving tasks. Activations in
choices. Greater relative fronto-parietal activity was associ left area 44, which corresponds to Broca's area, were
ated with a Subject's choice of longer term options. See 50 commonly found for reading, verbal working memory and
Kenning et al., “Neuroeconomics: an overview from an semantic generation. Right area 44 is engaged by nonverbal
economic perspective.” Brain Res. Bull., vol. 67, pp. 343 episodic retrieval tasks. Area 6 plays a role in spatial
354 (2005). processing (orientation of attention, space/motion percep
Brain Activation By Region tion and imagery), working memory, and motor-skill learn
Prefrontal Regions 55 ing. Midline area 6 activations correspond to SMA and are
The prefrontal cortex is involved in almost all high-level common for silent reading tasks. Area 8 is involved in
cognitive tasks. Prefrontal activations are particularly problem-solving tasks, possibly reflecting eye movements.
prominent during working memory and memory retrieval See Cabeza et al., “Imaging Cognition II: An Empirical
(episodic and semantic), and less prevalent during percep Review of 275 PET and fMRI Studies,” J. Cognitive Neu
tion and perceptual priming tasks. This pattern is consistent 60 rosci., vol. 12, pp. 1-47 (2000).
with the idea that the prefrontal cortex is involved in The frontopolar cortex has been shown to be active during
working memory processes, such as monitoring, organiza the initial stages of learning, gradually disengaging over the
tion, and planning. However, some of the same prefrontal course of learning. Frontopolar cortex activity specifically
regions engaged by working tasks are also recruited by correlates with the amount of uncertainty remaining between
simple detection tasks that do not involve a maintenance 65 multiple putative options that Subjects are simultaneously
component. Thus the prefrontal cortex is not devoted solely tracking. The frontopolar cortex is also active whenever
to working memory operations. Subjects depart from an a priori optimal option to check
US 9,495,684 B2
33 34
alternative ones. Thus the frontopolar cortex contribution to involvement of this region in demanding cognitive tasks,
learning and exploration appears to be associated with Such as working memory and episodic retrieval.
maintaining and Switching back and forth between multiple The inhibitory view postulates that the anterior cingulate
behavioral alternatives in search of optimal behavior. The is involved in Suppressing inappropriate responses. This idea
frontopolar cortex has also been implicated in memory accounts very well not only for its involvement in the Stroop
retrieval, relational reasoning, and multitasking behaviors. task, in which prepotent responses must be inhibited, but
These subfunctions are thought to be integrated in the also in working memory, in which interference from previ
general function of contingently Switching back and forth ous trials must be controlled. The initiation and inhibition
between independent tasks by maintaining distractor-resis views are not incompatible: the anterior cingulate cortex
tant representations of postponed tasks during the perfor 10 may both initiate appropriate responses and Suppress inap
mance of another task. For example, the frontopolar cortex propriate ones. Moreover, these views share the idea that the
is specifically activated when Subjects suspend execution of anterior cingulate cortex plays an “active' role in cognition
by controlling the operations of other regions, including the
an ongoing task set associated with a priori the largest prefrontal cortex.
expected future rewards in order to explore a possibly 15 In contrast, the motor view conceptualizes the anterior
more-rewarding task set. See Keochlin et al., “Anterior cingulate as a more “passive' structure: it receives cogni
Prefrontal Function and the Limits of Human Decision tive/motor “commands' from various regions (for example,
Making.” Science, vol. 318, pp. 594-598 (26 Oct. 2007). prefrontal cortex), and “funnels' them to the appropriate
Activation of the medial prefrontal cortex and anterior motor system. This view assumes that different anterior
paracingulate cortex indicate that a subject is thinking and cingulate regions are engaged, depending on whether
acting on the beliefs of others, for example, either by responses are ocular, manual, or verbal. For example, due to
guessing partner strategies or when comparing play with its close connections to the auditory cortex, area 32 is
another human to play with a random device, such as a assumed to play a role in Vocalization and speech. This idea
computer partner. Accordingly, these regions may be accounts for activations during tasks involving verbal mate
involved in intention detection, i.e., assessing the meaning 25 rials, such as Stroop, semantic generation, and verbal epi
of behavior from another agent. The tempo-parietal junction sodic retrieval tasks. See Cabeza et al., “Imaging Cognition
is also implicated in this function. Further, publication II: An Empirical Review of 275 PET and fMRI Studies,” J.
brand-related bias in the credibility of ambiguous news Cognitive Neurosci., vol. 12, pp. 1-47 (2000).
headlines is associated with activation changes in the medial Lying is associated with increased activity in several areas
prefrontal cortex. See Kenning et al., “Neuroeconomics: an 30 of the cortex, including the anterior cingulate cortex, the
overview from an economic perspective.” Brain Res. Bull. parietal cortex, and the Superior frontal gyms. See Henig,
vol. 67, pp. 343-354 (2005). “Looking for the Lie.” New York Times http://www.nyti
In situations in which people gain some useful good (e.g., mes.com/2006/02/05/magazine/
money, juice, or other incentive) by using judgment, acti 05lying.html?pagewanted print (5 Feb. 2006).
vation can be observed in the so-called “reward areas” of the 35 Parietal Regions
brain. Therefore, a “feeling of approval or utility may Parietal regions are consistently activated during tasks
correlate with the activation in the reward areas of the brain. involving attention, spatial perception and imagery, working
Reward areas of the brain include the ventral striatum and memory, spatial episodic encoding, episodic retrieval, and
the orbitofrontal prefrontal cortex-amygdala-nucleus skill learning. Medial parietal activations are frequently
accumbens circuit. Monetary payoffs indicate activation in 40 found during episodic memory retrieval. In general, lateral
the nucleus accumbens. The nucleus accumbens is densely parietal activations relate either to spatial perception/atten
innervated by dopaminergic fibers originating from neurons tion or to Verbal working memory storage. Parietal regions
in the midbrain. Sudden release of dopamine after an unex may be part of a dorsal occipito-parietal pathway involved
pected reward may lead to acceptance of risk. Accordingly, in spatial perception, and/or part of a “posterior attention
defects in the orbitofrontal cortex-amygdala-nucleus accum 45 system’ involved in disengaging spatial attention. These
bens reward circuit may accompany extreme risk-seeking spatial views account for parietal activations during spatial
behavior. This reward system is also associated with the tasks of perception, imagery, and episodic encoding, as well
perception of utility of objects. as for those during skill-learning tasks, which, typically,
Cingulate Regions involve an important spatial component.
Cingulate regions can be roughly classified as anterior 50 According to the working memory interpretation, parietal
(for example, areas 32 and 24), central (areas 23 and 31), and regions are involved in the storage of Verbal information in
posterior (posterior area 31, retrosplenial). Posterior cingu working memory. This is consistent with evidence that left
late activations are consistently seen during Successful epi posterior parietal lesions can impair verbal short-term
sodic memory retrieval, as are other posterior midline acti memory. See Cabeza et al., “Imaging Cognition II: An
Vations (e.g., medial parietal, cuneus, precuneus). Anterior 55 Empirical Review of 275 PET and fMRI Studies,” J. Cog
cingulate activations occur primarily in area 32 and are nitive Neurosci., vol. 12, pp. 1-47 (2000).
consistently found for S-R compatibility (Stroop test), work Temporal Regions
ing memory, semantic generation, and episodic memory The temporal lobes can be subdivided into four broad
tasks. regions: lateral (insula, 42, 22, 21, and 20), medial (areas 28,
There are three main views of the anterior cingulate 60 34-36, and hippocampal regions), posterior (area 37), and
function: initiation, inhibitory, and motor. According to the polar (area 38). Area 38 is likely to have a very important
initiation view, the anterior cingulate cortex is involved in role in cognition, for example, by linking frontal-lobe and
"attention to action,” that is, in attentional processes temporal-lobe regions.
required to initiate behavior. This is consistent with evidence Lateral temporal activations are consistently found for
that damage to this region sometimes produces akinetic 65 language and semantic memory retrieval and are mostly
mutism, that is, an almost complete lack of spontaneous left-lateralized. Spoken word-recognition tasks usually yield
motor or verbal behavior. This is also consistent with the bilateral activations, possibly reflecting the auditory com
US 9,495,684 B2
35 36
ponent of these tasks. The involvement of the left superior al, “Imaging Cognition II: An Empirical Review of 275 PET
and middle temporal gyms (areas 22 and 21) in language and fMRI Studies,” J. Cognitive Neurosci., vol. 12, pp. 1-47
operations is consistent with research on aphasic patients. (2000).
Since area 21 is also consistently activated during semantic Mesolimbic Dopamine System
retrieval tasks—not only for verbal but also for nonverbal 5 Activity in the striatum scales directly with the magnitude
materials—it is possible that this area reflects semantic, of monetary reward or punishment. The striatum is also
rather than linguistic, operations. This is Supported by the involved in Social decisions, above and beyond a financial
involvement of this region in object perception. component. The striatum also encodes abstract rewards Such
Medial-temporal lobe activations are repeatedly found for 10
as positive feeling as a result of mutual cooperation. In
episodic memory encoding and nonverbal episodic memory addition, the caudate is activated in situations where a
retrieval. The involvement of medial temporal regions in Subject has an intention to trust another. Emotional pro
episodic memory is consistent with lesion data. Based on cesses are reliably associated with a series of structures
PET data, encoding-related activations are more common in including the striatum and caudate, and areas of the midbrain
anterior hippocampal regions, whereas retrieval-related acti 15 and cortex to which they project, Such as the Ventromedial
Vations are more prevalent in posterior hippocampal regions, prefrontal cortex, orbitofrontal cortex, and anterior cingu
a pattern described as the hippocampal encoding/retrieval lated cortex, as well as other areas Such as the amygdala and
(HIPER) model. See Cabeza et al., “Imaging Cognition II: the insula. Indeed, subjects with lesions in the ventromedial
An Empirical Review of 275 PET and fMRI Studies,” J. prefrontal cortex and having associated emotional deficits
Cognitive Neurosci., vol. 12, pp. 1-47 (2000). are impaired in performing gambling tasks. The anterior
Occipito-Temporal Regions insula is associated with increased activation as unfairness
The engagement of temporo-occipital regions (areas 37. or inequity of an offer is increased. Activation of the anterior
19, 18, and 17) in cognitive tasks seems to be of two kinds insula predicts an Ultimatum Game player's decision to
activations associated with perceiving and manipulating either accept or reject an offer, with rejections associated
visuospatial information, and deactivations associated with 25 with significantly higher activation than acceptances. Acti
perceptual priming. Visual processing along the ventral Vation of the anterior insula is also associated with physi
pathway is assumed to be organized hierarchically, with cally painful, distressful, and/or disgusting stimuli. Thus, the
early image analyses engaging areas close to the primary anterior insula and associated emotion-processing areas may
visual cortex and higher-order object recognition processes play a role in marking an interaction as aversive and
involving more anterior areas. Consistent with this idea, 30 undeserving of trust in the future. See Sanfey, “Social
activations in areas 18 and 19 occur for most visuospatial Decision-Making: Insights from Game Theory and Neuro
tasks, whereas activations in area 37 are associated with science.” Science, vol. 318, pp. 598-601 (26 Oct. 2007).
object processing. For example, area 37 activation is found Activation in the ventral striatum is seen by fMRI when
when Subjects perceive objects and faces, maintain images Subjects provide a correct answer to a question, resulting in
of objects in working memory, and intentionally encode a reward. Similarly, a wrong answer and no payment results
in a reduction in activity (i.e., oxygenated blood flow) to the
objects. Perception-related occipital activations are ventral striatum. Moreover, activation of the reward centers
enhanced by visual attention and they therefore can be of the brain including the ventral striatum over and above
expected during visual-attentional tasks, as well as during that seen from a correct response and reward is seen when
demanding visual-skill learning tasks (e.g., mirror reading). 40 a subject receives a reward that is known to be greater than
Most activations in occipito-temporal regions occur dur that of a peer in the study. Thus, stimulation of the reward
ing the processing of visual information coming from eyes center appears to be linked not only to individual Success
(perception) or from memory (imagery), and weaken when and reward, but also to the Success and rewards of others.
the same information is repeatedly processed (priming). See See BBC news story “Men motivated by superior wage.”
Cabeza et al., “Imaging Cognition II: An Empirical Review 45 http://news.bbc.co.uk/1/hi/sci/tech/7108347.stm, (23 Nov.
of 275 PET and fMRI Studies,” J. Cognitive Neurosci., vol. 2007).
12, pp. 1-47 (2000). In a multi-round trust game, reciprocity expressed by one
Subcortical Regions player strongly predicts future trust expressed by their
With respect to activations in the basal ganglia, the partner—a behavioral finding mirrored by neural responses
thalamus, and the cerebellum, basal ganglia activations were 50 in the dorsal striatum as measured by fMRI. Analyses within
common during motor-skill learning, and the cerebellum and between brains show two signals—one encoded by
was consistently activated in several different processes. response magnitude, and the other by response timing.
Evolutionary, anatomical, neuropsychological, and func Response magnitude correlates with the “intention to trust’
tional neuroimaging evidence indicates that the cerebellum on the next play of the game, and the peak of these “intention
plays an important role in cognition. The cognitive role of 55 to trust” responses shifts its time of occurrence by 14
the cerebellum has been related as motor-preparation, sen seconds as player reputations develop. This temporal trans
sory acquisition, timing, and attention/anticipation. Each of fer resembles a similar shift of reward prediction errors
these views can account for Some cerebellar activations, but common to reinforcement learning models, but in the con
not for all of them. For example, the motor preparation view text of a social exchange. See King-Casas et al., “Getting to
accounts well for activations during tasks involving motor 60 Know You: Reputation and Trust in a Two-Person Economic
responses. Such as word production and conditioning, while Exchange,” Science, vol. 308, pp. 78-83 (1 Apr. 2005).
the sensory-acquisition view can accommodate activations Activity in the head of the caudate nucleus is associated
during perceptual tasks, such as Smelling. The timing view with the processing of information about the fairness of a
accounts for activations during tasks involving relations Social partner's decision and the intention to repay with
between Successive events, such as conditioning and skill 65 trust, as measured by hyperScan-fMRI. See Kenning et al.,
learning, while the attention/anticipation view explains acti “Neuroeconomics: an overview from an economic perspec
Vations during attention and problem solving. See Cabeza et tive.” Brain Res. Bull., vol. 67, pp. 343-354 (2005).
US 9,495,684 B2
37 38
Activation of the insular cortex is associated with the or agitation and can be heard, seen, touched, Smelt or tasted.
perception of bodily needs, providing direction to motivated Emotional exchanges may be visible displays of body ten
behaviors. For example, imaging studies have shown acti sion or movement, gestures, posture, facial expressions or
Vation of the insula in addicts with cue-indicated drug use of personal space; or audible displays such as tone of
craving, and activation of the insular cortex has been asso Voice, choice of pitch contour, choice of words, speech rate,
ciated with Subjective reports of drug craving. See Contreras etc. Humans also use touch, Smell, adornment, fashion,
et al., “Inactivation of the Interoceptive Insula Disrupts Drug architecture, mass media, and consumer products to com
Craving and Malaise Indicated by Lithium.” Science, vol. municate our emotional state. Universals of emotion that
318, pp. 655-658 (26 Oct. 2007). cross cultural boundaries have been identified, and cultural
Visual Cortex 10 differences have also been identified. For example love' is
The visual cortex is located in and around the calcarine generally categorized as a positive emotion in Western
fissure in the occipital lobe. In one visual cortex study, Societies, but in certain Eastern cultures there is also a
Subjects were shown two patterns in quick Succession. The concept for sad love. Accordingly, universal emotional
first appeared for just 15 milliseconds, too fast to be con triggers may be used to transcend cultural barriers.
sciously perceived by the viewer. By examining fMRI 15 When communicating with computers, people often treat
images of the brain, a specific image that had been flashed new media as if they were dealing with real people. They
in front of the subjects could be identified. The information often follow complex social rules for interaction and modify
was perceived in the brain even if the subjects were not their communication to Suit their perceived conversation
consciously aware of it. The study probed the part of the partner. Much research has focused on the use of facial
visual cortex that detects a visual stimulus, but does not actions and ways of coding them. Speech recognition sys
perceive it. It encodes visual information that the brain does tems have also attracted attention as they grow in capability
not process as “seen.’ See “Mind-reading machine knows and reliability, and can recognize both verbal messages
what you see.' NewScientist.com http://www.newscientist. conveyed by spoken words, and non verbal messages, such
com/article.ns?id=dn7304&feedId=online-news rss20 (25 as those conveyed by pitch contours.
Apr. 2005). 25 System responses and means of expressing emotions also
Hippocampus vary. Innovative prototypes are emerging designed to
Activation of the hippocampus can modulate eating respond indirectly, so the user is relatively unaware of the
behaviors linked to emotional eating and lack of control in response: for example by adaptation of material. Such as
eating. Activation of brain areas known to be involved in changing pace or simplifying or expanding content. Other
drug craving in addicted Subjects, such as the orbitofrontal 30 systems use text, voice technology, visual agents, or avatars
cortex, hippocampus, cerebellum, and striatum, Suggests to communicate. See Axelrod et al., “Smoke and Mirrors:
that similar brain circuits underlie the enhanced motivational Gathering User Requirements for Emerging Affective Sys
drive for food and drugs seen in obese and drug-addicted tems. 26th Int. Conf. Information Technology Interfaces/TI
subjects. See Wang et al., “Gastric stimulation in obese 2004, Jun. 7-10, 2004, Cavtat, Croatia, pp. 323-328.
Subjects activates the hippocampus and other regions 35 Skin Response
involved in brain reward circuitry.” PNAS, Vol. 103, pp. Mental state may be determined by detection of a skin
15641-45 (2006). response associated with a stimulus. One skin response that
Surrogate Markers of Mental State may correlate with mental state and/or brain activity is
Surrogate markers of mental state may include indicators galvanic skin response (GSR), also known as electrodermal
of attention, approval, disapproval, recognition, cognition, 40 response (EDR), psychogalvanic reflex (PGR), or skin con
memory, trust, or the like in response to a stimulus, other ductance response (SCR). This is a change in the electrical
than measurement of brain activity associated with the resistance of the skin. There is a relationship between
stimulus. sympathetic nerve activity and emotional arousal, although
Examples of Surrogate markers may include a skin one may not be able to identify the specific emotion being
response to a stimulus; a face pattern indicative of approval, 45 elicited. The GSR is highly sensitive to emotions in some
disapproval, or emotional state; eye movements or pupil people. Fear, anger, startle response, orienting response, and
movements indicating visual attention to an object; voice sexual feelings are all among the emotions which may
stress patterns indicative of a mental state, or the like. produce similar GSR responses. GSR is typically measured
Surrogate markers may be used in conjunction with brain using electrodes to measure skin electrical signals.
activity measurements for higher confidence in a predictive 50 For example, an Ultimate Game study measured skin
or interpretational outcome. For example, brain activation of conductance responses as a surrogate marker or autonomic
the caudate nucleus in combination with calm voice patterns index for affective state, and found higher skin conductance
may increase confidence in a predictor of trust between a activity for unfair offers, and as with insular activation in the
subject and a stimulus. Conversely, conflict between brain brain, this measure discriminated between acceptances and
activity and a surrogate marker may decrease confidence in 55 rejections of these offers. See Sanfey, “Social Decision
a predictive or interpretational outcome. For example, a Making: Insights from Game Theory and Neuroscience.”
pattern of activation of the insula diagnostic for fear, Science, vol. 318, pp. 598-601 (26 Oct. 2007). Other skin
together with a visual face image showing a Smile may responses may include flushing, blushing, goose bumps,
decrease the level of confidence that the subject is truly Sweating, or the like.
frightened by a stimulus. 60 Face Pattern Recognition
For example, emotion links to cognition, motivation, Mental state may also be determined by detection of facial
memory, consciousness, and learning and developmental feature changes associated with a stimulus, via pattern
systems. Affective communication depends on complex, recognition, emotion detection Software, face recognition
rule-based systems with multiple channels and redundancy software, or the like.
built into the exchange system, in order to compensate if one 65 For example, an emotional Social intelligence prosthetic
channel fails. Channels can include all five senses: for device has been developed that consists of a camera Small
example, increased heart-rate or Sweating may show tension enough to be pinned to the side of a pair of glasses,
US 9,495,684 B2
39 40
connected to a hand-held computer running image recogni relative to objects on a display, which information may be of
tion Software plus association Software that can read the interest to an entity 170 (e.g. http://ip.hamamatsu.com/en/
emotions these images show. If the wearer seems to be rd/publication/scientific american/common/pdf/scien
failing to engage his or her listener, the software makes the tific 06 08.pdf).
hand-held computer vibrate. The association Software can Eye movement and/or pupil movement may also be
detect whether someone is agreeing, disagreeing, concen measured by Video-based eye trackers. In these systems, a
trating, thinking, unsure, or interested, just from a few camera focuses on one or both eyes and records eye move
seconds of video footage. Previous computer programs have ment as the viewer looks at a stimulus. Contrast may be used
detected the six more basic emotional states of happiness, to locate the center of the pupil, and infrared and near
sadness, anger, fear, Surprise and disgust. The system can 10 infrared non-colluminated light may be used to create a
detect a sequence of movements beyond just a single facial corneal reflection. The vector between these two features
expression. The association program is based on a machine can be used to compute gaze intersection with a surface after
learning algorithm that was trained by showing it more than a calibration for a subject.
100 8-second video clips of actors expressing particular Two types of eye tracking techniques include bright pupil
emotions. The Software picks out movements of the eye 15 eye tracking and dark pupil eye tracking. Their difference is
brows, lips and nose, and tracks head movements such as based on the location of the illumination source with respect
tilting, nodding, and shaking, which it then associates with to the optics. If the illumination is coaxial with the optical
the emotion the actor was showing. When presented with path, then the eye acts as a retroreflector as the light reflects
fresh video clips, the Software gets people's emotions right off the retina, creating a bright pupil effect similar to red eye.
90 percent of the time when the clips are of actors, and 64 If the illumination source is offset from the optical path, then
percent of the time on footage of ordinary people. See the pupil appears dark.
“Device warns you if you’re boring or irritating.” NewSci Bright Pupil tracking creates greater iris/pupil contrast
entist http://www.newscientist.com/article/ allowing for more robust eye tracking with all iris pigmen
mg 19025456.500-device-warns-you-if-youre-boring-or-ir tation and greatly reduces interference caused by eyelashes
ritating.html (29 Mar. 2006). 25 and other obscuring features. It also allows for tracking in
In another approach, an imager, such as a CCD camera, lighting conditions ranging from total darkness to very
may observe expressed features of the user. For example, the bright light. However, bright pupil techniques are not rec
imager may monitor pupil dilation, eye movement, expres ommended for tracking outdoors as extraneous IR sources
Sion, or a variety of other expressive indicators. Such may interfere with monitoring.
expressive indicators may indicate a variety of emotional, 30 Eye tracking configurations can vary; in some cases the
behavioral, intentional, or other aspects of the user. For measurement apparatus may be head-mounted, in some
example, in one approach, systems have been developed for cases the head should be stable (e.g., stabilized with a chin
identifying an emotional behavior of a person based upon rest), and in some cases the eye tracking may be done
selected expressive indicators. Similarly, eye movement and remotely to automatically track the head during motion.
pupil dilation may be correlated to truthfulness, stress, or 35 Most eye tracking systems use a sampling rate of at least 30
other user characteristics. HZ. Although 50/60 Hz is most common, many video-based
Eye Movement Analysis eye trackers run at 240, 350 or even 1000/1250 Hz, which
Eye movement or pupil movement can be tested, for is recommended in order to capture the detail of the very
example, by measuring user pupil and/or eye movements, rapid eye movements during reading, or during studies of
perhaps in relation to items on a display. For example, a 40 neurology.
user's eye movement to a part of the screen containing an Eye movements are typically divided into fixations, when
advertisement may be of interest to an advertiser for pur the eye gaze pauses in a certain position, and Saccades, when
poses of advertisement placement or determining advertis the eye gaze moves to another position. A series of fixations
ing noticeability and/or effectiveness within a computerized and Saccades is called a scanpath. Most information from the
game world. For example, knowing that a user's eyes have 45 eye is made available during a fixation, not during a saccade.
been attracted by an advertisement may be of interest to an The central one or two degrees of the visual angle (the
advertiser. For example, a merchant may be interested in fovea) provide the bulk of visual information; input from
measuring whether a user notices a virtual world avatar larger eccentricities (the periphery) generally is less infor
having particular design characteristics. If the user exhibits mative. Therefore the locations of fixations along a scanpath
eye movements toward the avatar on a display, then the 50 indicate what information loci on the stimulus were pro
merchant may derive a mental state from repeated eye cessed during an eye tracking session. On average, fixations
movements vis a vis the avatar, or the merchant may last for around 200 milliseconds during the reading of
correlate eye movements to the avatar with other physiologi linguistic text, and 350 milliseconds during the viewing of
cal activity data Such as brain activation data indicating a a scene. Preparing a saccade towards a new goal takes
mental State Such as brand preference, approval or reward. 55 around 200 milliseconds.
In another embodiment, a Smart camera may be used that Scanpaths are useful for analyzing cognitive intent, inter
can capture images of a user's eyes, process them and issue est, and salience. Other biological factors (some as simple as
control commands within a millisecond time frame. Such gender) may affect the Scanpath as well. Eye tracking in
Smart cameras are commercially available (e.g., Hamamat human-computer interaction typically investigates the scan
Su’s Intelligent Vision System: http://ip.hamamatsu.com/en/ 60 path for usability purposes, or as a method of input in
product info/index.html). Such image capture systems may gaZe-contingent displays, also known as gaze-based inter
include dedicated processing elements for each pixel image faces.
sensor. Other camera systems may include, for example, a There are two primary components to most eye tracking
pair of infrared charge coupled device cameras to continu studies: statistical analysis and graphic rendering. These are
ously monitor pupil size and position as a user watches a 65 both based mainly on eye fixations on specific elements.
visual target moving, e.g., forward and backward. This can Statistical analyses generally sum the number of eye data
provide real-time data relating to pupil accommodation observations that fall in a particular region. Commercial
US 9,495,684 B2
41 42
Software packages may analyze eye tracking and show the tracking technology. Analyses may focus on visibility of a
relative probability of eye fixation on each feature on an target avatar, product, or logo in the context of a magazine,
avatar. This allows for a broad analysis of which avatar newspaper, website, virtual world, or televised event. This
elements received attention and which ones were ignored. allows researchers to assess in great detail how often a
Other behaviors such as blinks, saccades, and cognitive sample of consumers fixates on the target avatar, logo,
engagement can be reported by commercial software pack product, or advertisement. In this way, an advertiser can
ages. Statistical comparisons can be made to test, for quantify the Success of a given campaign in terms of actual
example, competitors, prototypes or Subtle changes to an visual attention.
avatar. They can also be used to compare participants in Eye tracking also provides avatar designers with the
different demographic groups. Statistical analyses may 10 opportunity to examine the visual behavior of a consumer
quantify where users look, Sometimes directly, and some while interacting with a target avatar. This may be used to
times based on models of higher-order phenomena (e.g., analyze distinctiveness, attractiveness and the tendency of
cognitive engagement). the avatar to be chosen for recognition and/or purchase. Eye
In addition to statistical analysis, it is often useful to tracking can be used while the target avatar is in the
provide visual depictions of eye tracking results. One 15 prototype stage. Prototype avatars can be are tested against
method is to create a video of an eye tracking testing session each other and against competitors to examine which spe
with the gaze of a participant Superimposed upon it. This cific elements are associated with high visibility and/or
allows one to effectively see through the eyes of the con appeal.
Sumer during interaction with a target medium. Another Another application of eye tracking research is in the field
method graphically depicts the scanpath of a single partici of automotive design. Eye tracking cameras may be inte
pant during a given time interval. Analysis may show each grated into automobiles to provide the vehicle with the
fixation and eye movement of a participant during a search capacity to assess in real-time the visual behavior of the
on a virtual shelf display of breakfast cereals, analyzed and driver. The National Highway Traffic Safety Administration
rendered with a commercial software package. For example, (NHTSA) estimates that drowsiness is the primary causal
a different color may represent one second of viewing time, 25 factor in 100,000 police-reported accidents per year. Another
allowing for a determination of the order in which products NHTSA study suggests that 80% of collisions occur within
are seen. Analyses such as these may be used as evidence of three seconds of a distraction. By equipping automobiles
specific trends in visual behavior. with the ability to monitor drowsiness, inattention, and
A similar method sums the eye data of multiple partici cognitive engagement driving safety could be dramatically
pants during a given time interval as a heat map. A heat map 30 enhanced. Lexus(R claims to have equipped its LS 460
may be produced by a commercial Software package, and automobile with the first driver monitor system in 2006,
shows the density of eye fixations for several participants providing a warning if the driver takes his or her eye off the
Superimposed on the original stimulus, for example, an road.
avatar on a magazine cover. Red and orange spots represent Eye tracking is also used in communication systems for
areas with high densities of eye fixations. This allows one to 35 disabled persons, allowing the user to speak, mail, Surf the
examine which regions attract the focus of the viewer. web and so on with only the eyes as tool. Eye control works
Commercial eye tracking applications include web usabil even when the user has involuntary body movement as a
ity, advertising, sponsorship, package design and automotive result of cerebral palsy or other disability, and/or when the
engineering. Eye tracking studies may presenting a target user wears glasses.
stimulus to a sample of consumers while an eye tracker is 40 Eye movement or pupil movement may be gauged from
used to record the activity of the eye. Examples of target a users interaction with an application.
stimuli may include avatars in the context of websites, An example of a measure of pupil movement may be an
television programs, sporting events, films, commercials, assessment of the size and symmetry of a user's pupils
magazines, newspapers, packages, shelf displays, consumer before and after a stimulus, such as light or focal point. In
systems (ATMs, checkout Systems, kiosks), and Software. 45 one embodiment, where the user interacts with a head
The resulting data can be statistically analyzed and graphi mounted display, the display may include image capturing
cally rendered to provide evidence of specific visual pat features that may provide information regarding expressive
terns. By examining fixations, saccades, pupil dilation, indicators. Such approaches have been described in
blinks, and a variety of other behaviors, researchers can scanned-beam display Systems such as those found in U.S.
determine a great deal about the effectiveness of a given 50 Pat. No. 6,560,028.
avatar in a given medium or associated with a given product. Voice Stress Analysis
A prominent field of eye tracking research is web usabil Voice stress analysis (VSA) technology records psycho
ity. While traditional usability techniques are often quite physiological stress responses that are present in the human
powerful in providing information on clicking and scrolling Voice when a person experiences a psychological stress in
patterns, eye tracking offers the ability to analyze user 55 response to a stimulus. Psychological stress may be detected
interaction between the clicks. This provides insight into as acoustic modifications in the fundamental frequency of a
which features are the most eye-catching, which features speaker's voice relative to normal frequency modulation of
cause confusion, and which ones are ignored altogether. the Vocal signal between 8-14 Hz, during speech in an
Specifically, eye tracking can be used to assess impressions emotionally neutral situation. In situations involving a stress
of an avatar in the context of search efficiency, branding, 60 response, the 8-14 HZ modulation may decrease as the
online advertisement, navigation usability, overall design, muscles Surrounding the Vocal cords contract in response to
and/or many other site components. Analyses may target an the reaction.
avatar on a prototype or competitor site in addition to the VSA typically records an inaudible component of human
main client site. voice, commonly referred to as the Lippold Tremor. Under
Eye tracking is commonly used in a variety of different 65 normal circumstances, the laryngeal muscles are relaxed,
advertising media. Commercials, print ads, online ads, and producing recorded Voice at approximately 12 Hz. Under
sponsored programs are all conducive to analysis with eye stress however, the tensed laryngeal muscles produce voice
US 9,495,684 B2
43 44
significantly lower than normal. The higher the stress, the Concentration Level: Extreme concentration might indi
lower down the Hertz scale voice waves are produced. One cate deception.
application for VSA is in the detection of deception. Anticipation Level: Is your subject anticipating your
Dektor Counterintelligence manufactured the PSE 1000, responses according to what he or she is telling you?
an analog machine that was later replaced by the PSE 2000. Embarrassment Level: Is your subject feeling comfort
The National Institute Of Truth Verification (NITV) then able, or does he feel Some level of embarrassment regarding
produced and marketed a digital application based on the what he or she is saying?
McQuiston-Ford algorithm. The primary commercial Sup Arousal Level: What triggers arousal in the subject? Is he
pliers are Dektor (PSE5128-software); Diogenes (Lantern or she interested in an object? Aroused by certain visual
10 stimuli?
software); NITV (CVSA Software); and Baker (Baker Deep Emotions: What long-standing emotions does a
software). subject experience?
VSA is distinctly different from LVA (Layered Voice Is he or she “excited' or “uncertain' in general?
Analysis). LVA is used to measure different components of SENSE’s “Deep' Technology: Is a subject thinking about
voice, such as pitch and tone. LVA is available in the form 15 a single topic when speaking, or are there several layers to
of hand-held devices and software. LVA produces readings a response (e.g., background issues, something that may be
Such as love, excitement, and fear. bothering him or her, planning, or the like). SENSE tech
One example of a commercially available layered voice nology can detect brain activity operating at a pre-conscious
analysis system is the SENSE system, sold by Nemesysco level.
Ltd (Natania, Israel). SENSE can analyze different layers The speaking mechanism is one of the most complicated
within the Voice, using multiple parameters to analyze each procedures the human body is capable of First, the brain has
speech segment. SENSE can detect various cognitive states, to decide what should be said, then air is pushed from the
Such as whether a subject is excited, confused, stressed, lungs upward to the Vocal cords, that must vibrate to produce
concentrating, anticipating a response, or unwillingly shar the main frequency. Now, the vibrated air arrives to the
ing information. The technology also can provide an in 25 mouth.
depth view of the Subjects range of emotions, including The tongue, the lips, the teeth, and the nose space turns the
those relating to love. SENSE technology can be further vibrated air into the Sounds that we recognize as phrases.
utilized to identify psychological issues, mental illness, and The brain is closely monitoring all these events, and listens
other behavioral patterns. The LVA technology is the secu to what comes out; if we speak too softly, too loudly, and if
rity version of the SENSE technology, adapted to identify 30 it is understandable to a listener. SENSE Technology ignores
the emotional situations a Subject is expected to have during what your Subject is saying, and focuses only on what the
formal/security investigations. brain is broadcasting.
The SENSE technology is made up of 4 sub-processes: Humans, unlike other mammals, are capable of predicting
1. The vocal waveform is analyzed to measure the pres or imagining the future. Most people can tell whether or not
ence of local micro-high frequencies, low frequencies, and 35 a certain response will cause them pleasure or pain. Lying is
changes in their presence within a single voice sample. not a feeling, it is a tool. The feeling structure around it will
2. A precise frequency spectrum of the Vocal input is be the one causing us to lie, and understanding the differ
sampled and analyzed. ences is crucial for making an analysis.
3. The parameters gathered by the previous steps are used The SENSE technology differentiates among 5 types of
to create a baseline profile for the subject. 40 lies:
4. The new Voice segments to be tested are compared with 1. Jokes—Jokes are not so much lies as they are untruths,
the Subject's baseline profile, and the analysis is generated. used to entertain. No long gain profit or loss will be earned
This input can be further processed by statistical learning from it, and usually, little or no extra feelings will be
algorithms to predict the probability of a deceptive or involved.
fraudulent sentence in a Subject's speech. Another layer that 45 2. White Lies You know you don’t want to say the truth,
is used in certain applications evaluates the conversation as as it may hurt someone else. White lies are lies, but the teller
a whole, and produces a final risk or QA value. usually experiences little stress or guilt.
The SENSE technology can detect the following emo 3. Embarrassment Lies—Same as for white lies, but this
tional and cognitive states: time directed internally. Nothing will be lost except the
Excitement Level Each of us becomes excited (or 50 respect of the listener, most likely for the short term.
depressed) from time to time. SENSE compares the pres 4. Offensive Lies. This is a unique lie, for its intention
ence of the Micro-High-frequencies of each sample to the is to gain something extra that could not be gained other
basic profile to measure the excitement level in each vocal wise.
Segment. 5. Defensive Lie The common lie to protect one’s self.
Confusion Level: Is your subject sure about what he or 55 The SENSE technology IS the old “Truster” technology,
she is saying? SENSE technology measures and compares with several additions and improvements. The old Truster
the tiny delays in a Subject's voice to assess how certain he was all about emotions in the context of Truth/Lie: SENSE
or she is. looks at emotions in general.
Stress Level: Stress may include the body's reaction to a When people get sexually aroused or feel “in love, the
threat, either by fighting the threat, or by fleeing. However, 60 pupils get wider, the lips get reddish, the skin of the face gets
during a spoken conversation neither option may be avail red. The Voice changes too. Increased excitement makes the
able. The conflict caused by this dissonance affects the whole voice higher and more concentrated. The SENSE
micro-low-frequencies in the voice during speech. technology can detect the increased excitement and the
Thinking Level: How much is your subject trying to find associated heightened concentration and anticipation.
answers? Might he or she be “inventing stories? 65 While each of the above described approaches to provid
S.O.S.: (Say Or Stop) Is your subject hesitating to tell ing expressive indicators has been described independently,
you something? in Some approaches, a combination of two or more of the
US 9,495,684 B2
45 46
above described approaches may be implemented to provide Verbal attributes or characteristics may be presented, such
additional information that may be useful in evaluating user as foreign language or accented speech, Such as a Southern
behavior and/or mental state. accent, a Boston accent, a Spanish accent, a British accent,
Specifying a Cohort-Linked Avatar Attribute and/or a or the like. Such voice variations may be computer-detect
Cohort-Linked Avatar able and/or computer-implemented. See, for example, U.S.
FIG. 5 illustrates an operational flow 500 representing Pat. No. 7.263,489 "Detection of characteristics of human
example operations related to specifying an avatar attribute. machine interactions for dialog customization and analysis.”
In FIG. 5 and in following figures that include various Prosodic features of speech Such as intonation, stress, and
examples of operational flows, discussion and explanation 10
other paralinguistic features of speech Such as voice quality,
may be provided with respect to the above-described system emotion, and speaking style may also be presented as
environments of FIGS. 1-4, and/or with respect to other characteristic.
examples and contexts. However, it should be understood A characteristic may also include paralanguage, a.k.a.,
that the operational flows may be executed in a number of vocalics, which involves nonverbal cues of the voice. Vari
other environment and contexts and/or in modified versions 15 ous acoustic properties of speech Such as tone, pitch, accent,
of FIGS. 1-3. Also, although the various operational flows or the like, collectively known as prosody, can be presented
are presented in the sequences illustrated, it should be as nonverbal characteristics. Paralanguage may be used to
understood that the various operations may be performed in present a characteristic, as well as voice qualities including
other orders than those which are illustrated, or may be Volume, pitch, tempo, rhythm, articulation, resonance,
performed concurrently. nasality, and/or accent. Vocalization cues also may be pre
After a start operation, operation 510 depicts presenting at sented as the characteristic, including emotions expressed
least one characteristic to at least one member of a popula during or associated with speech, such as laughing, crying,
tion cohort. For example, an attribute specification unit 250 and/or yawning. Vocalization cues also may include delivery
can transmit a characteristic for display to a member of nuances such as Volume and/or pitch modulation Such as
population cohort 102 on a presentation unit and/or display 25 whispering and shouting. Vocalization cues also may include
272. Optionally, a presented avatar characteristic may be Vocal segregates such as “um' in between spoken expres
encountered during an interaction of the member of popu sions, or “uh-huh.” “like.” “no way,” or other phrase in
lation cohort 102 with a virtual world; alternatively, the response to another's speech to indicate comprehension, to
characteristic may be one presented in a real world context, punctuate speech, and/or to manage contact during dialogue.
for Subsequent incorporation into an avatar. In one embodi 30 See, for example, U.S. Pat. No. 6,356,868, “Voiceprint
ment, a clothing specification unit 260 can present a pho identification system.”
tograph of a particular item of clothing independent of an Alternatively, a non-verbal attribute or characteristic
avatar via mobile display 276. In another embodiment, a including physical appearance and/or non-verbal communi
facial attribute specification unit 258 can present an eye cation may be presented. For example, body language such
shape and color to a member of population cohort 102 in the 35 as use of American Sign Language may be presented as the
context of an avatar in an online game via a desktop display characteristic.
274. Non-verbal communication characteristics may also
For example, a device 106, attribute specification unit include, for example, a facial expression, a gesture, a gaze,
250, and/or presentation unit 270 may present at least one and/or a posture. A characteristic may also include clothing,
characteristic to at least one member of a population cohort 40 hairstyle, adornment, shoes, and/or other communicative
102. In one embodiment, a body attribute specification unit props; or even architecture, a symbol, and/or an infographic.
362 may communicate via a network 374, for example, with Kinesic behaviors such as body movements, facial
a presentation device 364 to present a characteristic to expressions, and gestures may also be presented character
member of population cohort 102. A population cohort may istics. Kinesic behaviors may include, mutual gaze, Smiling,
include an ad hoc population cohort, or an established 45 facial warmth or pleasantness, childlike behaviors, direct
population cohort Such as an age-defined demographic body orientation, and the like. An attribute specification unit
group. 250 and/or presentation unit 270 can present a movement
In another embodiment, a speech specification unit of the characteristic Such as a kineme, which is a unit of visual
device 306 may present a speech characteristic Such as expression analogous to a phoneme, a unit of speech.
accent, dialect, tone, or pitch to a member of population 50 Presentable gestures may include emblems, illustrators,
cohort 102 via, for example, a personal communication affect displays, regulators, and/or adaptors. An emblem is a
device Such as a video-capable cellular phone. For example, gesture with a direct verbal translation such as a wave of the
the attribute specification unit 350 may initiate the presen hand; an illustrator is a gesture that depicts a concept that is
tation of an avatar with a particular characteristic Such as a Substantially simultaneously spoken, such as turning an
hair color or hair style to a member of population cohort 102 55 imaginary steering wheel while speaking about driving; an
in response to the member of population cohort 102 signing affect display is a gesture that conveys emotions, such as a
on to a social networking website. The presentation may be Smile or a frown; a regulator is a gesture that controls
through, for example, a widget on the Social networking interaction Such as a “shhh' sign placing an index finger
website. vertically at the center of the lips; and finally, an adaptor is
It should be understood that characteristics may be prof 60 a gesture that facilitates release of body tension, such as
itably combined to provide feedback about composite fea quick, repetitive leg movements or stretching.
tures of presented attributes and/or avatars. For example, an A presented characteristic may also include an advertising
attribute specification unit 350 may present a composite symbol and/or a brand name, design, symbol, or logo (e.g.,
voice, facial attribute, clothing attribute, and body attribute trademarks of corporations such as a BicR) pen, a Rolex(R)
characteristic to a member of population cohort 102 via an 65 watch, a McDonald's(R restaurant, a Hermes(R scarf, a Louis
online shopping experience, Such as a virtual personal Vuitton(R) bag, or a Les Paul.(R) guitar). Service marks may
shopper. also be presented as a characteristic.
US 9,495,684 B2
47 48
A presented characteristic may also include, for example, occur within the United States as long as at least one of these
a characteristic that is not typically presented in the context elements resides in the United States.
of an avatar, and/or a characteristic that an avatar cannot Operation 530 depicts associating the at least one physi
have. ological activity with at least one mental state. For example,
Operation 520 depicts measuring at least one physiologic 5 a device 206 and/or association unit 240 may associate the
activity of the at least one member of the population cohort, at least one physiologic activity with at least one mental
the at least one physiologic activity proximate to the at least state. For example, association unit 240 may associate
one characteristic. For example, a physiologic activity mea approval with a pattern of activation of the reward center,
surement unit 210, brain activity measurement unit 212, including for example, the striatum and caudate, and areas of
and/or Surrogate marker measurement unit 230 may measure 10 the midbrain and cortex to which they project, such as the
at least one physiologic activity of the at least one member ventromedial prefrontal cortex, orbitofrontal cortex, and
of the population cohort, the at least one physiologic activity anterior cingulated cortex. In one embodiment, an associa
proximate to the at least one characteristic. For example, tion unit 240 may search a database of functional brain
during presentation of the characteristic to the member of mapping that contains information about which brain
15 regions are associated with particular mental states and/or
the population cohort 102, a NIR module may detect brain functions. Such a database search may be keyed to brain
activity in the member of the population cohort 102 indica activation pattern, and/or to mental state and/or function. For
tive of approval of the characteristic. Of course other mea example, an association unit 240 may search measured brain
surement methods may be used including fMRI, MEG, activation information and/or Surrogate marker information
EEG, singly or in combination. Optionally, Surrogate mark for data consistent with trust. For example, a measured
ers of brain activity may be employed to detect physiologic activation of a region of the brain associated with trust, Such
activity, such as a voice response to a characteristic, an eye as the caudate nucleus, may be associated with trust as the
movement response to an object, a skin response to an at least one mental state. Alternatively or in addition, mea
object, or the like. Detected brain activity may be proximate Surement of a Surrogate marker Such as an approving voice
in time to presentation and/or viewing of the characteristic 25 response proximate to the at least one characteristic may be
by the member of the population cohort 102, as determined associated with approval as the at least one mental state by
by one or more of brain activity indicative of visual activity an association unit 140.
(e.g., activity in the visual cortex), selective attention (e.g., Operation 540 depicts specifying at least one avatar
activity in fronto-parietal areas), and/or perception (e.g., attribute based on the at least one mental state. For example,
activity in the ventral pathway). Detected physiologic activ 30 a device 206 and/or attribute specification unit 250 may
ity may be determined to be proximate in time using an eye specify at least one avatar attribute based on the at least one
movement or gaze tracking measurement to identify times mental state. For example, a device 206 and/or attribute
when a member of the population cohort 102 looks at a specification unit 250 may specify at least one facial feature
characteristic. In one embodiment, eye movement or gaZe or facial dimension as an avatar attribute for use in an avatar
tracking data may be matched with a time course of brain 35 based on an approving mental state identified in association
activity to associate a particular brain area activation with with a measured physiologic activity proximate to a pre
visual contact of the member of the population cohort 102 sented characteristic. In one embodiment, an association of
with a presented characteristic. a characteristic with a high degree of attention on the part of
As used herein, the term “proximate' may refer to “proxi a member of population cohort 102 may provide the basis
mate in time.” for example, a physiologic measurement that 40 for an attribute specification unit 250 to specify the charac
is proximate in time to a presentation of a characteristic to teristic as an attribute for incorporation into a cohort-linked
a member of population cohort 102 and/or member of aVatar.
population 105. In another embodiment, a voice specification unit 252
In another embodiment, eye movement or gaze tracking may specify a particular tonal quality of voice as the at least
data may be matched with a time course of Voice response, 45 one avatar attribute based on an association between the
skin response, or other Surrogate marker of brain activity to tonal voice quality and a positive emotional response as
associate a particular physiologic activity with visual contact measured by a brain activity measurement unit 212 and/or a
of the member of the population cohort 102 with a presented Surrogate marker measurement unit 230 and as associated by
characteristic. association unit 240. In another embodiment, a non-verbal
In the context of storing physiologic activity measurement 50 specification unit 356 may specify an object as an avatar
data, it should be understood that a data signal may first be attribute based on a positive mental state associated with the
encoded and/or represented in digital form (i.e., as digital object by emotion association module 342, for example.
data), prior to an assignment to at least one memory. For In another embodiment, an attribute specification unit 250
example, a digitally-encoded representation of user eye may specify an attribute that is a variant of the initially
movement data may be stored in a local memory, or may be 55 presented characteristic that provided the basis for the
transmitted for storage in a remote memory. identification of mental state. In this case, specification of
Thus, an operation may be performed relating either to a the variant of the characteristic may allow the system 100 to
local or remote storage of the digital data, or to another type titrate the reactions of a member of population cohort 102
of transmission of the digital data. Of course, as discussed among various flavors of an attribute. For example, the
herein, operations also may be performed relating to access 60 system may be used to explore reactions of a member of
ing, querying, processing, recalling, or otherwise obtaining population cohort 102 to various hairstyles of an avatar.
the digital data from a memory, including, for example, Accordingly, a short hairstyle may be presented, brain
transmission of the digital data from a remote memory. activity may be measured proximate to the presentation of
Accordingly, such operation(s) may involve elements the short hairstyle, a positive emotional reaction may be
including at least an operator (e.g., either human or com 65 associated with the brain activity, and based on the positive
puter) directing the operation, a transmitting computer, emotional mental state, a body attribute specification unit
and/or a receiving computer, and should be understood to 262 may specify a medium hairstyle as an avatar attribute for
US 9,495,684 B2
49 50
Subsequent presentation to the member of population cohort one characteristic not associated with the at least one mem
102. As the process repeats, differential reactions indicating ber of the population cohort 102. For example, if the
preference, approval, and/or disapproval of the member of member of the population cohort 102 has short curly hair, an
population cohort 102 for variations on the attribute will attribute specification unit 150 may present an avatar with
emerge to Suggest a “most preferred attribute for an avatar. long straight hair to the member of population cohort 102 via
Accordingly, in one embodiment, an attribute specifica presentation unit 170.
tion unit 350 may sendanavatar attribute and/or otheravatar In another embodiment, a clothing specification unit 260
output to a presentation device 364 for Subsequent presen may specify a particular color and/or style of clothing that
tation to a member of population cohort 102. The process contrasts with the clothing of the at least one member of a
may be repeated reiteratively as a means of refining an 10 population cohort 102, for presentation to the at least one
avatar to include characteristics or attributes that are linked member of a population cohort 102. Similarly, a speech
to a cohort in terms of preference or other mental state specification unit may present a particular speech attribute
eliciting quality. Such as a low pitch in an instance where the at least one
In some embodiments, an avatar attribute may include, for member of a population cohort 102 is determined to have a
example, an attribute of a three dimensional model, an 15 high-pitched Voice. Such contrasting characteristic presen
attribute of a two dimensional icon or image, an attribute of tations may elicit interesting mental state responses, particu
a text construct, an attribute of an audio construct, and/or an larly in cases where gender-opposite characteristics (such as
attribute of a personality connected with a screen name. An the voice pitch example above) are found to elicit favorable
avatar attribute may include an attribute of a real world physiologic responses in the member of a population cohort
object. In other embodiments, an avatar attribute may 102.
include, for example, an attribute of an embodiment, as of a Operation 604 depicts presenting at least one character
quality or a concept. In another example, an avatar attribute istic to at least one member of a established population
may include an attribute of an archetype. In another embodi cohort. For example, an attribute specification unit 150,
ment, an avatar attribute may include a manifestation or presentation device 364, presentation unit 270, and/or device
aspect of a continuing entity, not necessarily a person. For 25 106 may present at least one characteristic to at least one
example, a manifestation or aspect of a corporation may be member of a established population cohort. For example, an
an avatar attribute, as may be a specification for a class of attribute specification unit 250 may present an avatar wear
robots. Accordingly, a specified avatar attribute may be an ing Hello Kitty brand clothing as the at least one character
overall representation of what is happening in the population istic to a member of a demographic group including girls
cohort with respect to a presented characteristic. Other 30 ages 6-10 as the established population cohort. Alternatively,
examples of an avatar attribute are known to those of a speech specification unit 254 may present an avatar that
ordinary skill in the art. speaks with a British accent as the at least one characteristic
FIG. 6 illustrates alternative embodiments of the example to a demographic group including people living in Britain as
operational flow 500 of FIG. 5. FIG. 6 illustrates example the established population cohort. In another embodiment, a
embodiments where the presenting operation 510 may 35 facial attribute specification unit 258 may present an avatar
include at least one additional operation. Additional opera that includes a heavy beard as the at least one characteristic
tions may include operation 600, 602, 604, 606, and/or to a male demographic as the established population cohort.
operation 608. Operation 606 depicts presenting the at least one charac
Operation 600 depicts presenting the at least one charac teristic to at least one member of a established age cohort.
teristic to the at least one member of a population cohort, the 40 For example, an attribute specification unit 150, presentation
at least one characteristic associated with the at least one device 364, presentation unit 270, and/or device 106 may
member of the population cohort. For example, an attribute present at least one characteristic to at least one member of
specification unit 150, presentation device 364, presentation a established age cohort. For example, an attribute specifi
unit 270, and/or device 106 may present at least one char cation unit 250 may present an avatar wearing Van Dutch
acteristic to at least one member of a population cohort 102, 45 brand clothing as the at least one characteristic to a member
the at least one characteristic associated with the at least one of a demographic group including women aged 18-24 as the
member of the population cohort 102. For example, if the established age cohort. Alternatively, a speech specification
member of the population cohort 102 has brown hair and unit 254 may present an avatar that speaks with a British
brown eyes, an attribute specification unit 150 may present accent as the at least one characteristic to a demographic
an avatar with brown hair and brown eyes to the member of 50 group including British people aged 45-65 as the established
population cohort 102 via presentation unit 170. age cohort. In another embodiment, a facial attribute speci
In another embodiment, a clothing specification unit 360 fication unit 258 may present an avatar that includes an
may specify a particular color and/or style of clothing that approximation of Jennifer Aniston's nose as the at least one
matches the clothing of the at least one member of a characteristic to an age demographic between 18-24 years of
population cohort 102, for presentation to the at least one 55 age as the established age cohort.
member of a population cohort 102. Similarly, a speech Operation 608 depicts presenting the at least one charac
specification unit may present a particular speech attribute teristic to at least one member of a established gender
Such as a low pitch in an instance where the at least one cohort. For example, an attribute specification unit 150,
member of a population cohort 102 is determined to have a presentation device 364, presentation unit 270, and/or device
low-pitched voice. 60 106 may present at least one characteristic to at least one
Operation 602 depicts presenting the at least one charac member of a established gender cohort. For example, an
teristic to the at least one member of a population cohort, the attribute specification unit 250 may present an avatar having
at least one characteristic not associated with a member of an Oprah-like appearance as the at least one characteristic to
the population cohort. For example, an attribute specifica a member of a demographic group including women as the
tion unit 150, presentation device 364, presentation unit 270, 65 established gender cohort. Alternatively, a speech specifica
and/or device 106 may present at least one characteristic to tion unit 254 may present an avatar that speaks with a lilting
at least one member of a population cohort 102, the at least tone as the at least one characteristic to a demographic group
US 9,495,684 B2
51 52
including men as the established gender cohort. In another filters may be used to capture global texture information
embodiment, a body attribute specification unit 362 may about an a user's iris, and AdaBoost, a machine learning
present an avatar that includes a body builder's physique as algorithm, may be used to allow a presentation unit 370 to
the at least one characteristic to at least one member of a learn a discriminant classification principle from a pool of
male cohort as the established gender cohort. candidate iris feature sets. Iris image data may be thus
FIG. 7 illustrates alternative embodiments of the example grouped into race categories, for example, Asian and non
operational flow 500 of FIG. 5. FIG. 7 illustrates example Asian. See Qui et al., “Global Texture Analysis of Iris
embodiments where the presenting operation 510 may Images for Ethnic Classification.” Lecture notes in computer
include at least one additional operation. Additional opera science, Springer: Berlin/Heidelberg, Advances in Biomet
tions may include operation 700, 702, 704, and/or operation 10 rics, pp. 411-418 (2005).
7O6. Operation 702 depicts presenting at least one of a physical
Operation 700 depicts presenting the at least one charac appearance characteristic, a language characteristic, a cul
teristic to at least one member of a established ethnic cohort. tural characteristic, a personal interest characteristic, an
For example, an attribute specification unit 150, presentation educational characteristic, or a personality characteristic to
device 364, presentation unit 270, and/or device 106 may 15 the at least one member of the population cohort. For
present at least one characteristic to at least one member of example, an attribute specification unit 150, presentation
a established ethnic cohort, perhaps via a presentation unit device 364, presentation unit 270, and/or device 106 may
270. For example, an attribute specification unit 250 may present at least one of a physical appearance characteristic,
present an image including, for example, plaid clothing or a a language characteristic, a cultural characteristic, a personal
bagpipe as the at least one characteristic to a member of an interest characteristic, an educational characteristic, or a
ethnic group including Scotsmen as the established ethnic personality characteristic to the at least one member of the
cohort. Alternatively, a speech specification unit 254 may population cohort, perhaps via a presentation unit 270. For
present an avatar that speaks with a Southern drawl as the at example, an attribute specification unit 250 may present an
least one characteristic to a member of an ethnic group image including, for example, a tattoo as the at least one
including inhabitants of northern states as the established 25 physical appearance characteristic to a member of a popu
gender cohort. In another embodiment, a facial attribute lation cohort. Alternatively, a speech specification unit 254
specification unit 258 may present a Sun-tanned face as the may present an avatar that speaks with a French accent as the
at least one characteristic to at least one member of a at least one language characteristic to a member of a
Canadian cohort as the established ethnic cohort. population cohort. In another embodiment, an attribute
A number of methods of identifying ethnicity based on 30 specification unit 258 may present an item of sports equip
facial features are known in the art, for example, ethnicity ment Such as a tennis racquet as the at least one personal
identification may be formulated as a two-category classi interest characteristic to at least one member of a population
fication problem, for example, to classify the Subject as an cohort.
Asian or non-Asian. The input images may be resized to Operation 704 depicts presenting at least one of a facial
different scales. At each scale, a classic appearance-based 35 feature characteristic, a body feature characteristic, a cloth
face recognizer based on a linear discriminant analysis ing characteristic, or an accoutrement characteristic as the at
representation may be developed under a Bayesian statistical least one physical appearance characteristic. For example,
decision framework. An ensemble may then be constructed an attribute specification unit 150, clothing specification unit
by integrating classification results to arrive at a final 260, facial attribute specification unit 258, presentation
decision. The product rule may be used as an integration 40 device 364, presentation unit 270, and/or device 106 may
strategy. See Lu et al., “Ethnicity Identification from Face present at least one of a facial feature characteristic, a body
Images. Biometric Technology for Human Identification, feature characteristic, a clothing characteristic, or an accou
Eds. Jain et al., Proc. SPIE, Vol. 5404, pp. 114-123 (2004). trement characteristic as the at least one physical appearance
Subject ethnicity identification may be based on a number characteristic, perhaps via a presentation unit 270. For
of factors including skin and/or hair characteristics associ 45 example, an attribute specification unit 250 may present an
ated with ethnicity, Such as red hair among caucasians; Voice image including, for example, a piece of jewelry Such as a
and/or speech associated with ethnicity, such as French ring or a necklace as an accoutrement characteristic. Alter
accented English indicating French or French-Canadian natively, a facial attribute specification unit 258 may present
ethnicity; face pattern associated with ethnicity, such as eye an avatar having dimples as the facial feature characteristic.
shape, nose shape, face shape, or the like; and eye attributes 50 In another embodiment, a body attribute specification unit
Such as blue eyes among caucasians. In one embodiment, 262 may present a slender body type as the body feature
Gabor wavelets transformation and retina sampling from characteristic.
physiologic measurement data may be combined to extract Operation 706 depicts presenting at least one of a regional
key facial features, and Support vector machines may be dialect characteristic, an accent characteristic, a manner of
used for ethnicity classification. An experimental system has 55 speech characteristic, or a voice characteristic as the at least
used Gabor wavelets transformation and retina sampling in one language characteristic. For example, an attribute speci
combination to extract key facial features, and Support fication unit 150, voice specification unit 352, speech speci
vector machines were used for ethnicity classification, fication unit 354, presentation device 364, presentation unit
resulting in approximately 94% success for ethnicity esti 270, and/or device 106 may present at least one of a regional
mation under various lighting conditions. See Hosoi et al., 60 dialect characteristic, an accent characteristic, a manner of
“Ethnicity estimation with facial images. Sixth IEEE Inter speech characteristic, or a voice characteristic as the at least
national Conference on Automatic Face and Gesture Rec one language characteristic, perhaps via a presentation unit
ognition, pp. 195-200 (2004). Of course other methods of 270. For example, a voice specification unit 252 may present
ethnicity identification known in the art may be used. an avatar's speech having, for example, a Boston dialect as
An age, gender, and/or ethnicity characteristic may be 65 the at least one language characteristic. Alternatively, a
based on, for example, an iris pattern associated with an speech specification unit 254 may present an avatar having
asian user. For example, a bank of multichannel 2D Gabor an authoritative tone as the at least one language character
US 9,495,684 B2
53 54
istic. In another embodiment, a voice specification unit 252 Science interest characteristic, a mathematics interest char
may present speech including colloquialisms such as 'y'all' acteristic, or a college or university characteristic as the at
or “yins' as the manner of speech characteristic. least one educational interest characteristic, perhaps via a
FIG. 8 illustrates alternative embodiments of the example presentation unit 270. For example, an attribute specification
operational flow 500 of FIG. 5. FIG. 8 illustrates example unit 350 may present an avatar together with a book as the
embodiments where the presenting operation 510 may literature interest characteristic. Alternatively, a non-verbal
include at least one additional operation. Additional opera attribute specification unit 356 may present an avatar wear
tions may include operation 804, and/or operation 806. ing a computer as the Science interest characteristic. In
Operation 804 depicts presenting at least one of a gesture another embodiment, a clothing specification unit 260 may
characteristic, a food characteristic, a hairstyle characteris 10 present an avatar wearing the logo of a school as the college
tic, a cosmetics characteristic, or a facial hair characteristic or university characteristic.
as the at least one cultural characteristic. For example, an Operation 906 depicts presenting at least one of a talk
attribute specification unit 150, clothing specification unit ative personality characteristic, an aggressive personality
260, non-verbal attribute specification unit 256, voice speci characteristic, a deferential personality characteristic, a
fication unit 352, presentation device 364, presentation unit 15 humorous personality characteristic, or a whimsical person
270, and/or device 106 may present at least one of a gesture ality characteristic as the at least one personality character
characteristic, a food characteristic, a hairstyle characteris istic. For example, an attribute specification unit 150, cloth
tic, a cosmetics characteristic, or a facial hair characteristic ing specification unit 260, non-verbal attribute specification
as the at least one cultural characteristic, perhaps via a unit 256, voice specification unit 352, presentation device
presentation unit 270. For example, an attribute specification 364, presentation unit 270, and/or device 106 may present at
unit 350 may present an avatar's speech accompanied by least one of a talkative personality characteristic, an aggres
emphatic gesturing as the gesture characteristic. Alterna sive personality characteristic, a deferential personality
tively, a non-verbal attribute specification unit 356 may characteristic, a humorous personality characteristic, or a
present an avatar wearing traditional dress of an ethnicity as whimsical personality characteristic as the at least one
the at least one cultural characteristic. In another embodi 25 personality characteristic, perhaps via a presentation unit
ment, a facial attribute specification unit 358 may present 270. For example, an attribute specification unit 350 may
facial hair such as a mustache or a goatee as the facial hair present an avatar that rarely makes eye contact as the
characteristic. deferential personality characteristic. Alternatively, a speech
Operation 806 depicts presenting at least one of a hobby specification unit 354 may present an avatar that talks often
interest characteristic, a travel interest characteristic, a shop 30 and at a rapid rate as the talkative personality characteristic.
ping interest characteristic, a pet interest characteristic, a In another embodiment, a facial attribute specification unit
television or movie interest characteristic, a music interest 358 may present an avatar that smiles and or laughs fre
characteristic, a politics interest characteristic, or a sports quently as the humorous personality characteristic.
interest characteristic as the at least one personal interest FIG. 10 illustrates alternative embodiments of the
characteristic. For example, an attribute specification unit 35 example operational flow 500 of FIG. 5. FIG. 10 illustrates
150, clothing specification unit 260, non-verbal attribute example embodiments where the presenting operation 510
specification unit 256, voice specification unit 352, presen may include at least one additional operation. Additional
tation device 364, presentation unit 270, and/or device 106 operations may include operation 1000, and/or operation
may present at least one of a hobby interest characteristic, a 10O2.
travel interest characteristic, a shopping interest character 40 Operation 1000 depicts presenting at least one character
istic, a pet interest characteristic, a television or movie istic to at least one of a virtual world participant, a computer
interest characteristic, a music interest characteristic, a poli game participant, an online shopping participant, or a social
tics interest characteristic, or a sports interest characteristic networking website participant as the at least one member of
as the at least one personal interest characteristic, perhaps the population cohort. For example, an attribute specifica
via a presentation unit 270. For example, an attribute speci 45 tion unit 150, clothing specification unit 260, non-verbal
fication unit 350 may present an avatar together with a attribute specification unit 256, voice specification unit 352,
musical instrument as the music interest characteristic. presentation device 364, presentation unit 270, and/or device
Alternatively, a non-verbal attribute specification unit 356 106 may present at least one characteristic to at least one of
may present an avatar wearing a Star Trek t-shirt as the a virtual world participant, a computer game participant, an
television or movie interest characteristic. In another 50 online shopping participant, or a Social networking website
embodiment, a facial attribute specification unit 358 may participant as the at least one member of the population
present an avatar's face painted with a sports team's colors cohort, perhaps via a presentation unit 270. For example, an
as the sports interest characteristic. attribute specification unit 350 may present an avatar to a
FIG. 9 illustrates alternative embodiments of the example Second Life user as the virtual world participant. Alterna
operational flow 500 of FIG. 5. FIG. 9 illustrates example 55 tively, a non-verbal attribute specification unit 256 may
embodiments where the presenting operation 510 may present an avatar to a player of an online game. Such as
include at least one additional operation. Additional opera World of Warcraft as the computer game participant. In
tions may include operation 904, and/or operation 906. another embodiment, a facial attribute specification unit 358
Operation 904 depicts presenting at least one of a litera may present an avatar to a person visiting a myspace page
ture interest characteristic, a science interest characteristic, a 60 or a facebook page as the Social networking website par
mathematics interest characteristic, or a college or university ticipant.
characteristic as the at least one educational interest char Operation 1002 depicts presenting at least two character
acteristic. For example, an attribute specification unit 150, istics separated by a time delay to the at least one member
clothing specification unit 260, non-verbal attribute speci of a population cohort. For example, an attribute specifica
fication unit 256, voice specification unit 352, presentation 65 tion unit 150, clothing specification unit 260, non-verbal
device 364, presentation unit 270, and/or device 106 may attribute specification unit 256, voice specification unit 352,
present at least one of a literature interest characteristic, a presentation device 364, presentation unit 270, and/or device
US 9,495,684 B2
55 56
106 may present at least two characteristics separated by a remote device 306 for processing of the data, or all functions
time delay to the at least one member of a population cohort, of device 106 may be located in the home environment.
perhaps via a presentation unit 270. For example, an attri In one embodiment, fNIR module 214 may measure brain
bute specification unit 350 may present an avatar having a activation within milliseconds of a Subject encountering a
particular advertising logo displayed on an item of clothing, presented characteristic. For example, fNIR module 214
followed some time later by anotheravatar having a different may detect increased brain activity in the nucleus accum
logo displayed on the same avatar. Thus physiologic bens, SLEA, and thalamus within milliseconds of presenta
responses to the different logos may be measured and tion of a prepared food item to a member of population
compared. Alternatively, a non-verbal attribute specification cohort 102. Such a response may be considered proximate to
unit 256 may present an avatar having a particular physical 10 the presentation of the prepared food item.
feature such as blue eye color, followed some time later by Operation 1102 depicts measuring at least one physiologic
an avatar the same in all respects but for a change to green activity of the at least one member of the population cohort
eye color. Thus physiologic responses to the different eye using at least one of electroencephalography, computed
colors may be measured and compared. axial tomography, positron emission tomography, magnetic
FIG. 11 illustrates alternative embodiments of the 15 resonance imaging, functional magnetic resonance imaging,
example operational flow 500 of FIG. 5. FIG. 11 illustrates functional near-infrared imaging, or magnetoencephalogra
example embodiments where the measuring operation 520 phy, the at least one physiologic activity proximate to the at
may include at least one additional operation. Additional least one presented characteristic. For example, the device
operations may include operation 1100, 1102, and/or opera 206, physiologic activity measurement unit 210, brain activ
tion 1104. ity measurement unit 212, fMRI module 216, MEG module
Operation 1100 depicts measuring at least one physiologic 218, EEG module, PET module, and/or fNIR module 214
activity of the at least one member of the population cohort can measure at least one physiologic activity of the at least
using functional near-infrared imaging, the at least one one member of the population cohort using at least one of
physiologic activity proximate to the at least one presented electroencephalography, computed axial tomography, posi
characteristic. For example, the device 206, physiologic 25 tron emission tomography, magnetic resonance imaging,
activity measurement unit 210, brain activity measurement functional magnetic resonance imaging, functional near
unit 212, and/or fNIR module 214 can measure at least one infrared imaging, or magnetoencephalography, the at least
physiologic activity of the at least one member of the one physiologic activity proximate to the at least one pre
population cohort using functional near-infrared imaging, sented characteristic. In one embodiment, a member of the
the at least one physiologic activity proximate to the at least 30 population cohort 102 can be monitored by MEG module
one presented characteristic. In one embodiment, a member 218 during presentation of a characteristic to a member of
of the population cohort 102 can be monitored by fNIR population cohort 102. fMRI module 216, MEG module
module 214 during presentation of a characteristic to a 218, EEG module, PET module, and/or fNIR module 214
member of population cohort 102. fNIR module 214 can can measure brain activity as the physiologic activity of the
measure brain activity as the physiologic activity of the at 35 at least one member of the population cohort.
least one member of the population cohort. Operation 1104 depicts measuring at least one brain
Proximity of the fNIR measurement to presentation, activity surrogate marker of the at least one member of the
viewing, and/or perception of the characteristic can be population cohort, the at least one brain activity Surrogate
determined by presentation unit 270, device 206, physi marker proximate to the at least one presented characteristic.
ologic activity measurement unit 210, brain activity mea 40 For example, the device 206, physiologic activity measure
surement unit 212, and/or fNIR module 214. For example, ment unit 210, surrogate marker measurement unit 230, iris
the time of presentation of the characteristic can be matched response module 232, gaze tracking module 234, skin
with brain activity measured by fNIR module 214. In response module 236, and/or voice response module can
another embodiment, eye movement and/or gaze tracking measure at least one brain activity Surrogate marker of the at
analysis can determine the time that a subject’s eyes contact 45 least one member of the population cohort, the at least one
a presented characteristic, and this can be matched to the brain activity Surrogate marker proximate to the at least one
time of a measured brain activity by fNIR module 214. In presented characteristic. In one embodiment, Surrogate
still another embodiment, brain activity in the visual cortex marker measurement unit 230 can measure any of a variety
or other perception-indicative brain area or areas can be of physiological responses to a presented characteristic, the
measured by fNIR module 214 as an indicator of a subjects 50 physiological responses indicative of a brain activation
viewing of a presented characteristic; continued measure and/or mental state response to the presented characteristic.
ment of physiologic activity by fNIR module 214 can then For example, Surrogate marker measurement unit 230 may
measure a response to the viewing of the characteristic. measure a physiological attribute such as heart rate, respi
In one embodiment, the fNIR module 214 may be located ration, perspiration, temperature, skin coloring, skin electri
in a kiosk in a public area such as a shopping mall. In Such 55 cal response, eye movement, pupil dilation, voice stress,
an environment, images of individuals may be captured by body or facial tic, or the like, before, during, and/or after
photography or videography and compared with reference presentation of a characteristic to a member of population
population cohort 104 image profiles to identify members of cohort 102. Such measurements may include, for example,
population cohort 102. In another embodiment, presentation an increase in heart rate over a time interval as measured by
of a characteristic by, for example, presentation unit 270 and 60 a heart rate monitor embedded in a user interface; increased
measurement of physiologic activity by, for example fNIR eye movements as measured by an image capture device
module 214 may occur in a home computing environment, Such as a video camera, or changes in the galvanic skin
with characteristic presentation data sent by, for example, a response as measured by electrodes embedded in a user
remote attribute specification unit 350 to a presentation interface or included in device 106.
device 364 located in the home. The physiologic activity 65 In one embodiment, an iris response module 232 and/or
measurement unit 310 unit may be located in the home gaze tracking module 234 may acquire data from a user
environment and send measurement data via a network to a monitoring device such as a video capture device or a video
US 9,495,684 B2
57 58
communication device, for example, when a Subject's image include, for example, as discussed above, layered Voice
is captured as a photograph or video when using an appli analysis, Voice stress analysis, or the like.
cation, Such as a webcam, or when a Subjects image is The reaction of a Subject to a presented characteristic Such
captured when communicating via a photography or video as an advertisement in a computerized game world or in
based application. Other sources of image data may include another virtual world may be a recognizable Vocal excla
biometric data such as facial pattern data, eye scanning data, mation such as “Wow, that’s nice” that may be detectable
or the like. Such image data may indicate, for example, by a voice response module 338 such as a microphone
alertness, attention, approval, disapproval, or the like, as monitoring an interaction between the Subject and, for
discussed below.
Image data may include results of visual spectrum imag 10
example, a presentation device 364 and/or device 306. A
ing that can image changes in facial expression, body voice response module 338 may also include a speech
movement, or the like that can be indicative of a physiologi recognition function Such as a software program or compu
cal activity, brain activity, and/or mental state. User image tational device that can identify and/or record an utterance of
data may also be obtained from other kinds of imaging Such a Subject as speech or voice data.
as infrared imaging that can read a heat signature and/or 15 In another embodiment, an iris response module may
ultrasound imaging. Further, reflected image or refracted record changes in the movement of a Subject’s iris (with
image data may also be obtained by physiologic activity corresponding changes in the size of the pupil) before,
measurement unit 210 and/or Surrogate marker measure during, and/or after presentation of a characteristic to a
ment unit 230. Near infrared imaging may be used to test for member of a population cohort 102. Such measurements of
baseline physiologic states and metabolism, as well as physiologic activity that indicate brain activity and/or men
physiologic and metabolic changes. Image data may be of all tal state may be carried out at a time that is proximate to
or a portion of a member of population cohort 102 such as presentation of a characteristic to a member of population
a head-to-toe image, a face image, an image of fingers, an cohort 102.
image of an eye, or the like. Such images may be in the In one embodiment, a gaze tracking module 334 may
visual or non-visual wavelength range of the electromag 25 include a camera that can monitor a Subject's eye move
netic spectrum. ments in order to determine whether the subject looks at a
Alertness or attention can be measured, for example, by presented characteristic, for example, during a certain time
measuring eye movements, body movements, pointing period.
device manipulation, and/or task proficiency (e.g., are a Gaze tracking module 334 and/or iris response module
Subject’s eyelids drooping, is a Subjects head nodding, is a 30 332 may include a Smart camera that can capture images,
Subject failing or Succeeding to activate on-screen items process them and issue control commands within a milli
when prompted, does a subject respond to a sound, or the second time frame. Such smart cameras are commercially
like). Alertness or attention to, for example, an advertise available (e.g., Hamamatsu's Intelligent Vision System;
ment may be gauged from a Subjects interaction with the http://i.p.hamamatsu.com/en/product info/index.html). Such
advertisement. Interest in the advertisement as the presented 35 image capture systems may include dedicated processing
characteristic in the form of face pattern data (e.g., a Smile elements for each pixel image sensor. Other camera systems
on an image of the Subject's face), pointing device manipu may include, for example, a pair of infrared charge coupled
lation data (e.g., a mouse click on an onscreen advertisement device cameras to continuously monitor pupil size and
icon as the presented characteristic), and/or eye movement position as a user watches a visual target moving forward
data (e.g., repeated eye movements toward the advertise 40 and backward. This can provide real-time data relating to
ment as the presented characteristic), or the like. pupil accommodation relative to objects on, for example, a
FIG. 12 illustrates alternative embodiments of the display 372. (e.g. http://ip.hamamatsu.com/en/rd/publica
example operational flow 500 of FIG. 5. FIG. 12 illustrates tion/scientific american/common/pdf/scientific 0608.pdf).
example embodiments where the measuring operation 520 Eye movement and/or iris movement may also be mea
may include at least one additional operation. Additional 45 Sured by video-based eye trackers. In these systems, a
operations may include operation 1206. camera focuses on one or both eyes and records eye move
Operation 1206 depicts measuring the at least one brain ment as the viewer looks at a stimulus. Contrast may be used
activity surrogate marker of the at least one member of the to locate the center of the pupil, and infrared and near
population cohort using at least one of iris dilation or infrared non-colluminated light may be used to create a
constriction, gaze tracking, skin response, or voice response, 50 corneal reflection. The vector between these two features
the at least one brain activity Surrogate marker proximate to can be used to compute gaze intersection with a surface after
the at least one characteristic. For example, the device 206, a calibration for an user 106.
physiologic activity measurement unit 210, Surrogate Two types of gaze tracking or eye tracking techniques
marker measurement unit 230, iris response module 232, include bright pupil eye tracking and dark pupil eye track
gaze tracking module 234, skin response module 236, and/or 55 ing. Their difference is based on the location of the illumi
Voice response module can measure the at least one brain nation source with respect to the optics. If the illumination
activity surrogate marker of the at least one member of the is coaxial with the optical path, then the eye acts as a
population cohort using at least one of iris dilation or retroreflector as the light reflects off the retina, creating a
constriction, gaze tracking, skin response, or voice response, bright pupil effect similar to red eye. If the illumination
the at least one brain activity Surrogate marker proximate to 60 Source is offset from the optical path, then the pupil appears
the at least one presented characteristic. In one embodiment, dark.
a voice response module can measure speech captured by a Bright Pupil tracking creates greater iris/pupil contrast
microphone during presentation of a characteristic. Speech allowing for more robust eye tracking with all iris pigmen
or voice can be measured, for example, by examining voice, tation and greatly reduces interference caused by eyelashes
Song, and/or other vocal utterances of a subject before, 65 and other obscuring features. It also allows for tracking in
during, and/or after presentation of a characteristic to a lighting conditions ranging from total darkness to very
member of population cohort 102. Such measurements may bright light. However, bright pupil techniques are not rec
US 9,495,684 B2
59 60
ommended for tracking outdoors as extraneous IR sources are seen. Analyses such as these may be used as evidence of
may interfere with monitoring. specific trends in visual behavior.
Eye tracking configurations can vary; in Some cases the A similar method sums the eye data of multiple partici
measurement apparatus may be head-mounted, in some pants during a given time interval as a heat map. A heat map
cases the head should be stable (e.g., stabilized with a chin may be produced by a commercial Software package, and
rest), and in Some cases the eye tracking may be done shows the density of eye fixations for several participants
remotely to automatically track the head during motion. Superimposed on the original stimulus, for example, a maga
Most eye tracking systems use a sampling rate of at least 30 Zine cover. Red and orange spots represent areas with high
HZ. Although 50/60 Hz is most common, many video-based densities of eye fixations. This allows one to examine which
eye trackers run at 240, 350 or even 1000/1250 Hz, which 10 regions attract the attention and focus of the viewer.
is recommended in order to capture the detail of the very Commercial eye tracking applications include web usabil
rapid eye movements during reading, or during studies of ity, advertising, sponsorship, package design and automotive
neurology. engineering. Eye tracking studies often present a target
Eye movements are typically divided into fixations, when stimulus to a sample of consumers while an eye tracker is
the eye gaze pauses in a certain position, and saccades, when 15 used to record the activity of the eye. Examples of target
the eye gaze moves to another position. A series of fixations stimuli may include websites, television programs, sporting
and Saccades is called a scanpath. Most information from the events, films, commercials, magazines, newspapers, pack
eye is made available during a fixation, not during a saccade. ages, shelf displays, consumer systems (e.g., ATMs, check
The central one or two degrees of the visual angle (the out systems, kiosks), and Software. The resulting data can be
fovea) provide the bulk of visual information; input from statistically analyzed and graphically rendered to provide
larger eccentricities (the periphery) generally is less infor evidence of specific visual patterns. By examining fixations,
mative. Therefore the locations of fixations along a scanpath saccades, pupil dilation, blinks, and a variety of other
indicate what information loci on the stimulus were pro behaviors, researchers can determine a great deal about the
cessed during an eye tracking session. On average, fixations effectiveness of a given medium or product.
last for around 200 milliseconds during the reading of 25 A prominent field of eye tracking research is web usabil
linguistic text, and 350 milliseconds during the viewing of ity. While traditional usability techniques are often quite
a scene. Preparing a saccade towards a new goal takes powerful in providing information on clicking and scrolling
around 200 milliseconds. patterns, eye tracking offers the ability to analyze user
Scanpaths are useful for analyzing cognitive intent, inter interaction between the clicks. This provides insight into
est, and salience. Other biological factors (some as simple as 30 which features are the most eye-catching, which features
gender) may affect the Scanpath as well. Eye tracking in cause confusion, and which ones are ignored altogether.
human-computer interaction typically investigates the scan Specifically, eye tracking can be used to assess search
path for usability purposes, or as a method of input in efficiency, branding, online advertisement, navigation
gaZe-contingent displays, also known as gaze-based inter usability, overall design, and many other site components,
faces. 35 including avatar attributes. Analyses may target a prototype
There are two primary components to most eye tracking avatar or competitor avatars in addition to the main avatar.
studies: statistical analysis and graphic rendering. These are Eye tracking is commonly used in a variety of different
both based mainly on eye fixations on specific elements. advertising media. Commercials, print ads, online ads, and
Statistical analyses generally Sum the number of eye data sponsored programs are all conducive to analysis with eye
observations that fall in a particular region. Commercial 40 tracking technology. Analyses may focus on visibility of a
Software packages may analyze eye tracking and show the target product or logo in the context of a magazine, news
relative probability of eye fixation on each feature in a paper, website, virtual world, or televised event. This allows
website. This allows for a broad analysis of which site researchers to assess in great detail how often a sample of
elements received attention and which ones were ignored. consumers fixates on the target logo, product, or advertise
Other behaviors such as blinks, saccades, and cognitive 45 ment. In this way, an advertiser can quantify the Success of
engagement can be reported by commercial software pack a given campaign in terms of actual visual attention.
ages. Statistical comparisons can be made to test, for Eye tracking also provides package designers with the
example, competitors, prototypes or Subtle changes to an opportunity to examine the visual behavior of a consumer
avatar and/or web design. They can also be used to compare while interacting with a target package. This may be used to
participants in different demographic groups and/or popu 50 analyze distinctiveness, attractiveness and the tendency of
lation cohorts. Statistical analyses may quantify where Sub the package to be chosen for purchase. Eye tracking is often
jects look, sometimes directly, and sometimes based on used while the target product is in the prototype stage.
models of higher-order phenomena (e.g., cognitive engage Prototypes are tested against each other and against com
ment). petitors to examine which specific elements are associated
In addition to statistical analysis, it is often useful to 55 with high visibility and/or appeal.
provide visual depictions of eye tracking results. One Another application of eye tracking research is in the field
method is to create a video of an eye tracking testing session of automotive design. Eye tracking cameras may be inte
with the gaze of a participant Superimposed upon it. This grated into automobiles to provide the vehicle with the
allows one to effectively see through the eyes of the subject capacity to assess in real-time the visual behavior of the
during interaction with a presented characteristic. Another 60 driver. The National Highway Traffic Safety Administration
method graphically depicts the scanpath of a single partici (NHTSA) estimates that drowsiness is the primary causal
pant during a given time interval. Analysis may show each factor in 100,000 police-reported accidents per year. Another
fixation and eye movement of a participant during a search NHTSA study suggests that 80% of collisions occur within
on a virtual shelf display of breakfast cereals, analyzed and three seconds of a distraction. By equipping automobiles
rendered with a commercial software package. For example, 65 with the ability to monitor drowsiness, inattention, and
a different color may represent one second of viewing time, cognitive engagement driving safety could be dramatically
allowing for a determination of the order in which products enhanced. Lexus(R claims to have equipped its LS 460
US 9,495,684 B2
61 62
automobile with the first driver monitor system in 2006, least two presented characteristics. For example, the device
providing a warning if the driver takes his or her eye off the 206, physiologic activity measurement unit 210, brain activ
road. ity measurement unit 212, and/or Surrogate marker measure
Eye tracking is also used in communication systems for ment unit 230 can measure at least one physiologic activity
disabled persons, allowing the user to speak, mail, Surf the of the at least one member of the population cohort, the at
web and so with only the eyes as tool. Eye control works least one physiologic activity proximate to at least two
even when the user has involuntary body movement as a presented characteristics. In one embodiment, a combination
result of cerebral palsy or other disability, and/or when the of characteristics can be presented to a member of popula
user wears glasses. tion cohort 102 in the context of, for example an avatar. For
A Surrogate marker measurement unit may also measure 10 example, brain activity measurement unit 212 may measure
face pattern, for example, by measuring user facial features, brain activity during presentation of a set of avatar features,
perhaps in relation to a control user face pattern image Such as a facial feature and a voice feature, to a member of
captured when the user was not interacting with a presented population cohort 102.
characteristic. Alternatively, a Subject's face pattern may be Operation 1304 depicts measuring at least one physi
compared to an average face pattern compiled from a large 15 ologic activity of the at least one member of the population
number of faces in the population cohort. In one embodi cohort, the at least one physiologic activity proximate to at
ment, the reaction of a member of population cohort 102 to least one of a presented hairstyle characteristic or a pre
an onscreen avatar may be a Smile or a frown that may be sented eye color characteristic. For example, the device 206,
detectable by a camera monitoring the interaction. Informa physiologic activity measurement unit 210, brain activity
tion Suggesting that a user Smiles in response to viewing a measurement unit 212, and/or Surrogate marker measure
presented characteristic may be transmitted to association ment unit 230 can measure at least one physiologic activity
unit 340. of the at least one member of the population cohort, the at
A Surrogate marker measurement unit 330 may include a least one physiologic activity proximate to at least one of a
pen including electronic sensing capability that may include presented hairstyle characteristic or a presented eye color
the capability to monitor a subjects hand for temperature, 25 characteristic. In one embodiment, physiologic activity mea
blood flow, tremor, fingerprints, or other attributes. Reaction Surement unit 210 can measure brain activity and/or eye
time among young males aged 18-28 that play video games movement activity during presentation of an avatar to a
with some frequency may be distinguishable from average member of population cohort 102, the avatar having, for
reaction times of women and/or users in older age groups. example, wavy blond hair and/or blue eyes. Choice of
In another embodiment, a Surrogate marker measurement 30 hairstyle characteristic may be based on a population cohort
unit 330 and/or device 306 may anonymize physiologic 104 characteristic, a member of population cohort 102
activity measurement data acquired during a subjects inter characteristic, and/or previous avatar specification data.
action with one or more presented characteristics. Anony Operation 1306 depicts measuring at least one physi
mization of member of population cohort 102 data may be ologic activity of the at least one member of the population
accomplished through various methods known in the art, 35 cohort, the at least one physiologic activity proximate to at
including data coding, k-anonymization, de-association, least one presented characteristic of the member of the
pseudonymization, or the like. population cohort. For example, the device 206, physiologic
FIG. 13 illustrates alternative embodiments of the activity measurement unit 210, brain activity measurement
example operational flow 500 of FIG. 5. FIG. 13 illustrates unit 212, and/or Surrogate marker measurement unit 230 can
example embodiments where the measuring operation 520 40 measure at least one physiologic activity of the at least one
may include at least one additional operation. Additional member of the population cohort, the at least one physi
operations may include operation 1300. 1302, 1304, and/or ologic activity proximate to at least one presented charac
1306. teristic of the member of the population cohort. In one
Operation 1300 depicts measuring at least one physi embodiment, physiologic activity measurement unit 210 can
ologic activity of the at least one member of the population 45 measure brain activity and/or iris movement activity during
cohort in near real time, the at least one physiologic activity presentation of an appearance characteristic of the member
proximate to the at least one presented characteristic. For of the population cohort 102, for example, as detected by a
example, the device 206, physiologic activity measurement camera monitoring the member of the population cohort. In
unit 210, brain activity measurement unit 212, and/or sur another embodiment, physiologic activity measurement unit
rogate marker measurement unit 230 can measure at least 50 210 can measure brain activity and/or voice stress response
one physiologic activity of the at least one member of the during presentation of a voice characteristic of the member
population cohort 102 in near real time, the at least one of the population cohort 102, for example, as detected by a
physiologic activity proximate to the at least one presented microphone monitoring the member of the population
characteristic. In one embodiment, the brain activity mea cohort.
Surement unit 212 can measure brain activity in the member 55 FIG. 14 illustrates alternative embodiments of the
of population cohort 102 at the milliseconds-to-seconds time example operational flow 500 of FIG. 5. FIG. 14 illustrates
frame, inclusive of processing time. Accordingly, the meth example embodiments where the measuring operation 520
ods discussed herein, including Surrogate marker measure may include at least one additional operation. Additional
ment functions, can measure responses of a member of the operations may include operation 1400, 1402, 1404, and/or
population cohort 102 in near real time, which may include 60 1406.
a delay between the occurrence of a proximate and/or Operation 1400 depicts measuring at least one physi
response event and the use of the processed proximate ologic activity of the at least one member of the population
measurement data, e.g., for further processing and/or for cohort, the at least one physiologic activity proximate to at
Subsequent display. least one presented characteristic of a different member of
Operation 1302 depicts measuring at least one physi 65 the population cohort. For example, the device 206, physi
ologic activity of the at least one member of the population ologic activity measurement unit 210, brain activity mea
cohort, the at least one physiologic activity proximate to at Surement unit 212, and/or Surrogate marker measurement
US 9,495,684 B2
63 64
unit 230 can measure at least one physiologic activity of the Operation 1500 depicts associating at least one electrical
at least one member of the population cohort, the at least one brain activity with the at least one mental state. For example,
physiologic activity proximate to at least one presented the device 106, association unit 140, emotion association
characteristic of a different member of the population cohort module 242, attention association module 244, and/or cog
104. In one embodiment, brain activity measurement unit nition association module 346 can associate at least one
312 can measure brain activity during presentation of an electrical brain activity with at least one mental state. In one
appearance characteristic of a member of population cohort embodiment, association unit 240 can receive an electrical
104 different from the member of population cohort 102 that brain activity measurement, such as measurement of elec
is responding to the characteristic. For example, a middle trical activation of the hippocampus, in response to presen
aged member of population cohort 102 may be presented 10 tation of a characteristic to a member of population cohort
with a characteristic of a child-aged member of the popu 102. The electrical brain activity measurement may be
lation cohort 104, during which physiologic activity mea received from, for example, EEG module 220. Association
unit 240 can then search one or more functional brain
surement unit 210, brain activity measurement unit 212, mapping databases based on the electrical brain activity
and/or Surrogate marker measurement unit 230 can measure 15 measurement to find one or more matching mental states.
a physiological response to the presentation of the charac For example, activation of the hippocampus is associated in
teristic. the literature with enhanced motivational drive for food and
Operation 1402 depicts measuring at least one physi drugs. Thus an association may be made between hippocam
ologic activity of the at least one member of the population pus activation and enhanced motivational drive, for
cohort, the at least one physiologic activity proximate to at example, based on research findings (e.g., Wang et al.,
least one presented characteristic of a member of a different “Gastric stimulation in obese subjects activates the hip
population cohort. For example, the device 206, physiologic pocampus and other regions involved in brain reward cir
activity measurement unit 210, brain activity measurement cuitry.” PNAS, vol. 103, pp. 15641-45 (2006).
unit 212, and/or Surrogate marker measurement unit 230 can In another embodiment, attention association module 244
measure at least one physiologic activity of the at least one 25 can receive a brain activity measurement based on electrical
member of the population cohort, the at least one physi brain activity, such as a measurement from MEG module
ologic activity proximate to at least one presented charac 218. In this embodiment, an avatar with a specific set of
teristic of a member of a different population cohort. In one facial features as the at least one characteristic may be
embodiment, brain activity measurement unit 312 can mea presented to a member of population cohort 102, during
Sure brain activity during presentation of a speech charac 30 which presentation MEG module 218 can measure the
teristic of a member of a population cohort that is different brain's electrical activity based on indicated magnetic fields.
from the population cohort 104 to which member of popu For example, presentation of the avatar with the specific set
lation cohort 102 belongs. For example, for a population of facial features may elicit electrical activity in the pre
cohort 104 including males aged 18-24, a member of popu frontal and/or parietal areas of the brain. Association unit
lation cohort 102 (i.e., a male aged 18-24) may be presented 35 240 may thus match the activation measurement pattern with
with a characteristic of a female, and thus a member of a a known pattern of brain activation from research indicating
different population cohort, during which physiologic activ which brain areas are activated when attention is required.
ity measurement unit 210, brain activity measurement unit For example, activation of the thalamic reticular nucleus is
212, and/or Surrogate marker measurement unit 230 can also associated with selective attention. See Contreras et al.,
measure a physiological response to the presentation of the 40 “Inactivation of the Interoceptive Insula Disrupts Drug
female characteristic. Craving and Malaise Indicated by Lithium.” Science, vol.
Operation 1404 depicts measuring with permission at 318, pp. 655-658 (26 Oct. 2007).
least one physiologic activity of the at least one member of Operation 1502 depicts associating at least one brain
the population cohort, the at least one physiologic activity blood oxygen level with the at least one mental state. For
proximate to the at least one presented characteristic. For 45 example, the device 106, association unit 140, emotion
example, the device 206, physiologic activity measurement association module 242, attention association module 244.
unit 210, brain activity measurement unit 212, and/or sur and/or cognition association module 346 can associate at
rogate marker measurement unit 230 can measure with least one hemodynamic brain activity with at least one
permission at least one physiologic activity of the at least mental state. In one embodiment, association unit 240 can
one member of the population cohort, the at least one 50 receive a brain blood oxygen level activity measurement,
physiologic activity proximate to the at least one presented Such as measurement of activation of the right prefrontal and
characteristic. In one embodiment, after receiving permis parietal areas, in response to presentation of a characteristic
sion from a member of population cohort 102, brain activity to a member of population cohort 102. The hemodynamic
measurement unit 312 can measure brain activity during brain activity measurement indicating activation of the right
presentation of a physical characteristic to the member of 55 prefrontal and parietal areas may be received from, for
population cohort 102. For example, a member of popula example, fNIR module 214. Association unit 240 can then
tion cohort 102 may click an acceptance box asking for search one or more functional brain mapping databases
permission to measure a physiologic activity, for example, based on the brain activity measurement to find one or more
during a virtual world session, an online gaming session, an matching mental states. For example, activation of the right
online shopping session, a Social networking session, or the 60 prefrontal and parietal areas is associated in the literature
like. with attention. Thus an association may be made between
FIG. 15 illustrates alternative embodiments of the right prefrontal and parietal areas activation and increased
example operational flow 500 of FIG. 5. FIG. 15 illustrates attention to the presented characteristic by association unit
example embodiments where the associating operation 530 240, for example, based on research findings. See Cabeza et
may include at least one additional operation. Additional 65 al, “Imaging Cognition II: An Empirical Review of 275 PET
operations may include operation 1500, 1502, 1504, 1506, and fMRI Studies,” J. Cognitive Neurosci., vol. 12, pp. 1-47
and/or 1508. (2000).
US 9,495,684 B2
65 66
Operation 1504 depicts associating at least one of glucose activation of the nucleus accumbens may be received from,
metabolism or blood flow with the at least one mental state. for example, MEG module 218. Emotion association mod
For example, the device 106, association unit 140, emotion ule 242 and/or attention association module 244 can then
association module 242, attention association module 244. match the nucleus accumbens activation measurement to
and/or cognition association module 346 can associate at one or more corresponding mental states. For example,
least one of glucose metabolism or blood flow with at least activation of the nucleus accumbens is associated in the
one mental state. In one embodiment, association unit 240 literature with product preference. Thus an association can
can receive a glucose metabolism measurement, Such as be made between activation of the nucleus accumbens and
measurement of activation of the right hemisphere of the preference for the presented avatar possessing a product, by
bilateral fusiform gyms, in response to presentation of a face 10 association unit 240, for example, based on research find
characteristic to a member of population cohort 102. The ings. See Wise, “Thought Police: How Brain Scans Could
glucose metabolism brain activity measurement indicating Invade Your Private Life.” Popular Mechanics, (November
activation of the right hemisphere of the bilateral fusiform 2007).
gyms may be received from, for example, PET module 222. FIG. 16 illustrates alternative embodiments of the
Attention association module 244 can then match the brain 15 example operational flow 500 of FIG. 5. FIG. 16 illustrates
activity measurement to one or more corresponding mental example embodiments where the associating operation 530
states. For example, activation of the right hemisphere of the may include at least one additional operation. Additional
bilateral fusiform gyms is associated in the literature with operations may include operation 1600, 1602, 1604, 1606,
increased attention to faces. Thus an association can be and/or 1608.
made between activation of the right hemisphere of the Operation 1600 depicts associating activity in the pre
bilateral fusiform gyms and increased attention to the pre frontal cortex with a mental state indicating brand prefer
sented face characteristic by association unit 240, for ence. For example, the device 106, association unit 140,
example, based on research findings. See Cabeza et al. emotion association module 242, attention association mod
“Imaging Cognition II: An Empirical Review of 275 PET ule 244, and/or cognition association module 346 can asso
and fMRI Studies,” J. Cognitive Neurosci., vol. 12, pp. 1-47 25 ciate activity in the prefrontal cortex with a mental state
(2000). indicating brand preference. In one embodiment, association
Operation 1506 depicts associating activity in the ventro unit 240 can receive a brain activity measurement indicating
medial prefrontal cortex with a mental state indicating activation of the prefrontal cortex in response to presentation
approval. For example, the device 106, association unit 140, to a member of population cohort 102 of an image of a
emotion association module 242, attention association mod 30 product brand as the at least on characteristic. The brain
ule 244, and/or cognition association module 346 can asso activity measurement indicating activation of the prefrontal
ciate activity in the ventromedial prefrontal cortex with a cortex may be received from, for example, fNIR module
mental state indicating approval. In one embodiment, asso 314. Emotion association module 242, attention association
ciation unit 240 can receive a brain activity measurement module 244, and/or cognition association module 346 can
indicating activation of the Ventromedial prefrontal cortex, 35 then match the prefrontal cortex activation measurement to
in response to presentation to a member of population cohort one or more corresponding mental states. For example,
102 of an avatar having a characteristic of interest. The brain activation of the prefrontal cortex is associated in the
activity measurement indicating activation of the Ventrome literature with brand preference. Thus an association can be
dial prefrontal cortex may be received from, for example, made between activation of the prefrontal cortex and pref
fMRI module 216. Emotion association module 242 and/or 40 erence for the presented image of a product brand, by
attention association module 244 can then match the ven association unit 240, for example, based on research find
tromedial prefrontal cortex activation measurement to one or ings. See Kenning et al., “Neuroeconomics: an overview
more corresponding mental states. For example, activation from an economic perspective.” Brain Res. Bull., vol. 67,
of the ventromedial prefrontal cortex is associated in the pp. 343-354 (2005).
literature with preference or approval. Thus an association 45 Operation 1602 depicts associating activity in the dorso
unit 240 can make an association between activation of the lateral prefrontal cortex, the posterior parietal cortex, the
ventromedial prefrontal cortex and approval of the presented occipital cortex, and the left premotor area with a mental
avatar having the characteristic of interest, for example, state indicating brand preference. For example, the device
based on research findings. See Cabeza et al., “Imaging 106, association unit 140, emotion association module 242,
Cognition II: An Empirical Review of 275 PET and fMRI 50 attention association module 244, and/or cognition associa
Studies,” J. Cognitive Neurosci., vol. 12, pp. 1-47 (2000). tion module 346 can associate activity in the dorsolateral
Operation 1508 depicts associating activity in at least one prefrontal cortex, the posterior parietal cortex, the occipital
of the frontopolar cortex, the prefrontal cortex, the ventral cortex, and the left premotor area with a mental state
striatum, the orbitofrontal prefrontal cortex, the amygdala, indicating brand preference. In one embodiment, association
or the nucleus accumbens with a mental state indicating 55 unit 240 can receive a brain activity measurement indicating
approval. For example, the device 106, association unit 140, activation of the dorsolateral prefrontal cortex, the posterior
emotion association module 242, attention association mod parietal cortex, the occipital cortex, and the left premotor
ule 244, and/or cognition association module 346 can asso area in response to presentation to a member of population
ciate activity in at least one of the frontopolar cortex, the cohort 102 of a brand embodied in an avatar as the at least
prefrontal cortex, the ventral striatum, the orbitofrontal 60 on characteristic. The brain activity measurement indicating
prefrontal cortex, the amygdala, or the nucleus accumbens activation of the dorsolateral prefrontal cortex, the posterior
with a mental state indicating approval. In one embodiment, parietal cortex, the occipital cortex, and the left premotor
association unit 240 can receive a brain activity measure area may be received from, for example, fNIR module 314.
ment indicating activation of the nucleus accumbens in Emotion association module 242, attention association mod
response to presentation to a member of population cohort 65 ule 244, and/or cognition association module 346 can then
102 of an avatar possessing a product as the at least one match the dorsolateral prefrontal cortex, the posterior pari
characteristic. The brain activity measurement indicating etal cortex, the occipital cortex, and the left premotor area
US 9,495,684 B2
67 68
activation measurement to one or more corresponding men to one or more corresponding mental states. For example,
tal states. For example, activation of the dorsolateral pre activation of the insula is associated in the literature with
frontal cortex, the posterior parietal cortex, the occipital pain, distress, and other negative emotional states. Thus an
cortex, and the left premotor area is associated in the association can be made between emotional disapproval of
literature with brand preference. Thus an association can be the presented characteristic and activation of the insula, by
made by association unit 240 between preference for the emotional association module 242 and/or association unit
presented image of a product brand and activation of the 240, for example, based on research findings. See Kenning
dorsolateral prefrontal cortex, the posterior parietal cortex, et al., “Neuroeconomics: an overview from an economic
the occipital cortex, and the left premotor area, for example, perspective.” Brain Res. Bull., vol. 67, pp. 343-354 (2005).
based on research findings. This is particularly true when the 10 FIG. 17 illustrates alternative embodiments of the
target brand is the subject’s favorite brand. See Kenning et example operational flow 500 of FIG. 5. FIG. 17 illustrates
al., “Neuroeconomics: an overview from an economic per example embodiments where the associating operation 530
spective.” Brain Res. Bull., vol. 67, pp. 343-354 (2005). may include at least one additional operation. Additional
Further, there is evidence for a large-scale neural system for operations may include operation 1700, 1702, 1704, 1706,
visuospatial attention that includes the right posterior pari 15 and/or 1708.
etal cortex. See Cabeza et al., “Imaging Cognition II: An Operation 1700 depicts associating activity in the dorso
Empirical Review of 275 PET and fMRI Studies,” J. Cog lateral prefrontal cortex with a mental state indicating objec
nitive Neurosci., vol. 12, pp. 1-47 (2000). tive approval. For example, the device 106, association unit
Operation 1604 depicts associating activity in at least one 140, emotion association module 242, attention association
of the inferior precuneus, posterior cingulate, right parietal module 244, and/or cognition association module 346 can
cortex, right Superior frontal gyms, right Supramarginal associate activity in the dorsolateral prefrontal cortex with a
gyms, or the Ventromedial prefrontal cortex with a mental mental state indicating objective approval. In one embodi
state indicating brand preference. For example, the device ment, association unit 240 can receive a brain activity
106, association unit 140, emotion association module 242, measurement indicating activation of the dorsolateral pre
attention association module 244, and/or cognition associa 25 frontal cortex proximate to presentation of a characteristic to
tion module 346 can associate activity in at least one of the a member of population cohort 102. The brain activity
inferior precuneus, posterior cingulate, right parietal cortex, measurement indicating activation of the dorsolateral pre
right Superior frontal gyms, right Supramarginal gyms, or the frontal cortex may be received from, for example, MEG
Ventromedial prefrontal cortex with a mental state indicating module 218. Emotion association module 242 and/or asso
brand preference. In one embodiment, association unit 240 30 ciation unit 340 can then match the dorsolateral prefrontal
can receive a brain activity measurement indicating activa cortex activation measurement to one or more corresponding
tion of the inferior precuneus and the ventromedial prefron mental states. For example, activation of the dorsolateral
tal cortex in response to presentation to a member of prefrontal cortex is associated in the literature with objective
population cohort 102 of an avatar associated with a brand recognition of benefit despite an emotional perception of
as the at least on characteristic. The brain activity measure 35 unfairness. Thus an association can be made between acti
ment indicating activation of the inferior precuneus and the vation of the DLPFC and objective approval a presented
ventromedial prefrontal cortex may be received from, for characteristic, for example by emotion association module
example, EEG module 320 and/or fMRI module 316. Emo 342, for example, based on research findings. See Kenning
tion association module 242, attention association module et al., “Neuroeconomics: an overview from an economic
244, and/or cognition association module 346 can then 40 perspective.” Brain Res. Bull., vol. 67, pp. 343-354 (2005).
match the inferior precuneus and ventromedial prefrontal Operation 1702 depicts associating activity in the caudate
cortex activation measurement to one or more corresponding nucleus with a mental state indicating trust. For example, the
mental states. For example, activation of the inferior precu device 106, association unit 140, emotion association mod
neus and the Ventromedial prefrontal cortex is associated in ule 242, attention association module 244, and/or cognition
the literature with brand preference. Thus an association can 45 association module 346 can associate activity in the caudate
be made between preference for the presented avatar asso nucleus with a mental state indicating trust. In one embodi
ciated with a brand and activation of the inferior precuneus ment, association unit 240 can receive a brain activity
and the ventromedial prefrontal cortex, by association unit measurement indicating activation of the caudate nucleus
240, for example, based on research findings. See Kenning proximate to presentation of a characteristic to a member of
et al., “Neuroeconomics: an overview from an economic 50 population cohort 102. The brain activity measurement
perspective.” Brain Res. Bull., vol. 67, pp. 343-354 (2005). indicating activation of the caudate nucleus may be received
Operation 1606 depicts associating activity in at least one from, for example, MEG module 218 and/or fNIR module
of the insula, the lateral orbital frontal cortex, or the 214. Emotion association module 242 and/or association
amygdala with a mental state indicating emotional disap unit 340 can then match the caudate nucleus activation
proval. For example, the device 106, association unit 140, 55 measurement to one or more corresponding mental states.
and/or emotion association module 242 can associate activ For example, activation of the caudate nucleus is associated
ity in at least one of the insula, the lateral orbital frontal in the literature with trust-building and reciprocity in eco
cortex, or the amygdala with a mental state indicating nomic exchange. Thus an association can be made between
emotional disapproval. In one embodiment, association unit activation of the caudate nucleus and a mental state indi
240 and/or emotion association module 342 can receive a 60 cating trust in the context of presentation of a characteristic,
brain activity measurement indicating activation of the for example by emotion association module 342, for
insula in response to presentation of a characteristic to a example, based on research findings. See Kenning et al.,
member of population cohort 102. The brain activity mea “Neuroeconomics: an overview from an economic perspec
Surement indicating activation of the insula may be received tive.” Brain Res. Bull., vol. 67, pp. 343-354 (2005).
from, for example, fNIR module 314 and/or EEG module 65 Operation 1704 depicts associating activity in the hip
320. Emotion association module 242 and/or association pocampus with a mental state indicating novelty in a per
unit 340 can then match the insula activation measurement ceived object. For example, the device 106, association unit
US 9,495,684 B2
69 70
140, emotion association module 242, attention association association module 246, and/or association unit 240 can then
module 244, and/or cognition association module 346 can match the brain activity Surrogate marker to one or more
associate activity in the hippocampus with a mental state corresponding mental states. For example, detection of voice
indicating novelty in a perceived object. In one embodiment, patterns indicative of a calm mental state may indicate
association unit 240 can receive a brain activity measure approval toward a characteristic, particularly in combination
ment indicating activation of the hippocampus proximate to with brain activation measurement of caudate nucleus acti
presentation of a characteristic to a member of population vation as a predictor of trust toward the characteristic. Thus
cohort 102. The brain activity measurement indicating acti an association can be made between a calm Voice pattern and
Vation of the hippocampus may be received from, for a mental state indicating approval toward a presented char
example, MEG module 218 and/or fMRI module 316. 10 acteristic, for example by emotion association module 342,
Emotion association module 242, cognition association for example, based on research findings. See Sanfey, “Social
module 346, and/or association unit 340 can then match the Decision-Making: Insights from Game Theory and Neuro
hippocampus activation measurement to one or more cor science.” Science, vol. 318, pp. 598-601 (26 Oct. 2007).
responding mental states. For example, activation of the FIG. 18 illustrates alternative embodiments of the
hippocampus is associated in the literature with a central 15 example operational flow 500 of FIG. 5. FIG. 18 illustrates
role in processing novel stimuli. Thus an association can be example embodiments where the specifying operation 540
made between activation of the hippocampus and a mental may include at least one additional operation. Additional
state indicating perceived novelty in the context of presen operations may include operation 1800, 1802, and/or 1804.
tation of a characteristic, for example by cognition associa Operation 1800 depicts specifying the at least one pre
tion module 346, for example, based on research findings. sented characteristic as the at least one avatar attribute based
See Martin et al., “Human experience seeking correlates on the at least one mental state. For example, device 306,
with hippocampus Volume: Convergent evidence from attribute specification unit 350, voice specification unit 352,
manual tracing and Voxel-based morphometry.” Neuropsy and/or non-verbal attribute specification unit 356 can specify
chologia, Vol. 45, pp. 2874-81 (2007). the at least one presented characteristic as the at least one
Operation 1706 depicts associating activity in the hip 25 avatar attribute based on the at least one mental state. In one
pocampus with a mental state indicating lack of inhibition embodiment, attribute specification unit 350 can specify a
toward a perceived object. For example, the device 106, characteristic for presentation to a member of population
association unit 140, emotion association module 242, atten cohort 102 Such as a photographic image of a girl's face with
tion association module 244, and/or cognition association a specific hairstyle. After measurement of a physiologic
module 346 can associate activity in the hippocampus with 30 activity proximate to the presentation, and association of the
a mental state indicating lack of inhibition toward a per physiologic activity with a mental state, an attribute speci
ceived object. In one embodiment, association unit 240 can fication unit 250, non-verbal attribute specification unit 256,
receive a brain activity measurement indicating activation of and/or body attribute specification unit 262 can specify the
the hippocampus proximate to presentation of a character specific hairstyle as the at least one avatar attribute for
istic to a member of population cohort 102. The brain 35 incorporation into an avatar design. For example, if viewing
activity measurement indicating activation of the hippocam the hairstyle elicited approval in the member of population
pus may be received from, for example, PET module 222 cohort 102, specification of an approximation of the hair
and/or fNIR module 214. Emotion association module 242, style in the design of an avatar may result in an avatar that
cognition association module 346, and/or association unit is attractive to the member of population cohort 102 and/or
340 can then match the hippocampus activation measure 40 population cohort 104 in general.
ment to one or more corresponding mental states. For Operation 1802 depicts specifying the at least one avatar
example, activation of the hippocampus is associated in the attribute based on a mental state based on a physiologic
literature with a lack of inhibition toward a perceived object. activity at or above a defined threshold. For example, device
Thus an association can be made between activation of the 306, attribute specification unit 350, voice specification unit
hippocampus and a mental state indicating a lack of inhibi 45 352, and/or non-verbal attribute specification unit 356 can
tion toward a perceived object in the context of presentation specify at least one avatar attribute based on a mental state
of a characteristic, for example by emotion association based on a physiologic activity at or above a defined
module 342, for example, based on research findings. See threshold. In one embodiment, attribute specification unit
Wang et al., “Gastric stimulation in obese subjects activates 350 can specify an avatar attribute that is based on a
the hippocampus and other regions involved in brain reward 50 presented characteristic that elicits, for example, activity in
circuitry.” PNAS, vol. 103, pp. 15641-45 (2006). the frontopolar cortex above a specified intensity level, as
Operation 1708 depicts associating at least one brain measured by, for example, fMRI or fNIR. Because low level
activity Surrogate marker with at least one mental state. For activations of the frontopolar cortex may occur frequently
example, the device 106, association unit 140, emotion having little or no significance in terms of indicating mental
association module 242, attention association module 244. 55 state (an association unit 240 may nevertheless make an
and/or cognition association module 346 can associate at association based on low level activation of a brain area or
least one brain activity Surrogate marker with at least one Surrogate marker), a filter may be employed to allow only
mental state. In one embodiment, association unit 240 can mental state associations founded on activations above a
receive a brain activity Surrogate marker Such as a skin certain threshold to pass to an attribute specification unit
response measurement, voice stress measurement, eye 60 250. Such a filter may be employed at the level of the
movement measurement, and/or iris response measurement physiologic measurement unit 210 Such that measurements
proximate to presentation of a characteristic to a member of that do not meet or exceed the defined threshold level or
population cohort 102. The brain activity surrogate marker intensity are not transmitted to the association unit 240.
may be received from, for example, iris response module Alternatively, a filter may be employed at the stage of the
232, gaze tracking module 234, skin response module 236. 65 association unit 240 such that only associations between
and/or voice response module 238. Emotion association mental state and physiologic measurements at or above the
module 242, attention association module 244, cognition defined threshold are transmitted to attribute specification
US 9,495,684 B2
71 72
unit 250. In addition or alternatively, a filter may be associate the physiologic measurement with at least one
employed at the level of the attribute specification unit 250 mental state. Attribute specification unit 250 may then
Such that mental states grounded on physiological measure specify an avatar attribute based on the mental state, for
ments below the defined threshold are not specified. example an approving mental state. In a successive iteration,
Operation 1804 depicts specifying the at least one avatar 5 the attribute specification unit 250 may send an avatar
attribute based on a mental state based on a physiologic including the characteristic previously associated with an
activity signature. For example, device 306, attribute speci approving mental state to presentation unit 270 for presen
fication unit 350, voice specification unit 352, and/or non tation, perhaps to a different member of population cohort
verbal attribute specification unit 356 can specify at least 104. Alternatively, an avatar may be sent including a variant
one avatar attribute based on a mental state based on a 10 of the characteristic previously associated with an approving
physiologic activity signature. In one embodiment, attribute mental state to presentation unit 270 for presentation to the
specification unit 350 can specify an avatar attribute that is same member of population cohort 102 or to a different
based on a presented characteristic that elicits, for example, member of population cohort 104. Alternatively, an avatar
a specific constellation of brain activity and/or Surrogate may be sent including a different characteristic to presenta
marker(s) of brain activity, as measured by, for example, 15 tion unit 270 for presentation to the same member of
fMRI module 216, fNIR module 214, and/or by surrogate population cohort 102 or to a different member of population
marker measurement unit 230. For example, presentation of cohort 104. This process may be repeated in rapid succes
a face characteristic to a member of population cohort 102 sion to gauge preferences of a member of population cohort
may coincide with detection of activation of the visual 102 and/or other members of population cohort 104 for
cortex (perhaps indicating visual contact with the charac attributes of an avatar and/or an avatar as whole.
teristic); activation of the right prefrontal and parietal areas FIG. 20 illustrates a partial view of an example computer
(perhaps indicating visual vigilance, i.e., attention); activa program product 2000 that includes a computer program
tion of the right hemisphere of the bilateral fusiform gyms 2004 for executing a computer process on a computing
(perhaps indicating perception of the face characteristic); device. An embodiment of the example computer program
activation of the medial temporal regions (perhaps indicat 25 product 2000 is provided using a signal bearing medium
ing retrieval of an item-specific memory); and activation of 2002, and may include one or more instructions for present
the ventral striatum and decreased activity in the insula ing at least one characteristic to at least one member of a
(perhaps indicating a positive emotional response Such as population cohort; one or more instructions for measuring at
approval); increased frequency of eye movements to the face least one physiologic activity of the at least one member of
characteristic; and iris dilation. An association unit 240 may 30 the population cohort, the at least one physiologic activity
associate some or all of the above physiological measure proximate to the at least one presented characteristic; one or
ments with one or more mental states, in effect creating a more instructions for associating the at least one physiologi
signature mental state for the reaction of member of popu cal activity with at least one mental state; and one or more
lation cohort 102 to the face characteristic; alternatively, instructions for specifying at least one avatar attribute based
association unit 240 may identify a known physiologic 35 on the at least one mental state. The one or more instructions
activity signature or pattern that corresponds to a known may be, for example, computer executable and/or logic
mental State, i.e., a known association. implemented instructions. In one implementation, the sig
FIG. 19 illustrates alternative embodiments of the nal-bearing medium 2002 may include a computer-readable
example operational flow 500 of FIG. 5. FIG. 19 illustrates medium 2006. In one implementation, the signal bearing
example embodiments where the flow 500 may include 40 medium 2002 may include a recordable medium 2008. In
additional operation 1950. one implementation, the signal bearing medium 2002 may
Operation 1950 depicts performing successive iterations include a communications medium 2010.
of the presenting at least one characteristic to at least one FIG. 21 illustrates an example system 2100 in which
member of a population cohort, the measuring at least one embodiments may be implemented. The system 2100
physiologic activity of the at least one member of the 45 includes a computing system environment. The system 2100
population cohort, the at least one physiologic activity also illustrates a member of population cohort 102 using a
proximate to the at least one presented characteristic, and the device 2104, which is optionally shown as being in com
associating the at least one physiological activity with at munication with a computing device 2102 by way of an
least one mental state. For example, device 106, presentation optional coupling 2106. The optional coupling 2106 may
device 364, physiologic activity measurement unit 310, 50 represent a local, wide-area, or peer-to-peer network, or may
association unit 340, and/or attribute specification unit 350 represent a bus that is internal to a computing device (e.g.,
may perform Successive iterations of the presenting at least in example embodiments in which the computing device
one characteristic to at least one member of a population 2102 is contained in whole or in part within the device
cohort, the measuring at least one physiologic activity of the 2104). A storage medium 2108 may be any computer storage
at least one member of the population cohort, the at least one 55 media. In one embodiment, the computing device 2102 may
physiologic activity proximate to the at least one character include a virtual machine operating within another comput
istic, and the associating the at least one physiological ing device. In an alternative embodiment, the computing
activity with at least one mental state. device 2102 may include a virtual machine operating within
In one embodiment, device 206 can present a character a program running on a remote server.
istic to member of population cohort 102, during which time 60 The computing device 2102 includes computer-execut
physiologic activity measurement unit 210, brain activity able instructions 2110 that when executed on the computing
measurement unit 212, and/or Surrogate marker measure device 2102 cause the computing device 2102 to (a) present
ment unit 230 can measure a physiologic activity that is at least one characteristic to at least one member of a
proximate to the presentation of the characteristic to the population cohort; (b) measure at least one physiologic
member of population cohort 102. Association unit 240, 65 activity of the at least one member of the population cohort,
emotion association module 242, attention association mod the at least one physiologic activity proximate to the at least
ule 244, and/or cognition association module 246 may then one presented characteristic; (c) associate the at least one
US 9,495,684 B2
73 74
physiological activity with at least one mental state; and (d) identification unit 166. Device 106 may interact with one or
specify at least one avatar attribute based on the at least one more members of population cohort 104 and/or one or more
mental state. As referenced above and as shown in FIG. 21, members of population 105, including for example, member
in some examples, the computing device 2102 may option of population cohort 102. Population cohort 104 may be a
ally be contained in whole or in part within the device 2104. part of a population 105. Device 2206 may receive a
In FIG. 21, then, the system 2100 includes at least one specified cohort-linked avatar attribute and/or cohort-linked
computing device (e.g., 2102 and/or 2104). The computer avatar from device 106 and/or attribute specification unit
executable instructions 2110 may be executed on one or 150. Device 2206, population cohort identification module
more of the at least one computing device. For example, the 2250, demographic identification module, and/or selection
computing device 2102 may implement the computer-ex 10 module 2254 may identify and/or select a member of
ecutable instructions 2110 and output a result to (and/or population cohort 102. Cohort-linked avatar presentation
receive data from) the computing device 2104. Since the module 2260 and/or indication module 2270 may present to
computing device 2102 may be wholly or partially contained the identified and/or selected member of population cohort
within the computing device 2104, the device 2104 also may 102 a specified cohort-linked avatar attribute and/or cohort
be said to execute some or all of the computer-executable 15 linked avatar.
instructions 2110, in order to be caused to perform or FIG. 24 illustrates an operational flow 2400 representing
implement, for example, various ones of the techniques example operations related to indicating behavior in a popu
described herein, or other techniques. lation cohort. In FIG. 24 and in following figures that
The device 2104 may include, for example, a portable include various examples of operational flows, discussion
computing device, workstation, or desktop computing and explanation may be provided with respect to the above
device. In another example embodiment, the computing described system environments of FIGS. 1-4, FIGS. 22-23,
device 2102 is operable to communicate with the device and/or with respect to other examples and contexts. How
2104 associated with the member of population cohort 102 ever, it should be understood that the operational flows may
to receive information about physiologic activity of the be executed in a number of other environment and contexts
member of population cohort 102 for performing data access 25 and/or in modified versions of FIGS. 1-3 and/or FIGS.
and data processing, and to specify at least one avatar 22-23. Also, although the various operational flows are
attribute based on at least one mental state, the mental state presented in the sequences illustrated, it should be under
based on the physiologic activity of the member of popu stood that the various operations may be performed in other
lation cohort 102. orders than those which are illustrated, or may be performed
Although a member of population cohort 102 is shown/ 30 concurrently.
described herein as a single illustrated figure, those skilled After a start operation, operation 2410 depicts identifying
in the art will appreciate that a member of population cohort a member of a population cohort. For example, a device
102 and/or member of population 105 may be composed of 2206, population cohort identification module 2250, demo
two or more entities. Those skilled in the art will appreciate graphic identification module 2252, and/or selection module
that, in general, the same may be said of 'sender and/or 35 2254 may identify a member of a population cohort 102. In
other entity-oriented terms as such terms are used herein. one embodiment, population cohort identification module
Indicating Behavior in a Population Cohort 2250 can identify a member of population cohort 102 based
FIG. 22 illustrates a system 2200 in which embodiments on, for example, video images of the population cohort
may be implemented. The system 2200 includes a device member 102. For example, demographic module 2252 can
2206. The device 2206 may contain, for example, a popu 40 analyze, for example, facial features to identify, for example,
lation cohort identification module 2250, which may in turn a female population cohort member 102 based on face shape
include a demographic identification module 2252 and/or an and/or hairstyle. In another example, demographic module
selection module 2254. The device 2206 may also include 2252 can analyze video footage of a shopper at a particular
device 106 (or variants of device 106 such as device 206 or store to identify, for example, an asian shopper based on eye
device 306). The device 2206 may interact with at least one 45 shape and/or skin color. In one embodiment, a member of
member of population cohort 102 and/or member of popu population cohort 102 may be identified by selection module
lation 105. As discussed above, device 106 can present a 2254, accessed by a user of system 2200. For example, a
characteristic to member of population 105 and/or member user of system 2200 may choose from a menu of available
of population cohort 102. Further, device 106 can measure population cohorts stored in selection module 2254 a par
at least one physiologic activity of member of population 50 ticular population cohort as a way of choosing a member of
105 and/or member of population cohort 102, as discussed population cohort 102. As depicted in FIG. 23, device 2206
above. Association of the measured physiologic activity and/or population cohort identification module 2250 may be
with a mental state by association unit 140 can result in in communication with device 106 to access a specified
attribute specification unit 150 specifying a cohort-linked cohort-linked avatar attribute that corresponds to a selected
avatar and/or cohort-linked avatar attribute. Additional func 55 population cohort. Alternatively, device 2206 may include
tions of device 106 and its variants device 206 and device the cohort-linked avatar attribute specification functionality
306 are discussed above in detail, for example in the context of system 100 and/or device 106.
of FIG. 2 and FIG. 3. FIG.22 also illustrates the operational Operation 2420 depicts indicating at least one behavior in
flow of FIG. 24, discussed below. Device 2206 may also the member of the population cohort based on an association
include a cohort-linked avatar presentation module 2260 60 between the population cohort and at least one cohort-linked
and/or indication module 2270. avatar. For example, a device 2206, cohort-linked avatar
FIG. 23 illustrates an alternate of system 2200 in which presentation module 2260, indication module 2270, and/or
embodiments may be implemented. As described above in device 106 may indicate at least one behavior in the member
the context of FIG. 1 and system 100, device 106 may of the population cohort based on an association between the
contain, for example, a presentation unit 170, a physiologic 65 population cohort and at least one cohort-linked avatar. For
activity measurement unit 110, an association unit 140, an example, device 2206, indication module 2270, and/or
attribute specification unit 150, and/or a population cohort cohort-linked avatar presentation module 2260 can access a
US 9,495,684 B2
75 76
specified cohort-linked avatar appropriate for the identified A number of methods of identifying ethnicity based on
population cohort, perhaps from device 106. Device 2206, facial features are known in the art, for example, ethnicity
indication module 2270, and/or cohort-linked avatar presen identification may be formulated as a two-category classi
tation module 2260 can then indicate a future behavior such fication problem, for example, to classify the Subject as an
as an inquiry, an internet search, a shopping behavior, and/or Asian or non-Asian. The input images may be resized to
a purchasing behavior that is likely to occur upon presen different scales. At each scale, a classic appearance-based
tation of the specified cohort-linked avatar to a member of face recognizer based on a linear discriminant analysis
population cohort 102. representation may be developed under a Bayesian statistical
FIG. 25 illustrates alternative embodiments of the
example operational flow 2400 of FIG. 24. FIG. 25 illus 10
decision framework. An ensemble may then be constructed
trates example embodiments where the identifying operation by integrating classification results to arrive at a final
2410 may include at least one additional operation. Addi decision. The product rule may be used as an integration
tional operations may include operation 2500, 2502, and/or strategy. See Lu et al., “Ethnicity Identification from Face
operation 2504. Images. Biometric Technology for Human Identification,
Operation 2500 depicts identifying a member of a demo 15 Eds. Jain et al., Proc. SPIE, Vol. 5404, pp. 114-123 (2004).
graphic population cohort. For example, a device 2206, User ethnicity identification may be based on a number of
population cohort identification module 2250, demographic factors including skin and/or hair characteristics associated
identification module 2252, and/or selection module 2254 with ethnicity, such as red hair among caucasians; voice
can identify a member of a demographic population cohort. and/or speech associated with ethnicity, Such as French
In one embodiment, a demographic identification module accented English indicating French or French-Canadian
2252 can identify an age range in a member of population ethnicity; face pattern associated with ethnicity, such as eye
105 such as “over age 52 as the demographic population shape, nose shape, face shape, or the like; and eye attributes
based on an analysis of cloudiness of the eye associated with Such as blue eyes among caucasians. In one embodiment,
cataracts. In the United States, age-related lenticular changes Gabor wavelets transformation and retina sampling from
have been reported in 42% of those between the ages of 52 25 user-health test function outputs may be combined to extract
to 64, 60% of those between the ages 65 and 74, and 91% key facial features, and Support vector machines may be
of those between the ages of 75 and 85. Accordingly, may used for ethnicity classification. An experimental system has
include a camera that can analyze an individual’s eye for used Gabor wavelets transformation and retina sampling in
cloudiness to identify a member of population cohort 102. combination to extract key facial features, and Support
In another embodiment, demographic module 2252 may 30 vector machines were used for ethnicity classification,
identify a population member's ethnicity based on detection resulting in approximately 94% success for ethnicity esti
of a foreign language or accented speech, such as a Southern mation under various lighting conditions. See Hosoi et al.,
accent, a Boston accent, a Spanish accent, a British accent, “Ethnicity estimation with facial images. Sixth IEEE Inter
or the like. See, for example, U.S. Pat. No. 7,263.489 national Conference on Automatic Face and Gesture Rec
"Detection of characteristics of human-machine interactions 35 ognition, pp. 195-200 (2004). Of course other methods of
for dialog customization and analysis.” ethnicity identification known in the art may be used.
In another embodiment, demographic module 2252 may In another embodiment, an age, gender, and/or ethnicity
identify a population member as part of a hearing-impaired characteristic may be based on, for example, an iris pattern
demographic based on body language such as use of Ameri associated with an asian user. For example, a bank of
can Sign Language, for example captured by Video footage 40 multichannel 2D Gabor filters may be used by demographic
of the population member. module 2252 to capture global texture information about an
Operation 2502 depicts identifying a member of at least a user's iris, and AdaBoost, a machine learning algorithm,
one of an age, gender, or ethnicity population cohort. For may be used to allow a demographic module 2252 to learn
example, a device 2206, population cohort identification a discriminant classification principle from a pool of candi
module 2250, demographic identification module 2252, and/ 45 date iris feature sets. Iris image data may be thus grouped
or selection module 2254 can identify a member of at least into race categories, for example, Asian and non-Asian. See
one of an age, gender, or ethnicity population cohort. In one Quiet al., “Global Texture Analysis of Iris Images for Ethnic
embodiment, a demographic identification module 2252 can Classification.” Lecture notes in computer Science, Springer:
identify a member of an age, gender, or ethnicity population Berlin/Heidelberg, Advances in Biometrics, pp. 411-418
cohort based on visible an individuals objects such as 50 (2005).
clothing, hairstyle, and/or adornment such as jewelry, which The age, gender, and/or ethnicity characteristic also may
may be particularly informative as an indicators of gender, be based on, for example, a face pattern analysis indicating
age, and/or ethnicity. For example, a dress may help identify facial hair (e.g., a beard or moustache) signifying to the
a population members gender as female, and similarly a Suit demographic module 2252 that the user is male. The age,
with a necktie may help identify a population members 55 gender, and/or ethnicity characteristic also may be based on,
gender as male. for example, speech or Voice data Such as Mandarin Chi
Further, images of the population members face may be nese-accented Chinese or English speech signifying to the
analyzed by demographic module 2252 to identify a gender demographic module 2252 that the population member is of
by face shape. See Jain et al., “Gender identification using Chinese ethnicity. For example, three demographic charac
frontal facial images.” IEEE Multimedia and Expo Interna 60 teristics may be transmitted in the form of “elderly Chinese
tional Conference, 4 pp. (6-8 Jul. 2005); describing gender male.”
classification using frontal facial images in which 96% Alternatively, iris pattern data, face pattern data, and Voice
accuracy was reached using a Support Vector Machine or speech data may each independently relate to one demo
(SVM) in independent component analysis (ICA) space. Of graphic characteristic, such as gender. For example, iris
course other methods of gender identification known in the 65 pattern data may indicate a male user, face pattern data
art may be used. Voice analysis may also be used to identify detecting facial hair may indicate a male user, and Voice
a particular gender. pitch data may indicate a male user, resulting in a male
US 9,495,684 B2
77 78
gender demographic population cohort identification with a association between the population cohort and the at least
relatively high level of confidence. one cohort-linked avatar. For example, device 2206, cohort
Operation 2504 depicts selecting a member of a popula linked avatar presentation module 2260, and/or indication
tion cohort. For example, a device 2206, population cohort module 2270 can forecast at least one behavior of at least
identification module 2250, demographic identification one member of the population cohort based on an associa
module 2252, and/or selection module 2254 can select a tion between the population cohort and the at least one
member of a population cohort. For example, selection cohort-linked avatar. For example, indication module 2270
module 2254 may select group of local shoppers as a can forecast a click-through behavior in a member of
population cohort. In one embodiment, selection module population cohort 102 upon presentation of a cohort-linked
2254 may access from device 106, for example, a product 10 avatar in an advertisement to the member of population
avatar such as a BMW automobile that is associated with cohort 102, for example, during an interaction with a social
preference among, for example, Microsoft employees work networking site. In some embodiments, given a context that
ing in Redmond, Wash. Thus the BMW automobile may be provides access to items that are easily selected for further
a specified cohort-linked avatar that is known to be associ action, such as items on webpages, indication module 2270
ated with, for example, a mental state indicating preference 15 may provide a forecast of a clicking, typing, searching,
among a group of Microsoft employees working in Red shopping and/or purchasing behavior contingent upon pre
mond, Wash. Accordingly, selection module 2254 may senting, for example, an the image of a cohort-linked avatar
select the Microsoft employees working in Redmond, Wash. and/or cohort-linked avatar attribute to the member of
as a population cohort. Alternatively, a user of system 2200 population cohort 102, particularly where the mental state
may select an established demographic group as the popu associated with the avatar or attribute is a positive one Such
lation cohort and/or an ad hoc population cohort. as product preference.
FIG. 26 illustrates alternative embodiments of the Operation 2604 depicts predicting at least one behavior of
example operational flow 2400 of FIG. 24. FIG. 26 illus at least one member of the population cohort based on an
trates example embodiments where the indicating operation association between the population cohort and the at least
2420 may include at least one additional operation. Addi 25 one cohort-linked avatar. For example, device 2206, cohort
tional operations may include operation 2600, 2602, and/or linked avatar presentation module 2260, and/or indication
operation 2604. module 2270 can predict at least one behavior of at least one
Operation 2600 depicts indicating a likely behavior of at member of the population cohort based on an association
least one member of the population cohort based on an between the population cohort and the at least one cohort
association between the population cohort and the at least 30 linked avatar. For example, indication module 2270 can
one cohort-linked avatar. For example, device 2206, cohort predict a purchasing behavior as a result of cohort-linked
linked avatar presentation module 2260, and/or indication avatar presentation module 2260 presenting an image of a
module 2270 can indicate a likely behavior of at least one cohort-linked avatar to a member of population cohort 102,
member of the population cohort based on an association for example, during a shopping experience (e.g., in a shop
between the population cohort and the at least one cohort 35 window, on a facebook page, and/or on a personal electronic
linked avatar. For example, indication module 2270 can device of the member of population cohort 102). For
coordinate with a cohort-linked avatar presentation module example, a cohort-linked avatar music attribute may be
2260 to evaluate a context of presentation of a cohort-linked specified which is associated with approval in a population
avatar to a member of population cohort 102, for example, demographic that is known to buy music at a particular
during a shopping experience (e.g., on a monitor at a 40 internet outlet. Indication module 2270 may predict that
shopping mall, within a webpage, or on a personal electronic selective presentation of the cohort-linked avatar music
device of the member of population cohort 102). In some attribute to members of population cohort 104 will result in
embodiments, given a context that provides access to mar members of the population cohort 104 listening to the
keted items, indication module 2270 may provide an indi cohort-linked music attribute and/or other music associated
cation of a likelihood of a shopping and/or purchase behav 45 with the cohort-linked music attribute during a given visit to
ior contingent upon presenting the image of a cohort-linked the particular internet outlet.
avatar and/or cohort-linked avatar attribute to the member of FIG. 27 illustrates alternative embodiments of the
population cohort 102, particularly where the mental state example operational flow 2400 of FIG. 24. FIG. 27 illus
associated with the avatar or attribute is a positive one Such trates example embodiments where the indicating operation
as preference. 50 2420 may include at least one additional operation. Addi
For example, indication module 2270 may provide a tional operations may include operation 2700, 2702, 2704,
likelihood of an inquiry behavior as an indication of a and/or operation 2706.
behavior upon presentation of a cohort-linked avatar Such as Operation 2700 depicts indicating at least one purchasing
a particular furniture design to a member of a population behavior in the member of the population cohort based on an
cohort 102, the cohort-linked avatar furniture design asso 55 association between the population cohort and the at least
ciated with approval in members of the population cohort, one cohort-linked avatar. For example, device 2206, cohort
perhaps consisting of homeowners living in a specific Zip linked avatar presentation module 2260, and/or indication
code. Other behaviors associated with such feelings of module 2270 can indicate at least one purchasing behavior
approval in the context of the cohort-linked avatar furniture in the member of the population cohort based on an asso
design may also be indicated. Such as increased eye move 60 ciation between the population cohort and the at least one
ments toward the cohort-linked avatar furniture design, cohort-linked avatar. In one embodiment, a cohort-linked
increased attention toward the cohort-linked avatar furniture avatar presentation module 2260 can present to, for
design, and/or active steps to investigate and/or purchase the example, a group of registered Xbox LIVE users a cohort
cohort-linked avatar furniture design and/or items associated linked avatar in the form of an image of Master Chief from
with the cohort-linked avatar furniture design. 65 the Halo franchise of computer games, which has been
Operation 2602 depicts forecasting at least one behavior associated with an approving mental state among one or
of at least one member of the population cohort based on an more registered Xbox LIVE users. In one embodiment, the
US 9,495,684 B2
79 80
image of Master Chief may be paired with one or more shopper may benefit from the positive views toward the
marketed items, such that the approving mental state asso cohort-linked avatar virtual personal shopper among mem
ciated with the cohort-linked avatar Master Chief image may bers of population cohort 104 and member of population
be transferred to the paired item. In this way, indication cohort 102 specifically. Accordingly, indication module
module 2270 can indicate a likelihood that a member of 5 2270 may indicate future behavior of population cohort
population cohort 102 may purchase the paired item by member 102 Such as browsing a website, looking at a
virtue of its association with the cohort-linked avatar. product, placing an item on a wish list, communicating with
Operation 2702 depicts indicating at least one internet a friend about a product, and/or purchasing an item online by
usage behavior in the member of the population cohort virtue of the association with the cohort-linked avatar per
based on an association between the population cohort and 10 Sonal virtual shopper.
the at least one cohort-linked avatar. For example, device FIG. 28 illustrates alternative embodiments of the
2206, cohort-linked avatar presentation module 2260, and/or example operational flow 2400 of FIG. 24. FIG. 28 illus
indication module 2270 can indicate at least one internet trates example embodiments where the indicating operation
usage behavior in the member of the population cohort 2420 may include at least one additional operation. Addi
based on an association between the population cohort and 15 tional operations may include operation 2800, 2802, and/or
the at least one cohort-linked avatar. In one embodiment, a operation 2804.
cohort-linked avatar presentation module 2260 can present Operation 2800 depicts indicating at least one gameplay
to, for example, a member of population cohort 102 a behavior in the member of the population cohort based on an
cohort-linked avatar in the form of an audio clip that has association between the population cohort and the at least
been associated with a feeling of reward among one or more one cohort-linked avatar. For example, device 2206, cohort
members of population cohort 104. In one embodiment, an linked avatar presentation module 2260, and/or indication
audio clip may be presented (e.g., played) during the pre module 2270 can indicate at least one gameplay behavior in
sentation of a photo and/or video image of a marketed item, the member of the population cohort based on an association
such that the feeling of reward associated with the cohort between the population cohort and the at least one cohort
linked avatar audio clip may be transferred to the paired item 25 linked avatar. In one embodiment, in an online gaming
image. In this way, indication module 2270 can indicate that universe, a specific robot design may be a cohort-linked
a member of population cohort 102 may research, shop for, avatar that is associated with approval and/or trust in a
and/or purchase the paired item by virtue of its association population cohort 104 consisting of members of a faction
with the cohort-linked avatar. For example, cohort-linked within the gaming universe. Accordingly, indication module
avatar presentation module 2260 can play an audio clip Such 30 2270 may forecast a concordant action and/or a friendly
as a Bart Simpson tag line Such as "Eat my shorts during communication from a member of the faction Subsequent to
the presentation of an image on a website such as a hyper presentation of the cohort-linked avatar robot design to the
linked image of a marketed toy to a member of population member of the faction, because of feelings of trust and/or
cohort 102. Accordingly, indication module 2270 can predict approval associated with the presented cohort-linked avatar
that the member of population cohort 102 will associate a 35 robot design.
feeling of reward with the audio clip and the image of a toy, Operation 2802 depicts indicating at least one social
and will click on the hyperlinked image of the toy to learn interaction behavior in the member of the population cohort
more about it and/or to purchase it. based on an association between the population cohort and
Operation 2704 depicts indicating at least one shopping the at least one cohort-linked avatar. For example, device
behavior in the member of the population cohort based on an 40 2206, cohort-linked avatar presentation module 2260, and/or
association between the population cohort and the at least indication module 2270 can indicate at least one social
one cohort-linked avatar. For example, device 2206, cohort interaction behavior in the member of the population cohort
linked avatar presentation module 2260, and/or indication based on an association between the population cohort and
module 2270 can indicate at least one shopping behavior in the at least one cohort-linked avatar. In one embodiment, a
the member of the population cohort based on an association 45 Social networking website Such as facebook and/or myspace
between the population cohort and the at least one cohort may attach its brand to a cohort-linked avatar sports figure
linked avatar. In one embodiment, a cohort-linked avatar that is associated with approval and/or preference in a
presentation module 2260 can present to, for example, a population cohort 104 consisting of young girls who are fans
member of population cohort 102 a cohort-linked avatar in of a tennis Star, Such as Selena Williams. Accordingly,
the form of a computer-generated figure that incorporates 50 indication module 2270 may predict that other members of
several cohort-linked attributes that have been associated the population cohort 104 will, for example, register with
with a feeling of attention, trust, approval, and/or preference facebook and/or myspace, create a facebook and/or myspace
among one or more members of population cohort 104. In page, and/or sign up as a friend of Selena Williams on Selena
one embodiment, a virtual personal shopper in the form of Williams’ facebook and/or myspace page Subsequent to
a teenage girl on a website may be presented in which 55 presentation of a cohort-linked avatar image of Selena
various cohort-linked attributes of Hollywood celebrities Williams and/or a cohort-linked avatar attribute of Selena
such as Paris Hilton, Britney Spears, and Angelina Jolie are Williams together with a facebook and/or myspace brand
combined in the form of the teenage girl virtual personal identification, because of feelings of approval and/or pref
shopper as a cohort-linked avatar that appeals to teenage erence associated with, for example, the presented cohort
girls aged 14-19. In this example, the virtual personal 60 linked avatar image of Selena Williams.
shopper bearing, for example, a likeness of Paris Hilton's Operation 2804 depicts indicating at least one behavior in
face and hair, a t-shirt with a line from a Britney Spears Song at least one of an online shopper, an online gamer, a virtual
like "Oops I did it again,” and a voice like that of Angelina world participant, or a social networking site participant
Jolie may be presented to a member of population cohort based on an association between the population cohort and
102 Such that a feeling of attention, trust, approval, and/or 65 the at least one cohort-linked avatar. For example, device
preference is evoked. In this way, online shopping activities 2206, cohort-linked avatar presentation module 2260, and/or
conducted with the cohort-linked avatar virtual personal indication module 2270 can indicate at least one behavior in
US 9,495,684 B2
81 82
at least one of an online shopper, an online gamer, a virtual member of the population cohort based on a statistical
world participant, or a social networking site participant association between the population cohort and the at least
based on an association between the population cohort and one cohort-linked avatar. In one embodiment, cohort-linked
the at least one cohort-linked avatar. In one embodiment, a avatar presentation module 2260 may select for presentation
cohort-linked avatar musical group that is associated with 5 to members of population cohort 104 one of a set of
approval and/or preference in a population cohort 104 cohort-linked avatars, each of which are associated with
consisting of Austin, TeX. residents aged 20-40 may be approval in a population cohort 104 consisting of male
presented to a member of the population cohort 104 at an NASCAR race attendees aged 20-50, for example. Such a
event in a virtual world that involves marketing of a local set of cohort-linked avatars, for example, a group of beef
Austin, Tex. business. Accordingly, indication module 2270 10 jerky brands, may be evaluated by device 2206 for the most
may indicate a likelihood that members of the population statistically valid one of the group, to be presented by
cohort 104 in the virtual world will, for example, take steps cohort-linked avatar presentation module 2260. For
to learn more about the local Austin, Tex... business subse example, device 2206, cohort-linked avatar presentation
quent to presentation of the cohort-linked avatar musical module 2260, and/or indication module 2270 may rank the
group together with the local Austin, TeX. business, because 15 cohort-linked avatar beef jerky brands by the number of
offeelings of approval and/or preference associated with the population cohort members measured to establish the asso
presented cohort-linked avatar musical group. ciation between measured physiologic activity and approval;
FIG. 29 illustrates alternative embodiments of the by degree of physiologic activity associated with presenta
example operational flow 2400 of FIG. 24. FIG. 29 illus tion of the beef jerky brand in the measured population
trates example embodiments where the indicating operation cohort members; by geographical location of the measure
2420 may include at least one additional operation. Addi ments, or the like. Accordingly, indication module 2270 may
tional operations may include operation 2900, 2902, and/or identify one cohort-linked avatar beef jerky brand, such as
operation 2904. Jeff Foxworthy's beef jerky, as the one that is most likely to
Operation 2900 depicts indicating at least one behavior in be purchased if cohort-linked avatar presentation module
a member of a demographic population based on an asso 25 2260 presents a cohort-linked avatar Jeff Foxworthy's beef
ciation between the demographic population and at least one jerky to males aged 20-50 at a particular NASCAR event,
avatar linked to the demographic population. For example, based on higher approval associated with the cohort-linked
device 2206, cohort-linked avatar presentation module avatar Jeff Foxworthy's beef jerky compared to that of other
2260, and/or indication module 2270 can indicate at least cohort-linked avatar beef jerky brands.
one behavior in a member of a demographic population 30 Operation 2904 depicts indicating at least one behavior in
based on an association between the demographic popula the member of the population cohort based on an association
tion and at least one avatar linked to the demographic between the population cohort and at least one avatar linked
population. In one embodiment, a cohort-linked avatar audio to the member of the population cohort. For example, device
clip of George Clooney's Voice associated with approval 2206, cohort-linked avatar presentation module 2260, and/or
and/or preference in a population cohort 104 consisting of 35 indication module 2270 can indicate at least one behavior in
women aged 20-60 as a demographic population may be the member of the population cohort based on an association
presented to a member of the population cohort 104 as part between the population cohort and at least one avatar linked
of an advertisement for a men's cologne, for example. to the member of the population cohort. In one embodiment,
Accordingly, indication module 2270 may predict that a cohort-linked avatar presentation module 2260 may present
woman aged 20-60 to whom the cohort-linked avatar audio 40 to a member of population cohort 102 a cohort-linked avatar
clip of George Clooney's voice with the advertisement is that was specified based on one or more physiologic mea
presented may, for example, shop for the cologne as a result surements of the member of population cohort 102. For
offeelings of approval and/or preference associated with the example, an individual’s behavior may be predicted based
presented cohort-linked avatar audio clip of George Cloo on a reaction to a custom-specified cohort-linked avatar that
ney's voice. 45 is specific to the individual as a member of population cohort
Identification of a demographic population may be based 102.
on a fact-based attribute of a population cohort. For For example, device 2206 and/or device 106 may present
example, a population cohort Such as individuals wearing a an instance of media content such as an image of a personal
dress at the time of product purchase may result in classi electronic device like an iphone or a Zune media player to
fication of that population cohort as belonging to a female 50 an individual as a member of population cohort 102. Device
gender demographic. Such a classification, while not per 2206 and/or device 106 may then measure at least one
fect, may provide a likelihood of gender that may be physiologic activity at a time proximate to presentation of
confirmed by other means, such as detection of face shape the Zune, for example. Following association of the mea
by an imaging device, detection of voice pitch and/or speech Sured physiologic activity, Such as brain activity measured
pattern, name association with gender (e.g., Mary and Jane 55 by fNIR module 314, with a mental state such as preference,
as female names), or the like. Of course, information about attribute specification unit 350 may specify an attribute of
the demographics of a population cohort member may be the Zune and/or the image of the Zune, for example, as a
provided directly by the member of population cohort 102. cohort-linked avatar associated with preference in the mem
perhaps as part of a permissions process, or perhaps indi ber of population cohort 102. Accordingly, presentation of
rectly by a witness to the presentation of a cohort-linked 60 the cohort-linked avatar Zune image to the same member of
avatar attribute to a population cohort member. population cohort 102 in other contexts may provide a basis
Operation 2902 depicts indicating at least one behavior in for indication module 2270 to predicts a behavior in the
the member of the population cohort based on a statistical member of population cohort 102. For example, cohort
association between the population cohort and the at least linked avatar presentation module 2260 may present to the
one cohort-linked avatar. For example, device 2206, cohort 65 member of population cohort 102 the cohort-linked avatar
linked avatar presentation module 2260, and/or indication Zune image together with a particular audio clip of a new
module 2270 can indicate at least one behavior in the band, such that indication module 2270 can predict that the
US 9,495,684 B2
83 84
member of population cohort 102 will listen to the audio clip 3104 associated with the entity 3105 to receive information
or a sample of the audio clip, and/or purchase a song and/or Such as population cohort data and/or specified cohort
collection of Songs by the new band. linked avatar attribute data from a device 3105 for perform
FIG. 30 illustrates a partial view of an example computer ing data access and data processing.
program product 3000 that includes a computer program Although an entity 3105 is shown/described herein as a
3004 for executing a computer process on a computing single illustrated figure, those skilled in the art will appre
device. An embodiment of the example computer program ciate that an entity 3105 may be composed of two or more
product 3000 is provided using a signal bearing medium entities. Those skilled in the art will appreciate that, in
3002, and may include one or more instructions for identi general, the same may be said of “sender and/or other
fying a member of a population cohort; and one or more 10 entity-oriented terms as Such terms are used herein.
instructions for indicating at least one behavior in the One skilled in the art will recognize that the herein
member of the population cohort based on an association described components (e.g., steps), devices, and objects and
between the population cohort and at least one cohort-linked the discussion accompanying them are used as examples for
avatar. The one or more instructions may be, for example, the sake of conceptual clarity and that various configuration
computer executable and/or logic-implemented instructions. 15 modifications are within the skill of those in the art. Con
In one implementation, the signal-bearing medium 3002 sequently, as used herein, the specific exemplars set forth
may include a computer-readable medium 3006. In one and the accompanying discussion are intended to be repre
implementation, the signal bearing medium 3002 may sentative of their more general classes. In general, use of any
include a recordable medium 3008. In one implementation, specific exemplar herein is also intended to be representative
the signal bearing medium 3002 may include a communi of its class, and the non-inclusion of Such specific compo
cations medium 3010. nents (e.g., steps), devices, and objects herein should not be
FIG. 31 illustrates an example system 3100 in which taken as indicating that limitation is desired.
embodiments may be implemented. The system 3100 Those skilled in the art will appreciate that the foregoing
includes a computing system environment. The system 3100 specific exemplary processes and/or devices and/or tech
also illustrates an entity 3105 using a device 3104, which is 25 nologies are representative of more general processes and/or
optionally shown as being in communication with a com devices and/or technologies taught elsewhere herein, such as
puting device 3102 by way of an optional coupling 3106. in the claims filed herewith and/or elsewhere in the present
The optional coupling 3106 may represent a local, wide application.
area, or peer-to-peer network, or may represent a bus that is Those having skill in the art will recognize that the state
internal to a computing device (e.g., in example embodi 30 of the art has progressed to the point where there is little
ments in which the computing device 3102 is contained in distinction left between hardware and software implemen
whole or in part within the device 3104). A storage medium tations of aspects of systems; the use of hardware or soft
3.108 may be any computer storage media. In one embodi ware is generally (but not always, in that in certain contexts
ment, the computing device 3102 may include a virtual the choice between hardware and software can become
machine operating within another computing device. In an 35 significant) a design choice representing cost vs. efficiency
alternative embodiment, the computing device 3102 may tradeoffs. Those having skill in the art will appreciate that
include a virtual machine operating within a program run there are various vehicles by which processes and/or sys
ning on a remote server. tems and/or other technologies described herein can be
The computing device 3102 includes computer-execut effected (e.g., hardware, Software, and/or firmware), and that
able instructions 3110 that when executed on the computing 40 the preferred vehicle will vary with the context in which the
device 3102 cause the computing device 3102 to (a) identify processes and/or systems and/or other technologies are
a member of a population cohort; and (b) indicate at least deployed. For example, if an implementer determines that
one behavior in the member of the population cohort based speed and accuracy are paramount, the implementer may opt
on an association between the population cohort and at least for a mainly hardware and/or firmware vehicle: alterna
one cohort-linked avatar. As referenced above and as shown 45 tively, if flexibility is paramount, the implementer may opt
in FIG. 31, in some examples, the computing device 3102 for a mainly Software implementation; or, yet again alter
may optionally be contained in whole or in part within the natively, the implementer may opt for some combination of
device 3104. Computing device 3102 may optionally hardware, software, and/or firmware. Hence, there are sev
include any or all of the functions of device 2602 and/or eral possible vehicles by which the processes and/or devices
device 106. 50 and/or other technologies described herein may be effected,
In FIG. 31, then, the system 3100 includes at least one none of which is inherently superior to the other in that any
computing device (e.g., 3102 and/or 3104). The computer vehicle to be utilized is a choice dependent upon the context
executable instructions 3110 may be executed on one or in which the vehicle will be deployed and the specific
more of the at least one computing device. For example, the concerns (e.g., speed, flexibility, or predictability) of the
computing device 3102 may implement the computer-ex 55 implementer, any of which may vary. Those skilled in the art
ecutable instructions 3110 and output a result to (and/or will recognize that optical aspects of implementations will
receive data from) the computing device 3104. Since the typically employ optically-oriented hardware, Software, and
computing device 3102 may be wholly or partially contained or firmware.
within the computing device 3104, the device 3104 also may The foregoing detailed description has set forth various
be said to execute some or all of the computer-executable 60 embodiments of the devices and/or processes via the use of
instructions 3110, in order to be caused to perform or block diagrams, flowcharts, and/or examples. Insofar as
implement, for example, various ones of the techniques Such block diagrams, flowcharts, and/or examples contain
described herein, or other techniques. one or more functions and/or operations, it will be under
The device 3104 may include, for example, a portable stood by those within the art that each function and/or
computing device, workstation, or desktop computing 65 operation within Such block diagrams, flowcharts, or
device. In another example embodiment, the computing examples can be implemented, individually and/or collec
device 3102 is operable to communicate with the device tively, by a wide range of hardware, software, firmware, or
US 9,495,684 B2
85 86
virtually any combination thereof. In one embodiment, signal processors, computational entities such as operating
several portions of the subject matter described herein may systems, drivers, graphical user interfaces, and applications
be implemented via Application Specific Integrated Circuits programs, one or more interaction devices, such as a touch
(ASICs), Field Programmable Gate Arrays (FPGAs), digital pad or screen, and/or control systems including feedback
signal processors (DSPs), or other integrated formats. How loops and control motors (e.g., feedback for sensing position
ever, those skilled in the art will recognize that some aspects and/or Velocity; control motors for moving and/or adjusting
of the embodiments disclosed herein, in whole or in part, can components and/or quantities). A typical data processing
be equivalently implemented in integrated circuits, as one or system may be implemented utilizing any suitable commer
more computer programs running on one or more computers cially available components, such as those typically found in
(e.g., as one or more programs running on one or more 10 data computing/communication and/or network computing/
computer systems), as one or more programs running on one communication systems.
or more processors (e.g., as one or more programs running All of the above U.S. patents, U.S. patent application
on one or more microprocessors), as firmware, or as virtu publications, U.S. patent applications, foreign patents, for
ally any combination thereof, and that designing the cir eign patent applications and non-patent publications referred
cuitry and/or writing the code for the software and or 15 to in this specification and/or listed in any Application Data
firmware would be well within the skill of one of skill in the Sheet are incorporated herein by reference, to the extent not
art in light of this disclosure. In addition, those skilled in the inconsistent herewith.
art will appreciate that the mechanisms of the subject matter The herein described subject matter sometimes illustrates
described herein are capable of being distributed as a different components contained within, or connected with,
program product in a variety of forms, and that an illustra different other components. It is to be understood that such
tive embodiment of the subject matter described herein depicted architectures are merely exemplary, and that in fact
applies regardless of the particular type of signal bearing many other architectures can be implemented which achieve
medium used to actually carry out the distribution. the same functionality. In a conceptual sense, any arrange
Examples of a signal bearing medium include, but are not ment of components to achieve the same functionality is
limited to, the following: a recordable type medium such as 25 effectively “associated such that the desired functionality is
a floppy disk, a hard disk drive, a Compact Disc (CD), a achieved. Hence, any two components herein combined to
Digital Video Disk (DVD), a digital tape, a computer achieve a particular functionality can be seen as “associated
memory, etc.; and a transmission type medium Such as a with each other such that the desired functionality is
digital and/or an analog communication medium (e.g., a achieved, irrespective of architectures or intermedial com
fiber optic cable, a waveguide, a wired communications link, 30 ponents. Likewise, any two components so associated can
a wireless communication link, etc.). also be viewed as being “operably connected, or “operably
In a general sense, those skilled in the art will recognize coupled, to each other to achieve the desired functionality,
that the various aspects described herein which can be and any two components capable of being so associated can
implemented, individually and/or collectively, by a wide also be viewed as being “operably couplable.” to each other
range of hardware, software, firmware, or any combination 35 to achieve the desired functionality. Specific examples of
thereof can be viewed as being composed of various types operably couplable include but are not limited to physically
of “electrical circuitry.” Consequently, as used herein “elec mateable and/or physically interacting components and/or
trical circuitry’ includes, but is not limited to, electrical wirelessly interactable and/or wirelessly interacting compo
circuitry having at least one discrete electrical circuit, elec nents and/or logically interacting and/or logically inter
trical circuitry having at least one integrated circuit, elec 40 actable components.
trical circuitry having at least one application specific inte With respect to the use of substantially any plural and/or
grated circuit, electrical circuitry forming a general purpose singular terms herein, those having skill in the art can
computing device configured by a computer program (e.g., translate from the plural to the singular and/or from the
a general purpose computer configured by a computer singular to the plural as is appropriate to the context and/or
program which at least partially carries out processes and/or 45 application. The various singular/plural permutations are not
devices described herein, or a microprocessor configured by expressly set forth herein for sake of clarity.
a computer program which at least partially carries out While particular aspects of the present subject matter
processes and/or devices described herein), electrical cir described herein have been shown and described, it will be
cuitry forming a memory device (e.g., forms of random apparent to those skilled in the art that, based upon the
access memory), and/or electrical circuitry forming a com 50 teachings herein, changes and modifications may be made
munications device (e.g., a modem, communications Switch, without departing from the subject matter described herein
or optical-electrical equipment). Those having skill in the art and its broader aspects and, therefore, the appended claims
will recognize that the subject matter described herein may are to encompass within their scope all such changes and
be implemented in an analog or digital fashion or some modifications as are within the true spirit and scope of the
combination thereof. 55 subject matter described herein. Furthermore, it is to be
Those skilled in the art will recognize that it is common understood that the invention is defined by the appended
within the art to describe devices and/or processes in the claims. It will be understood by those within the art that, in
fashion set forth herein, and thereafter use engineering general, terms used herein, and especially in the appended
practices to integrate Such described devices and/or pro claims (e.g., bodies of the appended claims) are generally
cesses into data processing systems. That is, at least a 60 intended as “open’ terms (e.g., the term “including should
portion of the devices and/or processes described herein can be interpreted as “including but not limited to, the term
be integrated into a data processing system via a reasonable “having should be interpreted as “having at least, the term
amount of experimentation. Those having skill in the art will “includes should be interpreted as “includes but is not
recognize that a typical data processing system generally limited to.” etc.). It will be further understood by those
includes one or more of a system unit housing, a video 65 within the art that if a specific number of an introduced claim
display device, a memory such as volatile and non-volatile recitation is intended, such an intent will be explicitly recited
memory, processors such as microprocessors and digital in the claim, and in the absence of Such recitation no such
US 9,495,684 B2
87 88
intent is present. For example, as an aid to understanding, a media content identification unit configured to identify
the following appended claims may contain usage of the at least one instance of media content based on the at
introductory phrases “at least one' and “one or more' to least one mental state.
introduce claim recitations. However, the use of such 2. The system of claim 1 wherein the presentation unit
phrases should not be construed to imply that the introduc comprises: a display.
tion of a claim recitation by the indefinite articles “a” or “an 3. The system of claim 2 wherein the presentation unit
limits any particular claim containing Such introduced claim comprises: at least one of a desktop display or a mobile
recitation to inventions containing only one such recitation, display.
even when the same claim includes the introductory phrases 4. The system of claim 1 wherein the physiologic activity
“one or more' or “at least one' and indefinite articles such 10 measurement unit comprises: a brain activity measurement
unit.
as “a” or “an” (e.g., “a” and/or “an should typically be 5. The system of claim 1 wherein the physiologic activity
interpreted to mean “at least one' or “one or more'); the measurement unit comprises: a functional near infra-red
same holds true for the use of definite articles used to imaging module.
introduce claim recitations. In addition, even if a specific 15 6. The system of claim 4 wherein the physiologic activity
number of an introduced claim recitation is explicitly measurement unit comprises: a functional magnetic reso
recited, those skilled in the art will recognize that such nance imaging module.
recitation should typically be interpreted to mean at least the 7. The system of claim 4 wherein the physiologic activity
recited number (e.g., the bare recitation of “two recitations.” measurement unit comprises: a magnetoencephalography
without other modifiers, typically means at least two reci module.
tations, or two or more recitations). Furthermore, in those 8. The system of claim 4 wherein the physiologic activity
instances where a convention analogous to “at least one of measurement unit comprises: an electroencephalography
A, B, and C, etc. is used, in general Such a construction is module.
intended in the sense one having skill in the art would 9. The system of claim 4 wherein the physiologic activity
understand the convention (e.g., “a system having at least 25 measurement unit comprises: a positron emission tomogra
one of A, B, and C would include but not be limited to phy module.
systems that have A alone, B alone, C alone, A and B 10. The system of claim 1 wherein the physiologic
together, A and C together, B and C together, and/or A, B, activity measurement unit comprises: a Surrogate marker
and C together, etc.). In those instances where a convention measurement unit.
analogous to “at least one of A, B, or C, etc. is used, in 30 11. The system of claim 10 wherein the physiologic
general Such a construction is intended in the sense one activity measurement unit comprises: an iris response mod
having skill in the art would understand the convention (e.g., ule.
“a system having at least one of A, B, or C would include 12. The system of claim 10 wherein the physiologic
but not be limited to systems that have A alone, B alone, C activity measurement unit comprises: a gaze tracking mod
alone, A and B together, A and C together, B and C together, 35 ule.
and/or A, B, and C together, etc.). It will be further under 13. The system of claim 10 wherein the physiologic
stood by those within the art that virtually any disjunctive activity measurement unit comprises: a skin response mod
word and/or phrase presenting two or more alternative ule.
terms, whether in the description, claims, or drawings, 14. The system of claim 10 wherein the physiologic
should be understood to contemplate the possibilities of 40 activity measurement unit comprises: a voice response mod
including one of the terms, either of the terms, or both terms. ule.
For example, the phrase “A or B will be understood to 15. The system of claim 1 wherein the association unit
include the possibilities of “A” or “B” or “A and B.” comprises: an emotion association module.
With respect to the appended claims, those skilled in the 16. The system of claim 1 wherein the association unit
art will appreciate that recited operations therein may gen 45 comprises: an attention association module.
erally be performed in any order. Examples of such alternate 17. The system of claim 1 wherein the association unit
orderings may include overlapping, interleaved, interrupted, comprises: a cognition association module.
reordered, incremental, preparatory, Supplemental, simulta 18. The system of claim 1 wherein the media content
neous, reverse, or other variant orderings, unless context identification unit comprises: a media content identification
dictates otherwise. With respect to context, even terms like 50 unit configured to identify at least one of movie content,
“responsive to,” “related to,” or other past-tense adjectives television content, music content, literature content, internet
are generally not intended to exclude Such variants, unless content, or advertising content.
context dictates otherwise. 19. The system of claim 1 wherein the media content
identification unit comprises: a media content identification
What is claimed is: 55 unit configured to identify at least one preferred instance of
1. A system comprising: media content.
a presentation unit configured to present to at least one 20. The system of claim 1 wherein the media content
member of a population a plurality of avatars associ identification unit comprises: a media content identification
ated with at least one instance of media content; unit configured to identify at least one instance of media
a physiologic activity measurement unit configured to 60 content at least partly based on a measured mental state of
measure at least one physiologic activity of the at least an individual.
one member of the population, the at least one physi 21. A computer program product comprising: at least one
ologic activity proximate to at least one presented of a machine, article of manufacture, or composition of
avatar among the plurality of avatars associated with at matter including a signal-bearing medium bearing (a) one or
least one instance of media content; 65 more instructions for presenting to at least one member of a
an association unit configured to associate the at least one population a plurality of avatars associated with at least one
physiologic activity with at least one mental state; and instance of media content; (b) one or more instructions for
US 9,495,684 B2
89 90
measuring at least one physiologic activity of the at least one 26. The system of claim 25 wherein the computing device
member of the population, the at least one physiologic comprises: one or more of a personal digital assistant (PDA),
activity proximate to at least one presented avatar among the a personal entertainment device, a mobile phone, a laptop
plurality of avatars associated with at least one instance of computer, a tablet personal computer, a networked com
media content; (c) one or more instructions for associating 5 puter, a computing system comprised of a cluster of pro
the at least one physiologic activity with at least one mental cessors, a computing system comprised of a cluster of
servers, a workstation computer, and/or a desktop computer.
state; and (d) one or more instructions for identifying at least 27. The system of claim 25 wherein the computing device
one instance of media content based on the at least one is operable to receive collected population cohort data from
mental State. at least one memory.
10
22. The computer program product of claim 21, wherein 28. The system of claim 25 wherein the computing device
the signal-bearing medium includes a computer-readable is operable to associate from at least one memory at least one
medium. physiologic activity with at least one mental state.
23. The computer program product of claim 21, wherein 29. The system of claim 10 wherein the surrogate marker
the signal-bearing medium includes a recordable medium. 15 measurement unit comprises: a facial feature measurement
24. The computer program product of claim 21, wherein unit.
the signal-bearing medium includes a communications 30. A system comprising:
medium. a presentation unit configured to present to at least one
25. A system comprising: a computing device; and member of a population a plurality of avatars associ
instructions that when executed on the computing device ated with at least one instance of media content;
cause the computing device to (a) present to at least one a facial feature measurement unit configured to measure
at least one facial feature of the at least one member of
member of a population a plurality of avatars associated the population proximate to at least one presented
with at least one instance of media content; (b) measure at avatar among the plurality of avatars associated with at
least one physiologic activity of the at least one member of
the population, the at least one physiologic activity proxi 25
least one instance of media content;
mate to at least one presented avatar among the plurality of an association unit configured to associate the at least one
avatars associated with at least one instance of media facial feature with at least one mental state; and
content; (c) associate the at least one physiologic activity a media content identification unit configured to identify
at least one instance of media content based on the at
with at least one mental state; and (d) identify at least one least one mental state.
instance of media content based on the at least one mental
State. k k k k k

Us 9495684

Uploaded by

Document Informationclick to expand document information

Copyright:

Available Formats

Us 9495684

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Us 9495684

Uploaded by

Copyright:

Available Formats

USOO9495684B2

(12) United States Patent (10) Patent No.: US 9,495.684 B2

Related U.S. Application Data 2003/0177187 A1 9, 2003 Levine et al.

U.S. PATENT DOCUMENTS 2.83. A 1939. Riyet al. CCaC ( a.

six xxxii: sixx: {xf

presenting at least one characteristic to at least one member of a

associating the at least one physiologic activity with

specifying at least one avatar attribute based on the at least one

^* * * '' ... ***

You might also like