Cogsci Tutorial02

Cognitive Science Tutorial
APEX/CPM-GOMS
An Architecture for Modeling Huma Performance in Applied HCI Domain
Roger Remingto, Bonnie J*,hn lonsoera n o A V , , eed Da Michael atessa , & Michael FrMichael lal M , s 2a Robert HarriEric Dahlm, n
NASA Ames Research Center *Carnegie Mellon University 2Colorado State University
,
7 August 2002
Cognitive Science 2002 Fairfax, VA
What this tutorial will

Basics of Apex needed to build CPM-GOMS models
Hierarchical task decomposition Templates for routine task behaviors Automatic scheduling of primitive actions
Applying Apex to a simple HCI example

Text editing task Use predefined CPM-GOMS templates to predict performance
Enough to allow you to refine the models suppl write simple models for HCI tasks
7 August 2002
Fair warning
Apex is still a research tool
The models supplied will run but are not guaranteed to ma correct predictions We are still learning about how to use the architecture t important classes of HCI tasks (e.g. typing) The Apex architecture will change as a result of bug fixe features to make it easier to use
7 August 2002
Schedule
Overview of APEX & CPM-GOMS - - - B r e a k ( ~ 1 0 : 3 0 - 1 0 : 4 5 ) -(9:30 - 10:00) CPM-GOMS Text editing world (10:00 KLM-GOMS
CPM-GOMS Templates 11:45) Resource scheduling in Apex World building
Basic Apex functionality Sequential execution Further exploration of Primitive operators in Apex unctionality (11:45 f
Apex 12:15 )
Procedure Description Language (PDL) Wrap-Up (12:15 - 12:30) CMN-GOMS Overview of CD ROM & Web
Hierarchical task decompositionPage in Apex Q & A Selection rules Feedback
7 August 2002
Apex and CPM-GOMS

CPM-GOMS
Computational Architecture forMethodology for Human Performance Modeling Human Performance Modeling
Task Analysis Method ( Goals, Task Representation Language Operators,ethods, M Selection) Human Resources Resource Scheduler No Software Implementation
Software System Implemented inNo automatic resource schedulin Lisp Theory of Human Resources No built-in theory of Human (Cognitive, Perceptual, Motor) Resource Interaction
7 August 2002
Apex Apex
7 August 2002
Complex dynamic environments
Time Pressure: sers must make timely inputs U Multitasking Users juggle multiple tasks : S Predictability imilar patterns occur over time : Unpredictability nterruptions occur : I Mixed-initiativeecision authority distributed, includes : D other users and automated systems
7 August 2002 Cognitive Science 2002 Fairfax, VA 7
Motivations
Construct a modeling system that can make usef predictions about skilled operator behavior i complex dynamic environments Make cognitive modeling more accessible to non specialists, especially in the design phase
Reduce model development time Simplified cognitive architecture Reusable packets of psychological theory that can attach standard task analysis (templates)
7 August 2002
Resource Allocation
Apex simulates an agent planning and scheduling limited resources to accomplish multiple task A multi-tasking agent must allocate resources proactively and reactively
Mechanisms for task suspension and recovery Mechanisms for parallel task execution subject to resource and constraints logical dependencies
These mechanisms are also important for HCI predictions that arise from the interleaving o primitive cognitive, pe, c& pmotor cts r e tuala
7 August 2002
The Apex Architecture
Sim -
7 August 2002
10
GOM S
O A L S P E R A T O R S E T H O D S E L E C T I O N
See the following paper on your CD or the Apex ftp site: John, B. E. (in preparation) Information Processing and Skilled Behavior, John M. Carroll (ed.), Toward a multidisciplinary science of human computer interaction. Morgan Kaufman.
GOMS Components
Task Analysis
Goals Task decomposed into nested hierarchy of go : and subgoals Operators Hierarchy terminates in operators, whose : actions cause transitions between states Methods: Sequences of operators executed to accomplish a set of subgoals Selection Rules : Rules that determine which method to use
Performance Computation
Operator execution takes time Sequence of operators determines sequence of over behaviors and task time
7 August 2002
12
Varieties of GOMS
Keystroke-Level-Model (KLM)
Flat task structure Mentally prepare coupled with primitive
Card-Moran-Newell GOMS (CMN-GOMS)

Hierarchical goal decomposition Primitive operators in task domain (e.g. move mouse)
Cognitive-Perceptual-Motor (CPM-GOMS)
Combines hierarchical goal decomposition with primitive CPM resources based on Model Human Processor
7 August 2002
13
Text Edit World
7 August 2002
14
The fox jumped over the lazy quick brown dog

Several methods exist for moving text
Cut and paste from edit menu Cut and paste using keyboard shortcuts Delete intervening text and retype
Backspace over text Highlight and cut Highlight and delete
We will model cutting and pasting from the edit menu
7 August 2002
15
Highlight text
7 August 2002
16
Select dit menu E
7 August 2002
17
C Select ut
7 August 2002
18
Position text cursor
7 August 2002
19
E Select dit
7 August 2002
20
Select aste P
7 August 2002
21
Edit complete
7 August 2002
22
KLM-GOMS Model: KLM-GOMS Model: Sequential execution in Ap Sequential execution in Ap
7 August 2002
23
Keystroke Level Model (KLM

Allows designer to create a simple model of co HCI tasks involving key strokes All operators are primitives that can be assig completion times One mental operator placed with 5 heuristics Flat structure with assumption of sequential e makes it easy
7 August 2002
24
KLM model
Description Operator Mentally prepare (Heuristic Rule 0) M Move cursor to quick (Heuristic Rule 1) P Double-click mouse button K Move cursor to brown (Heuristic Rule 1) P Shift-click mouse button K Mentally prepare (Heuristic Rule 0) M Move cursor to Edit menu (Heuristic Rule 1) P Click mouse button K Move cursor to Cut menu item (Heuristic Rule 1) P Click mouse button K Mentally prepare (Heuristic Rule 0) M Move cursor to before fox (Heuristic Rule 1) P Click mouse button K Mentally prepare (Heuristic Rule 0) M Move cursor to Edit menu (Heuristic Rule 1) P Click mouse button K Move cursor to Paste menu item (Heuristic Rule 1) P Click mouse button K TOTAL PREDICTED TIME
7 August 2002 Cognitive Science 2002 Fairfax, VA
Duration (sec) 1.35 1.10 0.40 1.10 0.40 1.35 1.10 0.20 1.10 0.20 1.35 1.10 0.20 1.35 1.10 0.20 1.10 0.20 14.90
25
Main non-primitive PDL proce

(procedure :sequential (index (edit sentence)) indicate that steps are indicate that steps are executed in the order listed (mentally-prepare highlight-text-to-be-moved) executed in the order listed (move-cursor-1100 first-word-quick) (double-click-mouse first-word-quick) (move-cursor-1100 last-word-brown) (shift-click-mouse last-word-brown) (mentally-prepare cut) The indexclause is the The indexclause is the (move-cursor-1100 Edit-menu) pattern used to match pattern used to match (click-mouse Edit-menu) procedure calls. procedure calls. (move-cursor-1100 Cut-item) (click-mouse Cut-item) (mentally-prepare highlight-destination) (move-cursor-1100 destination-space) (click-mouse destination-space) (mentally-prepare paste) (move-cursor-1100 Edit-menu) When this stepenisble" isb "ed a When this stepena l""d it looks for a procedure (click-mouse Edit-menu) it looks for a procedure (move-cursor-1100 Paste-item) whose index clause whose index clause (click-mouse Paste-item) matches c"lick-mou" e li se matches c" ck-mou"s )
:sequential used to is :sequential used to is
7 August 2002
26
Top level PDL procedure

All Apex models begin with a high-level proced that invokes "do-domain" along with the top-le domain procedures
(procedure :sequential (index (do-domain)) (edit sentence) (stop))
stop is a special command stop is a special command to terminate the to terminate the simulation simulation
7 August 2002
27
Primitive procedures
The default:isncurrent : co The defaultconcurrent is
(procedure (index (mentally-prepare ?activity)) (profile memory) (step s1 (start-activity memory memory-act :duration 1350 => ?a)) (step s2 (terminate) (waitfor (completed ?a))))
Procedures that us Procedures that us resources must conta resources must conta " profi" eclause p l " rofi" eclause l
(procedure (index (move-cursor-1100 ?target)) (profile right-hand) (step s1 (start-activity right-hand mouse-move-act :object ?target :duration 1100 => ?a)) (step t (terminate) (waitfor (completed ?a))))
" termina" egenerates an t " erminate generates an t " event signalling end the event signalling end the of a procedure of a procedure
Duration specifies the time t Duration specifies the time t " waitf"rspecifies conditions by this activity to complete w o " aitfor specifies conditions by this activity to complete " by previous research for step execution by previous research for step execution
Running Apex
Double-click Double-click " ApexApp" ApexApp" "
7 August 2002
29
RunningSherpa
Lisp Lisp Listener Listener
Start Sherpa by by Start Sherpa clicking the clicking the "Start" button "Start" button
Sherpa Sherpa interface interface
7 August 2002
30
m Simworld enu
Click here Click here to see to see available available simworlds simworlds
7 August 2002
31
Loading from erpa Sh
SelectF" xKLM" from the Fo SelectoxKLM" from the " list. The name of the list. The name of the simworldmust be in the simworldmust be in the a dlis s s " allworl.li"pfile. " llworl.ds "pfile.
7 August 2002
32
Running the KLM model
Step Step Reset Reset Run Run Pause Pause
7 August 2002
33
KLM trace
7 August 2002
34
Inspecting the trace
Double-click on the Double-click on the "Task" label to label to "Task" inspect a task inspect a task
7 August 2002
35
Generating a PERT chart
Click here to Click here to generate a generate a PERT chart PERT chart
7 August 2002
36
KLM PERT chart
7 August 2002
37
CMN-GOMS : CMN-GOMS : Hierarchical task decomposit Hierarchical task decomposi in Apex in Apex
7 August 2002
38
Elements of a CMN-GOMS model

Top-leel oal v g Subgoals
Edit man r pt usci
e.g., ituni -ts , acui e- nittas , ed - t a k q r u - k execut - nittaskmo di y t xt eu , f -e , ver fy edi, m ve-t xt c t-ext, i - t o e ,u t paste te t - x, e.g., f te by movi wor ,or modi y xt ng ds del ng p asesor i er i wor ; eti hr , ns t ng ds hi i ghlght te as a ph e,or as xt ras ar tra y t xt bi r e , e.g., he ext to h i ed is a i t t f be i ght ghl phr ase mad up of w ds us e or , e hi hlght- hr secom pr s d- f wo rds, g i p a ie o r ta y e el s hi hlght-abi r r -t xt se u g i e
M et ods h
e i Sel ct on
7 August 2002
39
Top-level GOMS
goal: edit-manuscript . goal: edit-unit-task ... repeat until no more unit tasks . . goal: acquire unit-task ... remembered if task not . . goal: execute-unit-task ... if a unit task was found
7 August 2002
40
Top-level GOMS
goal: edit-manuscript . goal: edit-unit-task ... repeat until no more unit tasks . . goal: acquire unit-task ... remembered if task not . . goal: turn-page at end of manuscript page . ... if . . goal: get-from-manuscript . if a unit task was found . . goal: execute-unit-task ...
7 August 2002
41
Top-level GOMS
goal: edit-manuscript . goal: edit-unit-task ... repeat until no more unit tasks . . goal: acquire unit-task ... remembered if task not . . . goal: turn-page ... if at end of manuscript page . . . goal: get-from-manuscript if a unit task was found . . goal: execute-unit-task ... . . goal: modify-text . . . . verify-edit . 1.35
7 August 2002
42
Top-level GOMS
goal: edit-manuscript . goal: edit-unit-task ... repeat until no more unit tasks . . goal: acquire unit-task ... remembered if task not . . . goal: turn-page ... if at end of manuscript page . . . goal: get-from-manuscript if a unit task was found . . goal: execute-unit-task ... . . . goal: modify-text . . . [select: . goal: move-text * ... text is to be moved if . . . . goal: delete-phrase phrase is to be deleted ... a if . . . . goal: insert-word] word is to be inserted ... a if . . . . verify-edit 1.35
7 August 2002
43
Expansion of move-text
goal: move-text . goal: cut-text . goal: paste-text
7 August 2002
44
goal: move-text . goal: cut-text . . goal: highlight-text . . goal: issue-cut-command . goal: paste-text
7 August 2002
45
goal: move-text . goal: cut-text . . goal: highlight-text . . goal: issue-cut-command . . . move-cursor-to-edit-menu . . . click-mouse-button . . . move-mouse-to-cut-item . . . verify-highlight . . . click-mouse-button . goal: paste-text
1.10 0.20 1.10 1.35 0.20
7 August 2002
46
Full move-text
goal: move-text . goal: cut-text . . goal: highlight-text . . . [select**: goal: highlight-phrase-comprised-of-words . . . . move-cursor-to-first-word 1.10 . . . . double-click-mouse-button 0.40 . . . . move-cursor-to-last-word 1.10 . . . . shift-click-mouse-button 0.40 . . . . verify-highlight 1.35 . . . goal: highlight-arbitrary-text . . . . move-cursor-to-beginning-of-text . . . . press-mouse-button . . . . move-cursor-to-end-of-text . . . . release-click-mouse-button . . . . verify-highlight ] . . goal: issue-cut-command . . . move-cursor-to-edit-menu 1.10 . . . click-mouse-button 0.20 . . . move-mouse-to-cut-item 1.10 . . . verify-highlight 1.35 . . . click-mouse-button 0.20 . . . . . . . . . . . goal: paste-text . goal: position-cursor-at-insertion-point . . move-cursor-to-insertion-point 1.10 . . click-mouse-button 0.20 . . verify-position 1.35 . goal: issue-paste-command . . move-cursor-to-edit-menu 1.10 . . click-mouse-button 0.20 . . move-mouse-to-paste-item 1.10 . . verify-highlight 1.35 . . click-mouse-button 0.20
7 August 2002
47
Top-level CMN PDL

(procedure :sequential (index (do-domain)) (sevars initial) t- (edit manuscript) (stop) ) State vector with State vector with global variables global variables
(procedure :sequential (index (set--initial)) vars (set *task* remembered) (set *edit-task* move-text) (set *text-to-be-moved* all-words) (set *first-word-to-be-moved* quick) (set *last-word-to-be-moved* brown) (set *destination* space-before-fox) )
Top-level GOMS in PDL

goal: dit-manuscript e . goal: edit-unit-task ... repeat until no more unit tasks . . goal: cquire unit-task a ...if task not remembered e . . goal: xecute-unit-task if a unit task was found ... (procedure (indexedit manuscr)) t ( ip (step s(?this-step) 1 (select ?this-step (depends-on *task* ((remembered no-op) (forgotten uire-unit-))))) acq task (step s2 (execute-unit-task r s1)) ?this-step a variable that ) waitfo ? ( ?this-stis a variable that ep is (step t (terminate) or s1 ?s2))) bound toacquire-unit-task waitf ( ? ac bound to quire-unit-task
The select clause in The select clause in edit manuscript edit manuscript implements an "if" implements an "if" selection rule selection rule
New PDL Constructs

procedure :
:sequential steps executed serially in the order listed : oncurrent c steps executed in parallel, subject to resource constraints and logical dependencies
index: unique name of a procedure used to match stored proced step: a sub-procedure of a larger procedure step tag : unique label for each step, requiredurrent in :conc only waitfor enables sequencing of steps in a :concurrent procedur :
indicating dependencies -- events that satisfy conditions for execution
select: step-level terminate:

7 August 2002
clause that conditions the execution of that the evaluation of an expression causes an event signaling the end of a procedure
50
Execute unit task

. . . . . . ex . goal:ecute unit task . .goal: dify-text mo . . [select: . goal: move-text * . . . goal: delete-phrase . . . goal: insert-word] ... verify-edit
(procedure (index (execute-unit-task )) (step s(modify-tex?modification-task) 1 t (select ?modification-task (second (assoc a*e*)))) ask* v rlidit-t st (step s(verify ediwaitfor s1)) 2 ) (t ? (step t (terminawaitfor s1 ?s2))) (te) ?
?modification-task ?modification-task bound to value of bound to value of *edit-task* which is *edit-task* which is move-text move-text
7 August 2002
51
Modify text by moving

. . . . . . . . . . . . goal: modify-text . goal: move-text* . . goal: cut-text . . goal: paste-text
(procedure :sequential (indexmodify-text ve-tex)) ( mo t (cut-te) t x (paste-text ))
7 August 2002
52
Cut text
goal: cut-text . goal: highlight-text . goal: issue-cut-command
(procedure :sequential (indexcut-te)) ( xt (highlight-text ) (issue-menu-command dit-menu Cut-item)) E
7 August 2002
53
Verify highlight: primitive procedures

. . . verify-highlight.35 1 y (verifhighlight ?Menu-item) :duration startin s :duration a tartin a activity step specifies activity step specifie completion time in completion time in milliseconds milliseconds
(procedure (index (verify ?type ?item)) (profile memory) (step s1 (start-activity memory memory-act :duration 1350 (step t (terminate) f( r completed ?a)))) wait o (
7 August 2002
54
CMN-GOMS in PDL looks a lot like GLEAN

GLEAN was a GOMS tool created by ieras and K Dave s his colleagues at the University of Michigan i
NaturalGOMS language, NGOMSL Available for download from David web pages Kieras s Ask Scott Wood (instructor at a tutorial this aftern you want more information -- he was a major develope
Kieras D. E., Wood, S. D., K., & Hornof (1995). GLEAN: A , , Abotel , A. Computer-Based Tool for Rapid GOMS Model Usability Evaluation Interface Designs. UIST95 Proceedings .
7 August 2002
55
CMN primitives
Can be at any level of granularity
Operator execution assumed to be sequential
Usually events in the task domain

Mouse movement Mouse clicking Key presses
For really skilled behavior the assumption of s operator execution breaks down
Parallelism -- actions overlap to various degrees Need to go deeper in the cognitive architecture to c the parallelism that leads to overlap
7 August 2002
56
CPM-GOMS : CPM-GOMS : Interleaving templates to ac Interleaving templates to ac parallel and serial executi parallel and serial executi cognitive, perceptual, and m cognitive, perceptual, and m operators operators
Goal of CPM-GOMS
Model the time for highly skilled people to ac tasks by using methods made up of elementary Cognitive, Perceptual, and Motor operators Create cognitively-plausible, treusable that empl tes capture the parallelism and constraints in th methods Integrate templates into a model that allows predictions to flow from a CMN-GOMS task hier
Shield the analyst from the complexity of templates Keep the analyst in the task domain
7 August 2002
58
Highly Skilled
KLM and CMN-GOMS predict human behavior well
When the user knows the procedures of a domain well presented with a new task When operators can be assumed to work sequentially
CPM-GOMS is needed
When task becomes so routine that users perform act in parallel to achieve faster execution time Examples in the lab:
Card, Moran & Newell, 1983, Chapter 8, Section 4.-- Te Baskin & John, 1998 -- CAD John, et. al., 2002 -- ATM
Example in the field

Gray, et. al., 1993 -- Project Ernestine: telephone op
Pedagogical goals for the CPM-GOMS section

Show the relationship between CMN-GOMS (with i assumption of serial operators) and CPM-GOMS relaxes that assumption) Show how to go from CMN-GOMS to CPM-GOMS using Apex
Templates of common behaviors Interweaving of templates Sherpas PERT chart generator
Show how to finesse n interactive ld in Apex by a wor using ApexBuilder a global variable list, and *varlist*
We ll do this first
Interactive World Simulati

Requirements
Objects that can behave
Can be perceived by Apex Can be acted on by Apex Can change either in response to actions or by themsel
Expressed in a form that Apex can recognize them
A simple finessing of these requirements

Html storyboards define the objects, built with ApexBuilder extensions DreamWeaver to ApexConvertor translates this html into Apex-compatibl Global list of attribute-value pairs and procedures and change these to simulate interactivity
7 August 2002
61
Carnegie Mellon University s Masters of HCI & Masters of Software Engineer projects this year
(completed 2aug02)
Charge: Make it
easy to build simple worlds in Apex and possible to hook up to external world simul (the next few slides come directly from the presentations, and you have their easy-to-u system on the CD)
An Apex World
7 August 2002
63
World visualization
You have to run some model to get it to build the world just to get it to understand that there is a world. participant CI
7 August 2002
64
7 August 2002
65
7 August 2002
66
7 August 2002
67
7 August 2002
68
7 August 2002
69
7 August 2002
70
A few minutes later
7 August 2002
71
7 August 2002
72
7 August 2002
73
Converting a world created with ApexBuilder into Apex-compatible LISP
7 August 2002
74
7 August 2002
75
7 August 2002
76
7 August 2002
77
7 August 2002
78
7 August 2002
79
7 August 2002
80
Modifying a World: Using custom images or default iconic images
7 August 2002
81
7 August 2002
82
7 August 2002
83
7 August 2002
84
7 August 2002
85
Initialize file for the fo

(initialize-simulation (let* Words in ((world (make-instance locale :name world)) Words in the text (Human (make-instance Human :name agent :locale world the text :location (0 0 400))) (Mouse (create-Mouse world)) (quick (make-instance interface-object :name quick :locale world :pos (258 59) :dimensions (40 19))) Menus and Menus and (brown (make-instance interface-object :name brown :locale menu items world menu items :pos (305 57) :dimensions (47 17))) (Edit-menu (make-instance interface-object :name Edit-menu :locale world :pos (90 10) :dimensions (33 18))) Places of (File-menu (make-instance interface-object :name File-menu :locale world of Places interest in :pos (54 10) :dimensions (35 19))) interest in (Cut-item (make-instance interface-object :name Cut-item :locale the dtext worl the text :pos (146 65) :dimensions (145 20))) ( s p a c e - b e f o r e - f o x ( m a k e - i n s t a n c e i n t e r f a c e - o b j e c t : n a m e s p a c e - b e f o r e - f o x : l o ca :pos (60 56) :dimensions (6 19))) (Paste-item (make-instance interface-object :name Paste-item :locale world :pos (146 98) :dimensions (145 17)))) mapc #assemble (list Human Mouse quick brown Edit-menu File-menu Cut-item space-b ( Paste-item))))
Simple Interactivity: Using*VARLIST*

Use a globally-accessible list of attribute-va that can be read and changed through PDL as a simple simulation of interaction in the world In the fox example
The words start out visible and change to highlighte the are double- or shift-clicked The menu names are always visible, but the menu item change from invisible to visible when the menu name clicked The destination space, i.e., the space-before-fox, visible
7 August 2002
87
Additions to vars- nitial) (set i (procedure :sequential (index (set- -initial)) vars ; ; V a r i a b l e s t h a t e n c o d e i n f o r m a t i o n a b o u t t h e h u m a n u s er ;; just like in the CMN-GOMS model (see slide 54) ;; Variables that encode information about the initial st ;; These variables must match the names of the interface ;; the initialize file (set quick visible) We ll write PDL that We ll write PDL that (set brown visible) changes these values changes these values (set Edit-menu visible) as the model interacts as the model interacts with the world (set File-menu visible) with the world (set Cut-item invisible) (set space-before-fox visible) (set Paste-item invisible)) Cognitive Science 2002
7 August 2002 Fairfax, VA 88
Interacting through *varlist*

(to be digested at your leisure
Apex has the ability to call out to lisp to re manipulate t*varlist* he
See primitives-special.lisp on the CD or on the Apex
Reading values, lisp or PDL

(depends-onvar> <val <meth-list>) is a lisp function < > in the select clause (seen on slides 55 & 57) (extract ?info) is a PDL procedure that returns the the attribute ?info in *varlist* (variable ? ?val) is an event that waitfor var Apex can
Changing a value in *varlist*

(set var ?valis a call to a PDL procedure that sets ? ) of the attribute to be ? It also causes an event th var ? val. Apex can understand so it knows that the value has b changed. This event is the (variable above. var val) ? ?
Summary of World Building

We have just seen a simplified model of external world
ApexBuilder ApexConverter initialize.lisp -> -> *varlist* to hold the state of the world Several special primitives and lisp functions to rea change the *varlist* Suitable approach for many HCI tasks
Apex also supports communication with a complete interactive world when you nee
Build any world you wish in lisp within the Apex sy Have Apex operate an external simulation (e.g., CMU students have controlled a browser and the AMBR ATC through HLA) Both approaches are beyond the scope of this tutoria Cognitive Science 2002
Pedagogical goals for the CPM-GOMS section

Show the relationship between CMN-GOMS (with i assumption of serial operators) and CPM-GOMS relaxes that assumption) Show how to go from CMN-GOMS to CPM-GOMS using Apex
Templates of common behaviors Interweaving of templates Sherpas PERT chart generator
Show how to finesse n interactive ld in Apex by a wor using ApexBuilder a global variable list, and *varlist*
7 August 2002
91
The relationship between CMN-GOMS and CPM-GOMS

CMN-GOMS operators
Can be at any level from milliseconds to seconds to As long as seriality & independence of history are maintai
CPM-GOMS operators
Are at the level of single cognitive, perceptual, a actions, in the tens of milliseconds The cognitive, perceptual and motor processors are within themselves but parallel with respect to each Combine these into psychologically plausible, reusa patterns called templatesusing task analysis and the constraints of the Model Human Processor (Card, Mor Newell, 1983) Templates then implement the CMN-GOMS operators (Jo & Gray, CHI Tutorials, 1992-1995)
An example template: Fast-move-click

world
0 new-cursor-location 290
MHP resource dependencies logical dependencies logical and resource dependencies
vision
perceivetarget
50
50 initiatePOG
50 verifytarget pos
cognition
START initiate-movecursor
attendtarget 590 (FL)
50 initiatemouseDn 100 mouse Dn
right hand
movecursor
100 mouse Up
30
eye move
POG
from Gray and Boehm-Davis (2000)
7 August 2002
93
Composing Templates: A task may require two Fast in succession

0 new-cursor-location 290 perceivetarget 0 new-cursor-location 290 perceivetarget
50 START initiate-movecursor attendtarget 590 (FL) movecursor
50 initiatePOG
50 verifytarget pos
50 initiatemouseDn START initiate-movecursor
50 attendtarget 590 (FL)
50 initiatePOG
50 verifytarget pos
50 initiatemouseDn
100 mouse Dn 30 POG
100 mouse Up
movecursor
100 mouse Dn 30 POG
100 mouse Up
FAST M/C
FAST M/C
7 August 2002
94
But people seem to interweave templates when they become sufficient skilled at a task
50 initiatePOG
50 verifytarget pos
50 attendtarget
50 initiatePOG
50 initiatemouseDn START initiate-movecursor 590 (FL) 100 mouse Dn 100 mouse Up movecursor
50 verifytarget pos
50 initiatemouseDn
100 mouse Dn 30 POG
100 mouse Up
30 POG
FAST M/C
FAST M/C
7 August 2002
95
We used to do interweaving painstakingly by hand Project Mac in Now Apex will do this automaticall us through dynamic resource sched u
50 initiatePOG
50 verifytarget pos
50 attendtarget
50 initiatePOG
50 initiatemouseDn START initiate-movecursor 590 (FL) 100 mouse Dn 100 mouse Up movecursor
50 verifytarget pos
50 initiatemouseDn
100 mouse Dn 30 POG
100 mouse Up
30 POG
FAST M/C
FAST M/C
7 August 2002
96
Basic procedure
Create a CMN-GOMS-like model that bottoms out calls to pre-established templates Allow all primitive operators to contend for r (e.g., vision, audition, cognition, hands, vo
Apex has mechanisms to dynamically schedule resource of which is used in CPM-GOMS models The CMN goal hierarchy guides Apex to assign the re to the appropriate primitives to achieve cognitively interweaving Templates have been created for using a mouse, typin reacting to a sound or visual signal, all included templates.lisp file with the Apex system (see the Templates Manual included on the CD and on site)
Converting CMN-GOMS PDL into CPM-GOMS PDL A new concept calrank led
CMN-GOMS PDL -> CPM-GOMS (procedure (index (edit manuscript)) (step s1 (?this-step) (select ?this-step (depends-on *task* ((remembered no-op) (forgotten acqu ) (step s2 (execute-unit-task) (rank 1)) The rank clause in (step s2 (execute-unit-task) waitfor ( ?s1)) The rank clause in in each step tells (step t (terminate)k(2)) ?s2))) waitfor (ran ?s1 in each step tells Apex how to (step t (terminate) ( ?s1 ?s2))) waitfor Apex how to resolve resource resolve resource r . Thisrankwas aaitfor There will rarely conflicts a w . Thisank was w aitfor There will rarely conflicts be waitforsexcept in terminate steps in except in terminate steps in be waitfors CPM-GOMS models CPM-GOMS modelsScience 2002 Cognitive
A simplified The :ranked type
syntax: of procedu
CMN-GOMS Just change Just change :sequentiato l :sequentito al (procedure :sequential :rankedand Apex :r and Apex (index (modify-text move-text)) ankedin the rest fills in the rest fills (cut-text) (paste-text)) CPM-GOMS (procedure :ranked (index (modify-text move-text)) (cut-text) (paste-text))
Bottom out in pre-establish template steps

CMN-GOMS A move operator A move operator (procedure :sequential and aclick operator and click operator a (index (issue-menu-command ?Menu ?Menu-item)) change to the change to the (move-cursor-1100 ?Menu) fast-move-click fast-move-click (click-mouse ?Menu) operator operator (move-cursor-1100 ?Menu-item) established by Gray established by Gray (verify highlight ?Menu-item) and Boehm-Davis and Boehm-Davis (click-mouse ?Menu-item)) CPM-GOMS (procedure :ranked A changeunder the hood A changeunder the hood (index (issue-menu-command ?Menu ?Menu-i Movement time will be Movement time will be (fast-move-click calculated with s Law Fitts calculated with s Law Fitts :target ?Menu instead CMN sestimate of estimate instead CMN s of :effected-object ?Menu-item :effect-on-object visible) (fast-move-click Parameters are Parameters are :target ?Menu-item passed that relate to passed that relate to :effected-object ?Menu-item the world Cognitive Science 2002 the world 100 7 August 2002 Fairfax, VA :effect-on-object selected)
Run the model Click on the PERT Chart but
7 August 2002
101
Remember Now we ll
world vision
Fast-move-click? implement it in P
0 new-cursor-location 290 perceivetarget
MHP resource dependencies logical dependencies logical and resource dependencies
50
50 initiatePOG
50 verifytarget pos
cognition
START initiate-movecursor
attendtarget 590 (FL)
50 initiatemouseDn 100 mouse Dn
right hand
movecursor
100 mouse Up
30
eye move
POG
from Gray and Boehm-Davis (2000)

102
7 August 2002
Under the hood A quick peak at a templat

;FAST-MOVE-CLICK-R-HAND-ON-MOUSE Steps are either cognitive, Steps are either cognitive, (procedure perceptual, or motor (index (fast-move-click-R-hand-on-mouse perceptual, or motor :target ?target Labeledc, p, morfor convenience c, Labeled p, orfor convenience m :effected-object ?effected-object :effect-on-object ?effect)) A waitfor encodes the encodes the (step c1 (initiate-move-cursor ?target)) A waitfor (step m1 (step c2
(step c3 (step m2 (eye-movement ?target) ?c3)) (waitfor (step p1 (perceive-target-complex ?target) (variable ?target visible))) waitfor ?m2 (
logical dependencies seen logical dependencies seen (move-cursor ?target)?c1)) (waitfor in the PERT chart in the PERT chart (attend-target ?target)) (resource dependencies (resource dependencies are emergent) (initiate-eye-movement ?target) (waitfor ?c2)) are emergent)
(step c4 (verify-target-position ?target) ( waitfor ?c3 ?p1)) (step c5 (initiate-click ?target) ?m1)) waitfor ?c4 ( (step m3 (mouse-down ?target) ( waitfor ?m1 ?c5)) (step m4 (mouse-up ?target) ( waitfor ?m3)) (step t (terminate) ( waitfor ?m4 ?rvr1 ?rvr2)) )
7 August 2002 Cognitive Science 2002 Fairfax, VA
A step canitfoan wa r wa r A step canitfoan occurrence in the world occurrence in the world
103

;FAST-MOVE-CLICK-R-HAND-ON-MOUSE (procedure (index (fast-move-click-R-hand-on-mouse :target ?target :effected-object ?effected-object :effect-on-object ?effect)) (step c1 (initiate-move-cursor ?target)) (step m1 (move-cursor ?target)?c1)) (waitfor (step c2 (attend-target ?target)) (step c3 (initiate-eye-movement ?target) (waitfor ?c2)) (step m2 (eye-movement ?target) ?c3)) (waitfor (step p1 (perceive-target-complex ?target) (variable ?target visible))) waitfor ?m2 ( (step (step (step (step (step (step w1 c4 c5 m3 w2 m4 (WORLD?target fixated-upon) ( ?p1)) 0 waitfor (verify-target-position ?target) ( waitfor ?c3 ?p1)) (initiate-click ?target) ?m1)) waitfor ?c4 ( (mouse-down ?target) ( waitfor ?m1 ?c5)) (WORLD?effected-object ?effect) ?m3)) 0 waitfor( (mouse-up ?target) ( waitfor ?m3))
World steps set an World steps set an attribute in varlist the * attribute in varlist the * to a value (and can take to a value (and can tak time to do so) time to do so)
(step t (terminate) ( waitfor ?m4 ?rvr1 ?rvr2)) )


;FAST-MOVE-CLICK-R-HAND-ON-MOUSE Virtual resources encode the (procedure Virtual resources encode the (index (fast-move-click-R-hand-on-mouse intention to use a resource intention to use a resource :target ?target in the template in the template :effected-object ?effected-object :effect-on-object ?effect)) (step c1 (initiate-move-cursor ?target)) (step hvr1 (hold-resource right-hand-block :ancestor 1) ( waitfor ?c1)) (step m1 (move-cursor ?target)?c1)) (waitfor (step c2 (attend-target ?target)) (step hvr2 (hold-resource vision-block :ancestor 1) ( waitfor ?c2)) (step c3 (initiate-eye-movement ?target) (waitfor ?c2)) (step m2 (eye-movement ?target) ?c3)) (waitfor (step p1 (perceive-target-complex ?target) (variable ?target visible))) waitfor ?m2 ( (step rvr2 (release-resource vision-block :ancestor 1) ( waitfor ?p1)) (step w1 (WORLD?target fixated-upon) ( ?p1)) 0 waitfor (step c4 (verify-target-position ?target) ( waitfor ?c3 ?p1)) (step c5 (initiate-click ?target) ?m1)) waitfor ?c4 ( (step m3 (mouse-down ?target) ( waitfor ?m1 ?c5)) (step w2 (WORLD?effected-object ?effect) ?m3)) 0 waitfor( (step m4 (mouse-up ?target) ( waitfor ?m3)) (step rvr1 (release-resource right-hand-block :ancestor 1) ( waitfor ?m4)) (step t (terminate) ( waitfor ?m4 ?rvr1 ?rvr2)) ) Cognitive Science 2002
la la
Remember, our intention i analysts never have to under the hood!
The CPM-GOMS models we re providing

Fox-CPM
Follows this tutorial and the John (in preparation) chapter provided on the CD and at the ftp site
ATM-CPM-world
The models discussed in our recent CHI paper (John, et. al. 2002), also provided
CAD-world (this is cool, check it out)

A model of drawing an L-shape in a CAD system (Bask John, 1998) This contains both mousing templates and typing The template-level PDL code for typing
Is simply (type-string any string you want ) Is automatically generated with lisp code under the ho Implements TYPIST (John, 1996)
Do CPM-GOMS models built wit Apex match human performance

Fox-CPM
We have no data yet
ATM-CPM-world
Match to data reported in our recent CHI paper (see
CAD-world
Model matches the data in of Baskin & John (1998) as the hand-done CPM-GOMS models did Makes predictions of errors, which have been observe analysis of Baskin & John data, still in progress)
TYPIST
Automatically-generated template-level PDL implement theory that matched many experiments in transcriptio (but individual experiments not modeled yet)
Human-model comparisons
1400 No-Interleaving Model 1200 CPM-GOMS Model Mean for Subjects Fitts's Law
1000
800
600
400
200
7 August 2002
108
Summary of GOMS in Apex

Main points to remember
Apex can express three common versions of G using PDL: KLM, CMN-GOMS and CPM-GOMS Apex can generate CPM-GOMS models automatically from PDL that bottoms out in established templates Cognitively plausible, reusable templates a provided for constructing CPMGOMs models So far, the CPM-GOMS models built in Apex m human data as well as the same models constructed by hand
7 August 2002
109
Known bugs
Select clauses in an interactive world (discov last week)
Select clauses are giving odd behavior when they mu global variables that change during task execution The templates in the templates.lisp file work for A and fox-worlds, but may display this odd behavior in models We have worked around this problem in the CAD-world the CD (if you really want to know how, ask Bonnie J Mike Matessa off line)
PERT chart
Dependency lines are not drawn completely correctly dependency lines from world events and sometimes ac templates
Multitasking Under Time-Pressur Multitasking Under Time-Pressur Uncertainty Uncertainty
Multitasking skill
Delay answering phone until finished typing sen heuristic: prefer to delay interrupt until good st Pull over to side of road before studying map Drive back onto road (but drive to start point) t don Do something useful when stopped at a red light
Multitasking can be viewed as skilled behavior for managing task based on learnedtics domain-dependent applications of general heu tac
7 August 2002
112
Challenge
Create agents with human-level ability to emplo diverse multitask management tactics General heuristics underlying tactics architecture mechanisms Task-specific knowledge specialized representation elements task representation methodology
7 August 2002
113
Apex projects: all involve multi

Human-level competence in aviation tasks
Interface evaluation based on CPM-GOMS
Autonomous robots
7 August 2002
114
7 August 2002
115
Action selection architect requirements

Many domains of practical interest are demandin the sense that a skilled agent must : Cope with time-pressure
Cant deliberate endlessly
Cope with uncertainty

Cant completely know or predict world state Actions may fail or produce undesirable side-effects Preconditions may become unsatisfied Resource requirements may change during execution New, urgent tasks can arise at any time
7 August 2002
116
Two approaches that tdon rk wo

Classical planners
input: current world state, goal state output: detailed action sequence to achieve goal state can find solutions to hard problems
Classical schedulers
input: set of actions to do and constraints on order/timing output: schedule specifying when to do each action can seek optimal solutions
Problems: (1) w (2) tolerant of uncertainty slo ; in
7 August 2002
117
Reactive Planners
Coping with time-pressure
Stored plan library Heuristic or single-rule plan refinement
Coping with uncertainty

Action decisions deferred until just before execution Integrated contingency handling
but not very good at discooptimal solutions ver ng or solvinhard, novel problems g
7 August 2002
118
Procedure Description Langua (PDL)

(procedure (index (hold-altitude using mcp)) (profile right-hand) (step s1 (clear right-hand)) (step s2 (find-loc alt-hold-button => ?loc)) (step s3 (press-button ?loc right-hand) (waitfor (empty right-hand) (location alt-hold-button ?loc))) (step end (terminate) (waitfor (illuminated alt-hold-button)) (step aux1 (restart ?self) (waitfor (resumed ?self))))
concurrency reactivity hierarchy contingency-handl
7 August 2002
119
Multitasking in Apex
Concurrency control Rational interruption and resumption Graceful interruption and resumption Efficient use of resources
7 August 2002
120
Concurrency Control: PDL idioms

Converge Race Synchronize
(procedure (procedure (procedure (index (do-it)) (index (do-it)) (index (do-it)) (step s1 (do-A) (step s1 (do-A)) (step s1 (do-A) (step s2 (do-B) (step s2 (do-B) (step s2 (do-B) (step s3 (do-C) (waitfor (started ?s1))) (step s3 (do-C) (waitfor ?s1) (step s3 (terminate) (waitfor ?s1 ?s2) (waitfor ?s2)) (waitfor ?s1 ?s2))) (step s4 (terminate) (step s4 (terminate) (waitfor ?s3))) (waitfor ?s3)))
Rational interruption and resum Determining if tasks confli

Profile clause declares resource requirements
(profile (<resource> [tolerance]) )
Some tasks tolerate brief interruptions Conflict exists between tasks A and B if
and A and B both require resource R, Expected Duration (A) > Tolerance (B) or Expected Duration (B) > Tolerance
(A)
7 August 2002
122
Rational interruption and resump Resolving task conflicts

Compute priority based on task & situational fac
urgency (U): measure of time until deadline importance (I): cost of missing deadline (time cost) subjective workload (S): measure of task crowding
Urgency dominates if time enough to do everythin Importance dominates if some deadlines cannot be met
7 August 2002
123
Rational interruption and resump Resolving task conflicts
Simple Priority = Sm*x - S)*U SaI + (

Highest priority task gets resources Other tasks aborted or delayed
Priority values set with priority clause

(priority <urgency> <importance>)
7 August 2002
124
Graceful interruption and resump Capabilities

Interruption tolerance Interruption suppression
(interrupt-cost <cost>)
Transition behaviors
Interrupt-time, suspension-time, resume-time Illustrates contingency-handling
7 August 2002
125
Graceful interruption and resump PDL idioms for transition behav

(procedure (index (fly-cruise-leg using manual-control)) (step s1 (maintain-altitude) (interrupt-cost 5)) ... (step s12 (handoff-to-pilot-not-flying) (priority (importance 10) (urgency 10))) (waitfor (interrupted ?self))) (step s13 (monitor-pilot-not-flying) (waitfor (completed ?s12))) (step s14 (request-role-pilot-flying) (waitfor (resumed ?self))) ...)
7 August 2002
126
Efficient use of resource

Combine redundant tasks
(merge <condition> [<task pattern>])
Online scheduling to exploit slack

Using slack time a scheduling problem Apex scheduling mechanisms Concurrent recursive decomposition => tasks Priority-based allocation => schedule Non-deliberative use of scheduler unusual!
7 August 2002
127
Application-driven language deve

There is no commitment to keep the language as-is evolving as Apex modelers needs become better un New syntax to simplify/abbreviate common patter
e.g. sequential procedures
Default behaviors to avoid pathological behavio

e.g. weak persistence tendency
New architecture functionality and PDL construc access it when needed behavior difficult to re
e.g. rank instead of priority
Document new idioms as invented

Summary
Multitasking ability founded on tactical knowle derived from general heuristics Reactive planners can be extended to execute th heuristics in uncertain/time-pressured enviro Specifically, extensions for concurrency contro interruption handling and resource management facilitate use of multitasking tactics Understanding of what needs to be represented a notation is best for this purpose are improvi Apex applications are developed
7 August 2002
129
Wrap-up Wrap-up
7 August 2002
130
CD ROM
The CD ROM contains Apex 2.2b4 plus additional simworlds documentation, and papers ,
FolderStudentModels in "Apex Worlds" folder contain models " " from Bonnie Johns class that have not all been tested wi version of Apex To run the student models you must modify the allworlds .lisp file by eliminating the commenting semicolons before the world yo to run The "kitchen world" world was developed for an earlier ve Apex and only partly runs, but it does show a more comple simworld "Apex reference manual" documents the system Documentation of Apex Builder and papers are in the "Apex folder
7 August 2002
131
Extensions folder
The extensions folder in your Apex directory i to be used for code that add to the system or modifications you make to existing functions
Is read last so functions you write will overwrite existi If you write code you think will enhance the system let u we will look into incorporating it
7 August 2002
132
Website
// nas . ftp:eos.arc. a gov/outgoing/apex/apex
Latest versions of Apex (Apex 2.2b4) Macintosh and PC Some of the worlds Documentation Instructions on downloading and running Patches We are trying to update it regularly to keep it cur
7 August 2002
133
Human/Model Comparisons: CPM-GOMS model built with Ap

1400 1200
1000
me (m TiTime (ms)s)
800
600
No-Interleaving Mode Fittss Law CPM-GOMS Model S1- Mean for Trials S2- Mean for Trials
400
200
Card Slotot Card Sl
Cash Sl Cash Slotot
Card Sl Card Slotot
ithdraw WWithdraw
7 August 2002
Cash Sl Cash Slotot
Checking Checking
Correc Correctt
OK OK
No No
4 4
9 9
0 0
8 8
0 0
134
Human/Model Comparisons: No Interweaving

1400 1200
1000
Time (ms)
800
600
Fittss Law No-Interleaving Mode CPM-GOMS Model S1- Mean for Trials S2- Mean for Trials
400
200
Card Slot
Cash Slot
Card Slot
Withdraw
7 August 2002
Cash Slot
Checking
Correct
OK
No
135
Human/Model Comparisons: Fitts s Law only

1400 1200
1000
Time (ms)
800
600
Fittss Law No-Interleaving Mode CPM-GOMS Model S1- Mean for Trials S2- Mean for Trials
400
200
Card Slot
Cash Slot
Card Slot
Withdraw
7 August 2002
Cash Slot
Checking
Correct
OK
No
136

Cogsci Tutorial02

Uploaded by

Copyright:

Available Formats

Cogsci Tutorial02

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cogsci Tutorial02

Uploaded by

Copyright:

Available Formats

Cognitive Science Tutorial

Cognitive Science 2002 Fairfax, VA

What this tutorial will

Applying Apex to a simple HCI example

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

Apex and CPM-GOMS

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

Complex dynamic environments

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

The Apex Architecture

Cognitive Science 2002 Fairfax, VA

Card-Moran-Newell GOMS (CMN-GOMS)

Cognitive Science 2002 Fairfax, VA

Text Edit World

Cognitive Science 2002 Fairfax, VA

The fox jumped over the lazy quick brown dog

We will model cutting and pasting from the edit menu

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

Select dit menu E

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

Position text cursor

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

KLM-GOMS Model: KLM-GOMS Model: Sequential execution in Ap Sequential execution in Ap

Cognitive Science 2002 Fairfax, VA

Keystroke Level Model (KLM

Main non-primitive PDL proce

:sequential used to is :sequential used to is

Cognitive Science 2002 Fairfax, VA

Top level PDL procedure

Double-click Double-click " ApexApp" ApexApp" "

Cognitive Science 2002 Fairfax, VA

Lisp Lisp Listener Listener

Sherpa Sherpa interface interface

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

Loading from erpa Sh

Cognitive Science 2002 Fairfax, VA

Running the KLM model

Step Step Reset Reset Run Run Pause Pause

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

Inspecting the trace

Cognitive Science 2002 Fairfax, VA

Generating a PERT chart

Cognitive Science 2002 Fairfax, VA

KLM PERT chart

Cognitive Science 2002 Fairfax, VA

Cognitive Science 2002 Fairfax, VA

Elements of a CMN-GOMS model

Cognitive Science 2002 Fairfax, VA

Simple Interactivity: UsingVARLIST

Interacting through varlist

Changing a value in varlist