Model-driven engineering of multimodal user interfaces

Model-Driven Engineering of User Interfaces for Multimodal Web Applications Jean Vanderdonckt Université catholique de Louvain (UCL) Louvain School of Management (LSM) Information Systems Unit (ISYS) Belgian Laboratory of Computer-Human Interaction (BCHI) http://www.isys.ucl.ac.be/bchi

What is a multimodal user interface? Some terminology: notion of mode We have 5 Modes corresponding to our 5 human senses The human body has five major senses which operate to gather information from the world around us, sight, hearing, smell, taste, and touch. Any stimulus to one of the sense areas is detected by sensory nerves and is sent to the brain for interpretation.

What is a multimodal user interface? Combining several modes simultaneously: some are very hard! [Tison,2003]

What is a multimodal user interface? Some terminology: notion of modality Definition: a particular form of a mode Example: for auditive mode, we have: speech, music, voice, chorals, noise User interface for each mode thus involves one or many modalities A system is said to be Monomodal iff only one modality is involved Multimodal iff many modalities are involves Some modalities are still in their infancy: Olfactive user interfaces Aromatic user interfaces [Bodnar,2004]

What is a multimodal user interface? Some terminology: notion of media Definition: is a hardware/software device allowing interaction between a user and a system according to a given set of modalities Example: for auditive mode, we have several media possible: speech, music, voice, chorals, noise User interface for each mode thus involves one or many modalities [Bellik,2002]

Why are multimodal user interfaces important? Graphical user interfaces – classical human-computer interface paradigm Users with limited literacy and typing skills Handheld devices Need for hands and eyes free Difficult to support mouse and keyboard interaction Web access anyplace, anywhere, anytime Need for newer, better interface paradigm Multimodal user interface is a possible answer… Adaptation to the context of use (user, platform, environment) More natural and efficient human-computer interaction [IntuiLab,2006]

Why are multimodal user interfaces challenging? For Web Engineering Since presentation and navigation are no longer visual only, how can we design vocal, tactile and multimodal presentation and navigation within a page or across pages? For Model-Driven Approach So far, little research has been devoted to modelling the various aspects involved in multimodal web applications. Therefore, there is a need for identifying the appropriate implementation characteristics which will be the turned into high level design options to be supported by model-to-model and model-to-code transformations For Human-Computer Interaction: Developing a user interface for any combination of graphical, vocal, and tactile interaction modalities remains a challenge because of synchronization issues in time and space For Usability Engineering It is not because we are able to develop a user interface with any combination of any modalities that the resulting user interface is usable or tractable for a end user. We know little about the usability guidelines to be respected in this area

Concepts: The CARE Properties From Amodeus European Project (1995!) Complementarity Assignment Redundancy Equivalence Put that there [Bolt,1979]

What are the appropriate abstractions? Design space that supports effectively and efficiently the progress of the development life cycle towards a final system while assuring some form of quality Design space consists of an n-dimensional space where each dimension is denoted by a single design option Each design option has a finite set of design values E.g., label location (left-aligned, centered, right-aligned)

Why a design space? Advantages: Clarifies the development process in terms of options Facilitates the incorporation in the development life cycle of design options as an abstraction covered by a software Supports the tractability of more complex design problems Shortcomings: In theory: design options could be very numerous (even infinite) In practice: impossible to consider a large amount of design options Too complex or expensive to implement Design options do not necessarily address users’ needs and requirements Out of the designer’s scope of understanding, imagination or background Design decision is not always clear (may violate some usability principles or guidelines)

Multimodal Web Applications The languages in which they are implemented (e.g., XHTML, VoiceXML, X+V): Restrict the amount of possible UIs to obtain Set directly the CARE properties to Assignment, Equivalence and Redundancy Limited to graphic, voice, and tactile modalities [W3C,2006]

Solution space Cameleon reference framework UsiXML language: Graphical interaction Vocal interaction Multimodal interaction Design option-based transformational approach: Step 1: Construct the Task & Domain Models Step 2: From Task & Domain to Abstract User Interface (AUI) Step 3: From AUI to Concrete User Interface (CUI) Step 4: From CUI to FUI [Stanciulescu et al.,2005]

Design options for graphical web UIs Design options for graphical UIs Sub-task presentation Sub-task navigation Concretization of navigation & control Navigation concretization Control concretization

Design options for graphical web UIs Sub-task presentation: specifies the appearance of each sub-task in the final user interface. Sub-task presentation separated combined one by one combined box tabbed dialog box float window many at once float window all in one areas with separators group boxes bulleted list numbered list Abstract level Concrete level

Sub-task presentation – example Separated sub-task presentation Combined sub-task presentation T11 T12 T13 T1 AC11 AC12 AC13 AC1 T11 T12 T13 T1 AC11 AC12 AC13 NAC LHS RHS NAC LHS RHS

Design options for graphical web UIs Navigation concretization: identifies the placement and the cardinality of the objects/logically grouped set of objects that ensure the navigation. Navigation concretization [Stanciulescu,2006] placement cardinality local global simple multiple

Sub-task navigation - example Local placement for navigation objects Global placement for navigation object AC1 T11 T12 T13 T1 AC11 AC12 AC13 AIC111 AIC121 AIC131 AC1 T11 T12 T13 T1 AC11 AC12 AC13 AIC11 NAC LHS RHS NAC LHS RHS

Design options for graphical web UIs Sub-task navigation : specifies the way in which the dialog control is transferred from one sub-task to another. Sub-task navigation asynchronous sequential (synchronous)

Design options for graphical web UIs Concretization of navigation and control : specifies if the navigation and control are ensured or not by the same object. Concretization of navigation & control separated combined

Design options for graphical web UIs Control concretization : specifies the placement and the cardinality of the objects that ensure the control. Control concretizations placement cardinality local global multiple simple

Design options for multimodal web UIs Consider the ergonomic criteria for the evaluation of human-computer interfaces Compatibility, consistency, work load, adaptation, dialog control, guidance, error management Associate a corresponding CARE property to each design value. Design options for multimodal UIs [Scapin & Bastien,1997] [Vanderdonckt,1995] Prompt Grouping / Distinction of items Immediate feedback Guidance

Design options for multimodal web UIs Prompt : refers to the interaction channels available in order to lead the users to take specific actions whether it be data entry or other tasks. vocal (A) graphical (A) multimodal (E, C, R)

Design options for multimodal web UIs Grouping/Distinction of items : concerns the organization of information in relation to one another. Input : any information input from the user to the system vocal (A) graphical (A) multimodal (E, C, R) Output : any information output from the system to the user vocal (A) graphical (A) multimodal (E, C, R)

Design options for multimodal web UIs Immediate feedback : concerns the system responses to users’ action vocal (A) graphical (A) multimodal (E, C, R)

Design options for multimodal web UIs Guidance : refers to the means available to advise, orient, inform, instruct and guide the users throughout their interactions with the system Guidance for input: any guidance offered to the user in order to guide him with the input Textual (A), Iconic (A), Acoustic (A), Speech (A) Multimodal (E, C, R) Guidance for output: any guidance offered to the user in order to guide him with the feedback Textual (A), Iconic (A), Acoustic (A), Speech (A) Multimodal (E, C, R)

Multimodal example Multimodal text field Prompt: multimodal (R) Grouping for input: multimodal (E) Immediate feedback: multimodal (R) Guidance: Input: iconic (A) Output: iconic (A) [Stanciulescu,2006] Prompt: multimodal Please say your name (vocal + graphical) 1 Input: multimodal Juan Gonzalez 2 (vocal + graphical) Juan Gonzalez Guidance for input: iconic Guidance for feedback: iconic Feedback:multimodal Your answer was: Juan Gonzalez 3 (graphical + vocal)

Vocal Concrete Interaction Objects VocalContainers: VocalGroup VocalForm VocalMenu VocalConfirmation VocalIndividualComponents: VocalOutput: VocalFeedback VocalPrompt VocalMenuItem VocalInput VocalNavigation Break Exit EventTypes: error, help, noInput, noMatch

Vocal Concrete Interaction Objects Example: virtual ice cream shop

Multimodal Concrete Interaction Objects MultimodalCIO: combination of graphical CIO and vocalCIO outputText + inputText vocalPrompt + vocalInput outputText + inputText + vocalInput + vocalFeedback + guidance groupBox + radioButtons vocalMenu + vocalItems + vocalInput groupBox + radioButtons + vocalMenu + vocalItems + vocalInput + guidance Graphical Vocal Multimodal

Concrete UI Relationships GraphicalRelationships GraphicalTransition GraphicalAdjacency GraphicalContainment VocalRelationships VocalTransition VocalAdjacency VocalContainment Synchronization: between vocalCIOs and graphicalCIOs

Tool support MultimodaliXML – five software modules: IdealXML TransformiXML GrafiXML CFB Generator XHTML+Voice Generator tool

Multimodal Concrete Interaction Objects Syntax example: multimodal text field with equivalent input and redundant output

Specification of transformation Graph-based patterns Syntactically typed patterns Textual concrete syntax of patterns: UsiXML syntax

How transformation rules work… LHS: Find an occurrence of LHS into G NAC: Check that NAC does not match into G. If there is a match then skip to another occurrence of LHS RHS: Replace LHS by RHS [Limbourg,2004] Transformation rule Initial Model Resulting Model

UsiXML syntax of transformation rules NAC NAC RHS mappings

Transformation rule NAC: LHS: RHS: Multimodal prompt Guidance for input Multimodal input Guidance for output Vocal feedback

Case studies 2 case studies: Virtual Polling system and Car rental system

Conclusion Model-driven Engineering of Multimodal Web applications is possible Requirements fulfillment: Support for multimodal input/output CARE properties support for input/output modalities Approach based on design space composed of 13 design options Design space covers a subset of the total number of UIs which can be developed with X+V Methodology covers a subset of the UIs which can be developed with X+V

Conclusion Advantages of combining design options into a design space: Vertues of a design space Descriptive Generative Comparative [Stanciulescu & Vanderdonckt,2006] [Beaudouin-Lafon,2004]

Future work Investigate the implementation of a new transformation engine: any suggestion? Reinforce existing vocal components Extend the ontology with new vocal components Design space improvement Extend transformation catalogs Adaptation of multimodal web user interface to the context of use

References See videos and papers on MultimodaliXML at: http://www.usixml.org/index.php?view=page&idpage=51 Stanciulescu, A., Vanderdonckt, J., Design Options for Multimodal Web Applications, Proc. of 6th Int. Conf. on Computer-Aided Design of User Interfaces CADUI’2006 (Bucharest, 6-8 June 2006), Chapter 4, Springer-Verlag, Berlin, 2006, pp. 41-56 Stanciulescu, A., Limbourg, Q., Vanderdonckt, J., Michotte, B., Montero, F., A Transformational Approach for Multimodal Web User Interfaces based on UsiXML, Proc. of 7th ACM Conf. on Multimodal Interfaces ICMI’2005 (Trento, 4-6 October, 2005), ACM Press, New York, 2005, pp. 259-266

Thank you very much for your attention For more information and downloading, http://www.isys.ucl.ac.be/bchi http://www.usixml.org User Interface eXtensible Markup Language http://www.similar.cc European network on Multimodal UIs Special thanks to all members of the team!

Model-driven engineering of multimodal user interfaces

More Related Content

What's hot

What's hot (19)

Similar to Model-driven engineering of multimodal user interfaces

Similar to Model-driven engineering of multimodal user interfaces (20)

More from Jean Vanderdonckt

More from Jean Vanderdonckt (20)

Recently uploaded

Recently uploaded (20)

Model-driven engineering of multimodal user interfaces