research-article

Engineering Annotations: A Generic Framework for Gluing Design Artefacts of Interactive Systems

Authors:

Marco Winckler,

Philippe Palanque,

Olivier Nicolas,

Laurent GoncalvesAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 6, Issue EICS

Article No.: 174, Pages 1 - 36

https://doi.org/10.1145/3535063

Published: 17 June 2022 Publication History

Abstract

Along the design process of interactive system multiple intermediate artefacts (such as user interface prototypes, task models, dialog models?) are created, tested, revised, and improved until the development team produces a full-fledged system. However, relevant information for describing the design solution and/or supporting design decisions (such as rational about the design, decisions made, recommendations, etc.) are not explicitly captured in the models/artefacts, hence the need for annotations. Many approaches argue against information duplication to increase maintainability of the artefacts. Nonetheless, annotations created on one artefact are usually relevant to other artefacts/models. So that, there is a need for tools and techniques to coordinate annotations across artefacts/models which is the contribution of the present work. In this paper, we propose a model-based approach that was conceived to handle annotations in a systematic way along the development process of interactive systems. As part of the solution, we propose an annotation model built upon the W3C's Web Annotation Data Model. The feasibility of the approach is demonstrated by means of a tool suite featuring a plugin, which has been deployed and tested over the multi-artefacts. The overall approach is illustrated on the design of an interactive cockpit application performing two design iterations. The contribution brings two main benefits for interactive systems engineering: i) it presents a generic pattern for integrating information in multiple usually heterogenous artefacts throughout the design process of interactive systems; and ii) it highlights the need for tools helping to rationalize and to document the various artefacts and the related decisions made during interactive systems design.

References

[1]

ISO 9241--210 2008. Ergonomics of human system interaction-Part 210: Human-centered design for interactive systems. Standard. International Organization for Standardization, Geneva, CH.

[2]

Beaudouin-Lafon, M., Mackay, W. 2002. Prototyping tools and techniques. In The human-computer interaction handbook, Julie A. Jacko and Andrew Sears (Eds.). L. Erlbaum Associates Inc., Hillsdale, NJ, USA 1006--1031.

[3]

Marc Rettig. 1994. Prototyping for tiny fingers. Communications of the ACM 37, 4 (April 1994), 21--27.

Digital Library

[4]

Michael McCurdy, Christopher Connors, Guy Pyrzak, Bob Kanefsky, and Alonso Vera. 2006. Breaking the fidelity barrier: an examination of our current characterization of prototypes and an example of a mixed-fidelity success. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06), Rebecca Grinter, Thomas Rodden, Paul Aoki, Ed Cutrell, Robin Jeffries, and Gary Olson (Eds.). ACM, New York, NY, USA, 1233--1242. DOI=http://dx.doi.org/10.1145/1124772.1124959.

Digital Library

[5]

Youn-Kyung Lim, Erik Stolterman, and Josh Tenenberg. 2008. The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Trans. Comput.-Hum. Interact. 15, 2, Article 7 (July 2008), 27 pages.

Digital Library

[6]

Diaper, D. & Stanton, N. A. (eds.) The Handbook of Task Analysis for Human-Computer Interaction. Lawrence Erlbaum Associates, 2004. 650 pages.

[7]

Winckler, M., Vanderdonckt, J., Trindade, F., Stanciulescu, A. Cascading Dialog Modeling with UsiXML. International Workshop on the Design, Verification and Specification of Interactive Systems (DSVIS'2008). Kingston, Ontario, Canada, July 16--18 2008. Springer LNCS 5136. pp. 121--135.

[8]

William Buxton. 1983. Lexical and pragmatic considerations of input structures. SIGGRAPH Comput. Graph. 17, 1 (January 1983), 31--37.

Digital Library

[9]

A metamodel for the runtime architecture of an interactive system: the UIMS tool developers workshop. SIGCHI Bulletin, 24, 1 (Jan. 1992), 32--37.

Digital Library

[10]

Célia Martinie, David Navarre, Philippe A. Palanque, Camille Fayollas. A generic tool-supported framework for coupling task models and interactive applications. EICS 2015: 244--253.

[11]

Regina Bernhaupt, Philippe Palanque, Dimitri Drouet, Celia Martinie. Enriching Task Models with Usability and User Experience Evaluation Data. In: Proceedings of the 7th IFIP WG 13.2 International Working Conference, HCSE 2018, Sophia Antipolis, France, September 3--5, 2018, Revised Selected Papers. Springer, LNCS, volume 11262, Pages 146--163.

[12]

Eric Barboni, Jean-François Ladry, David Navarre, Philippe A. Palanque, Marco Winckler. Beyond modelling: an integrated environment supporting co-execution of tasks and systems models. EICS 2010: 165--174.

[13]

Maristella Agosti, Giorgetta Bonfiglio-Dosio, and Nicola Ferro. 2007. A historical and contemporary study on annotations to derive key features for systems design. Int. J. Digit. Libr. 8, 1 (October 2007), 1--19. DOI=http://dx.doi.org/10.1007/s00799-007-0010-0

Digital Library

[14]

Jean-Luc Hak, Marco Winckler, David Navarre. PANDA: prototyping using annotation and decision analysis. EICS 2016: 171--176

[15]

Marisela Gutierrez Lopez, Gustavo Rovelo, Mieke Haesen, Kris Luyten, Karin Coninx. Capturing Design Decision Rationale with Decision Cards. INTERACT (1) 2017: 463--482

[16]

Web Annotation Data Model - W3. Available at: https://www.w3.org/TR/annotation-model

[17]

Catherine C. Marshall. 1997. Annotation: from paper books to the digital library. In Proceedings of the second ACM international conference on Digital libraries (DL '97). ACM, New York, NY, USA, 131--140. DOI=10.1145/263690.263806 http://doi.acm.org/10.1145/263690.263806

Digital Library

[18]

Catherine C. Marshall. 1998. Toward an ecology of hypertext annotation. In Proceedings of the ninth ACM conference on Hypertext and hypermedia : links, objects, time and space--structure in hypermedia systems: links, objects, time and space--structure in hypermedia systems (HYPERTEXT '98). ACM, New York, NY, USA, 40--49. DOI=http://dx.doi.org/10.1145/276627.276632

Digital Library

[19]

Bringay S., Barry C., Charley J., Annotations: A new type of document in the Electronic Health Record. Paper presented at the 2nd International Conference on Document Research and Development in Sciences, arts and business: DOCAM 2004, University of California, Berkeley, Etats-Unis, octobre 2004.

[20]

Robert Sanderson, Paolo Ciccarese, Herbert Van de Sompel, 2013. Open Annotation Data Model, Community Draft. Available at: http://www.openannotation.org/spec/core/

[21]

Robert Sanderson, Paolo Ciccarese, Benjamin Young, 2015. Web Annotation Data Model, Working Draft.

[22]

José Kahan and Marja-Ritta Koivunen. 2001. Annotea: an open RDF infrastructure for shared Web annotations. In Proceedings of the 10th international conference on World Wide Web (WWW '01). ACM, New York, NY, USA, 623--632. DOI=http://dx.doi.org/10.1145/371920.372166

Digital Library

[23]

Agosti M. and Ferro N. 2007. A formal model of annotations of digital content. ACM Trans. Inform. Syst. 26, 1, Article 3 (November 2007), 57 pages. DOI = 10.1145/1292591.1292594 http://doi.acm.org/10.1145/1292591.1292594

Digital Library

[24]

Manuel Zacklad. Annotation : attention, association, contribution. Annotations dans les Documents pour l'Action, Hermes science publications, pp.29--46, 2007. sic 00180781

[25]

Gaëlle Lortal, Myriam Lewkowicz, Amalia Todirascu-Courtier. Modélisation de l'activité d'annotation discursive pour la conception d'un collecticiel support à l'herméneutique. IC - 16èmes Journées francophones d'Ingénierie des Connaissances, May 2005, Nice, France. Presses universitaires de Grenoble, pp.169--180, 2005. hal-01023979

[26]

Lortal G., Lewkowicz M., Todirascu-Courtier A., 2005, Annotation: Textual Media for Cooperation, in Proceedings of Annotation for Cooperation Workshop November 24--25th (p.41--50)

[27]

Amir M. Naghsh, Andy Dearden, and Mehmet B. Özcan. 2005. Investigating annotation in electronic paper-prototypes. In Proceedings of the 12th international conference on Interactive Systems: design, specification, and verification (DSVIS'05), Stephen W. Gilroy and Michael D. Harrison (Eds.). Springer-Verlag, Berlin, Heidelberg, 90--101. DOI=http://dx.doi.org/10.1007/11752707_8

Digital Library

[28]

Gutierrez, Marisela & Rovelo Ruiz, Gustavo & Luyten, Kris & Haesen, Mieke & Coninx, Karin. (2018). Re-thinking Traceability: A Prototype to Record and Revisit the Evolution of Design Artefacts. 196--208. 10.1145/3148330.3148334.

[29]

Silva, T. R., Hak, J-L., Winckler, M. Nicolas, O. A Comparative Study of Milestones for Featuring GUI Prototyping Tools. Journal of Software Engineering and Applications (JSEA), Vol.10 No.6, June 23, 2017, ISSN Online: 1945--3124, ISSN Print: 1945--3116, PP. 564--589.

[30]

S. Fish, Robert & Kraut, Robert & D. P. Leland, Mary. (1988). Quilt: A collaborative tool for cooperative writing. ACM SIGOIS Bulletin. 9. 30--37. 10.1145/966861.45414.

[31]

Müller-Birn C., Tina Klüwer, André Breitenfeld, Alexa Schlegel, and Lukas Benedix. 2015. neonion: Combining Human and Machine Intelligence. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing (CSCW'15 Companion). ACM, New York, NY, USA, 223--226.

Digital Library

[32]

Ciccarese P.,Marco Ocana,Tim Clark (2012) DOMEO: a web-based tool for semantic annotation of online documents. Bio-Ontologies 2011. http://bio-ontologies.knowledgeblog.org/297

[33]

Di Donato F.& Morbidoni, Christian & Fonda, Simone & Piccioli, Alessio & Grassi, Marco & Nucci, Michele. (2013). Semantic annotation with Pundit: A case study and a practical demonstration. 10.1145/2517978.2517995.

[34]

Elias M. and Bezerianos A. 2012. Annotating BI visualization dashboards: needs & challenges. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1641--1650.

Digital Library

[35]

Landay, J. A., & Myers, B. A. (1995, May). Interactive sketching for the early stages of user interface design. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 43--50). ACM Press/Addison-Wesley Publishing Co.

[36]

Li, C & Mcmahon, Chris & Newnes, Linda. (2009). Annotation in design processes: Classification of approaches. DS 58--8: Proceedings of ICED 09, the 17th International Conference on Engineering Design. 251--262.

[37]

Navarre D., Philippe A. Palanque, Rémi Bastide, Amélie Schyn, Marco Winckler, Luciana Porcher Nedel, Carla Maria Dal Sasso Freitas A Formal Description of Multimodal Interaction Techniques for Immersive Virtual Reality Applications. INTERACT 2005: 170--183.

[38]

Martinie C., Palanque P., Barboni E., Winckler M., Ragosta M., Pasquini A., and Lanzi P. 2011. Formal tasks and systems models as a tool for specifying and assessing automation de-signs. In Proceedings of the 1st International Conference on Application and Theory of Automation in Command and Con-trol Systems (ATACCS '11). IRIT Press, France, 50--59

[39]

Sy, O. & Bastide, Rémi & Palanque, Philippe & Le, D & Navarre, David. (2000). PetShop: a CASE tool for the Petri Net based specification and prototyping of CORBA systems. Petri Nets 2000. 78.

Digital Library

[40]

Creissac Campos J., Camille Fayollas, Michael D. Harrison, Célia Martinie, Paolo Masci, and Philippe Palanque. 2020. Supporting the Analysis of Safety Critical User Interfaces: An Exploration of Three Formal Tools. ACM Trans. Comput.-Hum. Interact. 27, 5, Article 35 (October 2020), 48 pages.

Digital Library

[41]

Martinie C., Palanque P., Winckler M. Structuring and Com-position Mechanisms to Address Scalability Issues in Task Models. IFIP TC13 Human Computer Interaction 2011 (INTERACT). p:134--152. Springer-Verlag.

[42]

Navarre D., Palanque P., Ladry J-F. & Barboni E. ICOs: A model-based user interface description technique dedicated to interactive systems addressing usability, reliability and scala-bility. ACM Trans. Comput.-Hum. Interact., 16(4), 18:1--18:56. 2009

Digital Library

[43]

Scapin D. L. & J. M. Christian Bastien (1997) Ergonomic criteria for evaluating the ergonomic quality of interactive systems, Behaviour & Information Technology, 16:4--5, 220--231.

[44]

Liao Y., Lezoche M., Panetto H., Boudjlida N. (2011) Semantic Annotation Model Definition for Systems Interoperability. In: Meersman R., Dillon T., Herrero P. (eds) On the Move to Meaningful Internet Systems: OTM 2011 Workshops. OTM 2011. Lecture Notes in Computer Science, vol 7046. Springer, Berlin, Heidelberg

[45]

Barboni E., Célia Martinie, David Navarre, Philippe A. Palanque, Marco Winckler. Bridging the gap between a behavioural formal description technique and a user interface description language: Enhancing ICO with a graphical user interface markup language. Sci. Comput. Program. 86: 3--29 (2014)

[46]

Danivska, V., & Appel-Meulenbroek, R. (Eds.). (2021). A Handbook of Management Theories and Models for Office Environments and Services (1st ed.). Routledge. https://doi.org/10.1201/9781003128786.

[47]

Shotton, David. (2010). CiTO, the Citation Typing Ontology. Journal of biomedical semantics. 1 Suppl 1. S6. 10.1186/2041--1480--1-S1-S6.

[48]

Paternò F. (2003) ConcurTaskTrees: An Engineered Notation for Task Model. The Handbook of Task Analysis for Human-Computer Interaction. pp. 483--503. Lawrence Erlbaum Associates.

[49]

Tankeu-Choitat A., D. Navarre, P. Palanque, Y. Deleris, J. Fabre and C. Fayollas, "Self-Checking Components for Dependable Interactive Cockpits Using Formal Description Techniques," 2011 IEEE 17th Pacific Rim International Symposium on Dependable Computing, 2011, pp. 164--173.

Digital Library

[50]

Barboni E., Jean-François Ladry, David Navarre, Philippe Palanque, and Marco Winckler. 2010. Beyond modelling: an integrated environment supporting co-execution of tasks and systems models. In Proceedings of the 2nd ACM SIGCHI symposium on Engineering interactive computing systems (EICS '10). Association for Computing Machinery, New York, NY, USA, 165--174.

Digital Library

[51]

Langer P., Konrad Wieland, Manuel Wimmer, Jordi Cabot: From UML Profiles to EMF Profiles and Beyond. TOOLS (49) 2011: 52--67.

[52]

Bouzekri E., A. Canny, C. Fayollas, C. Martinie, P. Palanque, E. Barboni, Y. Deleris, C. Gris. Engineering issues related to the development of a recommender system in a critical context: Application to interactive cockpits, International Journal of Human-Computer Studies, Volume 121, 2019, Pages 122--141, ISSN 1071--5819, https://doi.org/10.1016/j.ijhcs.2018.05.001.

[53]

Bouzekri, E., Martinie, C., Palanque, P., Atwood, K., Gris, C. (2021). Should I Add Recommendations to My Warning System? The RCRAFT Framework Can Answer This and Other Questions About Supporting the Assessment of Automation Designs. In:, et al. Human-Computer Interaction -- INTERACT 2021. INTERACT 2021. Lecture Notes in Computer Science(), vol 12935. Springer, https://doi.org/10.1007/978--3-030--85610--6_24

[54]

Hamon A., Philippe Palanque, José Luís Silva, Yannick Deleris, and Eric Barboni. 2013. Formal description of multi-touch interactions. In Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems (EICS '13). Association for Computing Machinery, New York, NY, USA, 207--216. https://doi.org/10.1145/2494603.2480311

Digital Library

[55]

Fayollas C., Jean-Charles Fabre, Philippe Palanque, Éric Barboni, David Navarre, and Yannick Deleris. 2013. Interactive cockpits as critical applications: a model-based and a fault-tolerant approach. Int. J. Crit. Comput.-Based Syst. 4, 3 (December 2013), 202--226. https://doi.org/10.1504/IJCCBS.2013.058407

Digital Library

[56]

ARINC 661--5, Prepared by Airlines Electronic Engineering Committee. Cockpit Display System Interfaces to User Systems. ARINC Specification 661--5; (2013)

[57]

Palanque, P., Cockburn, A., Désert-Legendre, L., Gutwin, C., Deleris, Y. (2019). Brace Touch: A Dependable, Turbulence-Tolerant, Multi-touch Interaction Technique for Interactive Cockpits. In: Romanovsky, A., Troubitsyna, E., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2019. Lecture Notes in Computer Science(), vol 11698. Springer, Cham. https://doi.org/10.1007/978--3-030--26601--1_4

[58]

Canny A. et al., "Divide to Conquer: Functional Decomposition to Support Model-Based Engineering of Command and Control of Cyber-Physical Systems," 2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), 2019, pp. 694--701.

[59]

Gregory J., Lucy Berthoud, Theo Tryfonas, Alain Rossignol, and Ludovic Faure. 2020. The long and winding road: MBSE adoption for functional avionics of spacecraft. J. Syst. Softw. 160, C (Feb 2020). https://doi.org/10.1016/j.jss.2019.110453

Digital Library

[60]

Martinie C., Philippe Palanque, Elodie Bouzekri, Andy Cockburn, Alexandre Canny, and Eric Barboni. 2019. Analysing and Demonstrating Tool-Supported Customizable Task Notations. Proc. ACM Hum.-Comput. Interact. 3, EICS, Article 12 (June 2019), 26 pages. https://doi.org/10.1145/3331154

Digital Library

[61]

Martinie C., Philippe Palanque, Marco Winckler, and Stéphane Conversy. 2010. DREAMER: a design rationale environment for argumentation, modeling and engineering requirements. In Proceedings of the 28th ACM International Conference on Design of Communication (SIGDOC '10). Association for Computing Machinery, New York, NY, USA, 73--80. https://doi.org/10.1145/1878450.1878463

Digital Library

[62]

Fayollas C., Célia Martinie, David Navarre, and Philippe Palanque. 2016. Engineering mixed-criticality interactive applications. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16). Association for Computing Machinery, New York, NY, USA, 108--119. https://doi.org/10.1145/2933242.2933258

Digital Library

[63]

ARP 4754A, Guidelines for Development of Civil Aircraft and Systems, published by SAE International, 2010.

[64]

Rocha Silva T., Jean-Luc Hak, Marco Winckler. Testing Prototypes and Final User Interfaces Through an Ontological Perspective for Behavior-Driven Development. HCSE/HESSD 2016: 86--107.

[65]

Rocha Silva T., Jean-Luc Hak, Marco Winckler. A Behavior-Based Ontology for Supporting Automated Assessment of Interactive Systems. 11th IEEE International Conference on Semantic Computing, ICSC 2017, San Diego, CA, USA, January 30 - February 1, 2017. IEEE Computer Society 2017, ISBN 978--1--5090--4284--5: 250--257.

[66]

DO-333 Formal Methods Supplement to DO-178C and DO-278A, published by RTCA and EUROCAE December 13, 2011.

[67]

DO-178C / ED-12C, Software Considerations in Airborne Systems and Equipment Certification, published by RTCA and EUROCAE, 2012

[68]

Snyder J. 2017. Vernacular Visualization Practices in a Citizen Science Project. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). Association for Computing Machinery, New York, NY, USA, 2097--2111. https://doi.org/10.1145/2998181.2998239

Digital Library

Cited By

Quere CMenin AJulien RWu HWinckler M(2024)HandyNotes: using the hands to create semantic representations of contextually aware real-world objects2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00049(265-275)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VR58804.2024.00049

Index Terms

Engineering Annotations: A Generic Framework for Gluing Design Artefacts of Interactive Systems
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

A Functional Classification of Text Annotations for Engineering Design
Abstract
Describing and supplementing geometric shapes (parts) and layouts (assemblies) with relevant information is key for successful product design communication. 3D annotation tools are widely available in commercial systems, but they are ...
Highlights
- A classification is proposed for shifting traditional annotation practices towards the Model-Based Definition paradigm.
Experiential role of artefacts in cooperative design
C&T '09: Proceedings of the fourth international conference on Communities and technologies

The role of material artefacts in supporting distributed and co-located work practices has been well acknowledged within HCI and CSCW research. In this paper, we show that in addition to their ecological, coordinative and organizational support, ...
The LEMO annotation framework: weaving multimedia annotations with the web

Cultural institutions and museums have realized that annotations contribute valuable metadata for search and retrieval, which in turn can increase the visibility of the digital items they expose via their digital library systems. By exploiting ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 6, Issue EICS

EICS

June 2022

736 pages

EISSN:2573-0142

DOI:10.1145/3544787

Editor:
Jeff Nichols
Apple Inc., United States

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 June 2022

Published in PACMHCI Volume 6, Issue EICS

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
122
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)2

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Quere CMenin AJulien RWu HWinckler M(2024)HandyNotes: using the hands to create semantic representations of contextually aware real-world objects2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00049(265-275)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VR58804.2024.00049

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents