research-article

A usability scale for handheld augmented reality

Authors:

Marc Ericson C. Santos,

Takafumi Taketomi,

Christian Sandor,

Goshiro Yamamoto,

Hirokazu KatoAuthors Info & Claims

VRST '14: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology

Pages 167 - 176

https://doi.org/10.1145/2671015.2671019

Published: 11 November 2014 Publication History

Abstract

Handheld augmented reality (HAR) applications must be carefully designed and improved based on user feedback to sustain commercial use. However, no standard questionnaire considers perceptual and ergonomic issues found in HAR. We address this issue by creating a HAR Usability Scale (HARUS).

To create HARUS, we performed a systematic literature review to enumerate user-reported issues in HAR applications. Based on these issues, we created a questionnaire measuring manipulability -- the ease of handling the HAR system, and comprehensibility -- the ease of understanding the information presented by HAR. We then provide evidences of validity and reliability of the HARUS questionnaire by applying it to three experiments. The results show that HARUS consistently correlates with other subjective and objective measures of usability, thereby supporting its concurrent validity. Moreover, HARUS obtained a good Cronbach's alpha in all three experiments, thereby demonstrating internally consistency.

HARUS, as well as its decomposition into individual manipulability and comprehensibility scores, are evaluation tools that researchers and professionals can use to analyze their HAR applications. By providing such a tool, they can gain quality feedback from users to improve their HAR applications towards commercial success.

References

[1]

Bangor, A., Kortum, P. T., and Miller, J. T. 2008. An empirical evaluation of the system usability scale. Intl. Journal of Human--Computer Interaction 24, 6, 574--594.

[2]

Cronbach, L. J., and Meehl, P. E. 1955. Construct validity in psychological tests. Psychological bulletin 52, 4, 281.

[3]

Dey, A., and Sandor, C. 2014. Lessons learned: Evaluating visualizations for occluded objects in handheld augmented reality. International Journal of Human-Computer Studies 72, 1011, 704--716.

Digital Library

[4]

Dünser, A., Grasset, R., and Billinghurst, M. 2008. A survey of evaluation techniques used in augmented reality studies. Human Interface Technology Laboratory New Zealand.

[5]

Dünser, A., Billinghurst, M., Wen, J., Lehtinen, V., and Nurminen, A. 2012. Exploring the use of handheld ar for outdoor navigation. Computers & Graphics 36, 8, 1084--1095. Graphics Interaction Virtual Environments and Applications 2012.

Digital Library

[6]

Fowler, F. J., and Cosenza, C. 2008. International Handbook of Survey Methodology. Taylor & Francis, ch. Writing Effective Questions, 136--159.

[7]

Gabbard, J., and Swan, J. 2008. Usability engineering for augmented reality: Employing user-based studies to inform design. Visualization and Computer Graphics, IEEE Transactions on 14, 3 (May), 513--525.

Digital Library

[8]

Gervautz, M., and Schmalstieg, D. 2012. Anywhere interfaces using handheld augmented reality. Computer 45, 7 (July), 26--31.

Digital Library

[9]

Hart, S. G. 2006. Nasa-task load index (nasa-tlx); 20 years later. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, Sage Publications, 904--908.

[10]

Hix, D., Gabbard, J., Swan, J. E., I., Livingston, M., Hollerer, T., Julier, S., Baillot, Y., and Brown, D. 2004. A cost-effective usability evaluation progression for novel interactive systems. In System Sciences, 2004. Proceedings of the 37th Annual Hawaii International Conference on, 10 pp.--.

Digital Library

[11]

Huang, W., Huang, W., Diefes-Dux, H., and Imbrie, P. K. 2006. A preliminary validation of attention, relevance, confidence and satisfaction model-based instructional material motivational survey in a computer-based tutorial setting. British Journal of Educational Technology 37, 2, 243--259.

[12]

Kamarainen, A. M., Metcalf, S., Grotzer, T., Browne, A., Mazzuca, D., Tutwiler, M. S., and Dede, C. 2013. Ecomobile: Integrating augmented reality and probeware with environmental education field trips. Computers & Education 68, 0, 545--556.

Digital Library

[13]

Krosnick, J. A., and Presser, S. 2010. Handbook of Survey Research. Emerald Group Publishing Limited, ch. Question and Questionnaire Design, 263--313.

[14]

Kruijff, E., Swan, J., and Feiner, S. 2010. Perceptual issues in augmented reality revisited. In Mixed and Augmented Reality (ISMAR), 2010 9th IEEE International Symposium on, 3--12.

[15]

Kurkovsky, S., Koshy, R., Novak, V., and Szul, P. 2012. Current issues in handheld augmented reality. In Communications and Information Technology (ICCIT), 2012 International Conference on, 68--72.

[16]

Langlotz, T., Mooslechner, S., Zollmann, S., Degendorfer, C., Reitmayr, G., and Schmalstieg, D. 2012. Sketching up the world: in situ authoring for mobile augmented reality. Personal and Ubiquitous Computing 16, 6, 623--630.

Digital Library

[17]

Lee, G. A., Dunser, A., Kim, S., and Billinghurst, M. 2012. Cityviewar: A mobile outdoor ar application for city visualization. In Mixed and Augmented Reality (ISMAR-AMH), 2012 IEEE International Symposium on, 57--64.

[18]

Lewis, J. R., and Sauro, J. 2009. The factor structure of the system usability scale. In Human Centered Design. Springer, 94--103.

Digital Library

[19]

Messick, S. 1990. Validity of test interpretation and use. Tech. rep., Educational Testing Service.

[20]

Messick, S. 1995. Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist 50, 9, 741.

[21]

Mulloni, A., Seichter, H., and Schmalstieg, D. 2011. User experiences with augmented reality aided navigation on phones. In Mixed and Augmented Reality (ISMAR), 2011 10th IEEE International Symposium on, 229--230.

Digital Library

[22]

Mulloni, A., Seichter, H., and Schmalstieg, D. 2011. Handheld augmented reality indoor navigation with activity-based instructions. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, ACM, New York, NY, USA, MobileHCI '11, 211--220.

Digital Library

[23]

Nielsen, J. 1994. Usability Engineering. Elsevier.

[24]

Olsson, T., and Salo, M. 2011. Online user survey on current mobile augmented reality applications. In Mixed and Augmented Reality (ISMAR), 2011 10th IEEE International Symposium on, 75--84.

Digital Library

[25]

Pilke, E. M. 2004. Flow experiences in information technology use. International journal of human-computer studies 61, 3, 347--357.

Digital Library

[26]

Radhakrishna, R. B. 2007. Tips for developing and testing questionnaires/instruments. Journal of Extension 45, 1, 1--4.

[27]

Rekimoto, J. 1997. A magnifying glass approach to augmented reality systems. Presence 6, 4, 399--412.

Digital Library

[28]

Ryu, Y. S., and Smith-Jackson, T. L. 2006. Reliability and validity of the mobile phone usability questionnaire (mpuq). Journal of Usability Studies 2, 1, 39--53.

Digital Library

[29]

Santos, M., Chen, A., Taketomi, T., Yamamoto, G., Miyazaki, J., and Kato, H. 2014. Augmented reality learning experiences: Survey of prototype design and evaluation. Learning Technologies, IEEE Transactions on 7, 1 (Jan), 38--56.

[30]

Schall, G., Mendez, E., Kruijff, E., Veas, E., Junghanns, S., Reitinger, B., and Schmalstieg, D. 2009. Handheld augmented reality for underground infrastructure visualization. Personal and Ubiquitous Computing 13, 4, 281--291.

Digital Library

[31]

Swan, J., and Gabbard, J. L., 2003. Perceptual and ergonomic issues in mobile augmented reality for urban operations.

[32]

Veas, E., and Kruijff, E. 2008. Vesp'r: design and evaluation of a handheld ar device. In Mixed and Augmented Reality, 2008. ISMAR 2008. 7th IEEE/ACM International Symposium on, 43--52.

Digital Library

[33]

Veas, E. E., and Kruijff, E. 2010. Handheld devices for mobile augmented reality. In Proceedings of the 9th International Conference on Mobile and Ubiquitous Multimedia, ACM, 3.

Digital Library

[34]

Veas, E., Grasset, R., Ferencik, I., Grnewald, T., and Schmalstieg, D. 2013. Mobile augmented reality for environmental monitoring. Personal and Ubiquitous Computing 17, 7, 1515--1531.

Digital Library

[35]

White, S., and Feiner, S. 2009. Sitelens: Situated visualization techniques for urban site visits. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, CHI '09, 1117--1120.

Digital Library

Cited By

Basumatary DMaity R(2024) Design, development and testing of BodoRao —A markerless augmented reality application for endangered language in primary education Journal of Computer Assisted Learning10.1111/jcal.13056Online publication date: 20-Aug-2024
https://doi.org/10.1111/jcal.13056
Pott KAgotai D(2024)Spatial Computing Through an HCI Lens - UX Evaluation Based on SituatednessHCI International 2024 Posters10.1007/978-3-031-61950-2_12(102-113)Online publication date: 7-Jun-2024
https://doi.org/10.1007/978-3-031-61950-2_12
Graser SSchrepp MBöhm S(2024)UXAR-CT – An Approach for Measuring UX for Mobile Augmented Reality Applications in Corporate TrainingHuman-Centered Design, Operation and Evaluation of Mobile Communications10.1007/978-3-031-60458-4_15(211-234)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-60458-4_15
Show More Cited By

Index Terms

A usability scale for handheld augmented reality
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Accessibility
  2. Human computer interaction (HCI)

Recommendations

A usability study of multimodal input in an augmented reality environment

In this paper, we describe a user study evaluating the usability of an augmented reality (AR) multimodal interface (MMI). We have developed an AR MMI that combines free-hand gesture and speech input in a natural way using a multimodal fusion ...
Investigating the balance between virtuality and reality in mobile mixed reality UI design: user perception of an augmented city
NordiCHI '14: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational

Examples of mixed reality mobile applications and research combining virtual and real world data in the same view have emerged during recent years. However, currently there is little knowledge of users' perceptions comparing the role of virtual and real ...
Study on User-Centered Usability Elements of User Interface Designs in an Augmented Reality Environment
Virtual, Augmented and Mixed Reality. Design and Interaction
Abstract
In order to complete the augmented reality (AR) user interface (UI) design simply and quickly, the usable factors were studied in this work. The main focus of interface design is to increase usability. Various factors should be considered together ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '14: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology

November 2014

238 pages

ISBN:9781450332538

DOI:10.1145/2671015

Conference Chairs:
Rynson Lau
City University of Hong Kong
,
Dinesh Manocha
University of North Carolina, Chapel Hill
,
Program Chairs:
Taku Komura
University of Edinburgh
,
Aditi Majumder
University of California, Irvine
,
Weiwei Xu
Hangzhou Normal University

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques
Disney Research: Disney Research
NVIDIA
SIGCHI: ACM Special Interest Group on Computer-Human Interaction
UEDINI: Division of Informatics, University of Edinburgh
SICSA: The Scottish Informatics and Computer Science Alliance
Presence: Presence: Teleoperators and Virtual Environments

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

VRST '14

Sponsor:

SIGGRAPH
Disney Research
SIGCHI
UEDINI
SICSA
Presence

VRST '14: The 20th ACM Symposium on Virtual Reality Software and Technology

November 11 - 13, 2014

Edinburgh, Scotland

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Upcoming Conference

VRST '24

Sponsor:
sigchi
sigchi

30th ACM Symposium on Virtual Reality Software and Technology

October 9 - 11, 2024

Trier , Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

62
Total Citations
View Citations
1,121
Total Downloads

Downloads (Last 12 months)140
Downloads (Last 6 weeks)10

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Basumatary DMaity R(2024) Design, development and testing of BodoRao —A markerless augmented reality application for endangered language in primary education Journal of Computer Assisted Learning10.1111/jcal.13056Online publication date: 20-Aug-2024
https://doi.org/10.1111/jcal.13056
Pott KAgotai D(2024)Spatial Computing Through an HCI Lens - UX Evaluation Based on SituatednessHCI International 2024 Posters10.1007/978-3-031-61950-2_12(102-113)Online publication date: 7-Jun-2024
https://doi.org/10.1007/978-3-031-61950-2_12
Graser SSchrepp MBöhm S(2024)UXAR-CT – An Approach for Measuring UX for Mobile Augmented Reality Applications in Corporate TrainingHuman-Centered Design, Operation and Evaluation of Mobile Communications10.1007/978-3-031-60458-4_15(211-234)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-60458-4_15
Pinheiro CFigueiredo JPereira TSantos C(2024)Design and Usability Assessment of Multimodal Augmented Reality System for Gait TrainingRobot 2023: Sixth Iberian Robotics Conference10.1007/978-3-031-59167-9_36(437-449)Online publication date: 27-Apr-2024
https://doi.org/10.1007/978-3-031-59167-9_36
Díaz MÁlvarez-Gallego CCaro IPortela J(2023)Incorporating Augmented Reality Tools into an Educational Pilot Plant of Chemical EngineeringEducation Sciences10.3390/educsci1301008413:1(84)Online publication date: 12-Jan-2023
https://doi.org/10.3390/educsci13010084
Weiss HStirling L(2023)Usability Evaluation of an Augmented Reality Sensorimotor Assessment Tool for Astronauts2023 IEEE Aerospace Conference10.1109/AERO55745.2023.10115780(1-13)Online publication date: 4-Mar-2023
https://doi.org/10.1109/AERO55745.2023.10115780
Chen XLi SLi GXue BLiu BFang YSeo JKim IKim J(2023)Effects of building information modeling prior knowledge on applying virtual reality in construction education: lessons from a comparison studyJournal of Computational Design and Engineering10.1093/jcde/qwad09110:5(2036-2048)Online publication date: 12-Oct-2023
https://doi.org/10.1093/jcde/qwad091
Kaur DMantri A(2023)Augmented reality based interactive table-top environment for real-time visualization of control theory concepts: An empirical studyEducation and Information Technologies10.1007/s10639-023-12050-729:5(5309-5330)Online publication date: 17-Jul-2023
https://doi.org/10.1007/s10639-023-12050-7
De Luca VBarba MD’Errico GNuzzo BDe Paolis L(2023)A user experience analysis for a mobile Mixed Reality application for cultural heritageVirtual Reality10.1007/s10055-023-00840-w27:4(2821-2837)Online publication date: 8-Aug-2023
https://doi.org/10.1007/s10055-023-00840-w
Tsimerman EFratkina P(2023)An EON-XR Augmented Reality Application for Motivation Stimulation and Vocabulary Training of Master’s Degree Students Majoring in Civil EngineeringThe World of Games: Technologies for Experimenting, Thinking, Learning10.1007/978-3-031-48016-4_3(35-44)Online publication date: 18-Nov-2023
https://doi.org/10.1007/978-3-031-48016-4_3
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents