research-article

A mixed reality approach for interactively blending dynamic models with corresponding physical phenomena

Authors:

Samsun Lampotang,

Benjamin LokAuthors Info & Claims

ACM Transactions on Modeling and Computer Simulation (TOMACS), Volume 20, Issue 4

Article No.: 22, Pages 1 - 23

https://doi.org/10.1145/1842722.1842727

Published: 05 November 2010 Publication History

Abstract

The design, visualization, manipulation, and implementation of models for computer simulation are key parts of the discipline. Models are constructed as a means to understand physical phenomena as state changes occur over time. One issue that arises is the need to correlate models and their components with the phenomena being modeled. For example, a part of an automotive engine needs to be placed into cognitive context with the diagrammatic icon that represents that part's function. A typical solution to this problem is to display a dynamic model of the engine in one window and the engine's CAD model in another. Users are expected to, on their own, mentally blend the dynamic model and the physical phenomenon into the same context. However, this contextualization is not trivial in many applications.

Our approach expands upon this form of user interaction by specifying two ways in which dynamic models and the corresponding physical phenomena may be viewed, and experimented with, within the same human interaction space. We present a methodology and implementation of contextualization for diagram-based dynamic models using an anesthesia machine, and then follow up with a human study of its effects on spatial cognition.

References

[1]

Banks, J., Carson, J. S., Nelson, B., and Nicol, D. 2009. Discrete-Event System Simulation. M. Horton, Ed. Prentice Hall, Upper Saddle River, NJ.

[2]

Barnes, M. 1996. Virtual reality and simulation. In Proceedings of the Winter Simulation Conference. 101--110.

Digital Library

[3]

Bradski, G. 2000. The OpenCV library. Dr. Dobb's J. 25, 11, 120--126.

[4]

Brooks, F. P. 1999. What's real about virtual reality&quest; IEEE Comput. Graph. Appl. 19, 6, 16--27.

Digital Library

[5]

Burdea, G. and Coiffet, P. 2003. Virtual Reality Technology. John Wiley and Sons, New York, NJ.

Digital Library

[6]

Cellier, F. E. 1991. Continuous System Modeling. Springer, New York.

Digital Library

[7]

Dabney, J. B. and Harman T. L. 1997. Mastering Simulink. Prentice Hall PTR, Upper Saddle River, NJ.

Digital Library

[8]

Fischler, I., Kaschub, C., Lizdas, D., and Lampotang, S. 2008. Understanding of anesthesia machine function is enhanced with a transparent reality simulation. Simul. Healthcare 3, 26--32.

[9]

Fishwick, P., Davis, T., and Douglas, J. 2005. Model representation with aesthetic computing: Method and empirical study. ACM Trans. Model. Comput. Simul. 15, 3, 254--279.

Digital Library

[10]

Fishwick, P. A. 1995. Simulation Model Design and Execution: Building Digital Worlds. Prentice Hall PTR, Upper Saddle River, NJ.

Digital Library

[11]

Fishwick, P. A. 2004. Toward an integrative multimodeling interface: A human-computer interface approach to interrelating model structures. Simul. 80, 9, 421--432.

[12]

Gallagher, A., Ritter, E., Champion, H., Higgins, G., Fried, M., Moses, G., Smith, C., and Satava, R. 2005. Virtual reality simulation for the operating room: Proficiency-Based training as a paradigm shift in surgical skills training. Ann. Surgery 241, 2, 364--372.

[13]

Goldstone, R. and Son, J. 2005. The transfer of scientific principles using concrete and idealized simulations. J. Learn. Sci. 14, 1, 69--110.

[14]

Grant, H. and Lai, C. 1998. Simulation modeling with artificial reality technology (SMART): An integration of virtual reality and simulation modeling. In Proceedings of the Winter Simulation Conference. 437--442.

Digital Library

[15]

Hegarty, M., Montello, D., Richardson, A., Ishikawa, T., and Lovelace, K. 2006. Spatial abilities at different scales: Individual differences in aptitude-test performance and spatial-layout learning. Intell. 34, 2, 151--176.

[16]

Huttenlocher, J. and Presson, C. 1973. Mental rotation and the perspective problem. Cogn. Psycho. 4, 277--299.

[17]

Ishii, H. and Ullmer, B. 1997. Tangible bits: Towards seamless interfaces between people, bits and atoms. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 234--241.

Digital Library

[18]

Just, M. A. and Carpenter, P. 1985. Cognitive coordinate systems: Accounts of mental rotation and individual differences in spatial ability. Psychol. Rev. 92, 137--171.

[19]

Kay, A. 1990. User interface: A personal view. In The Art of Human-Computer Interface Design, B. Laurel, Ed. Addison-Wesley Professional, New York.

[20]

Kelton, W., Sadowski, R., and Sturrock, D. 2003. Simulation with ARENA. S. Jeans, Ed. McGraw-Hill Science/Engineering/Math, New York.

Digital Library

[21]

Lampotang, S., Lizdas, D., Liem, E., and Dobbins, W. 2006a. The virtual anesthesia machine simulation. “http://vam.anest.ufl.edu/members/standard/vam.html”

[22]

Lampotang, S., Lizdas, D., Gravenstein, N., and Liem, E. 2006b. Transparent reality, A simulation based on interactive dynamic graphical models emphasizing visualization. Educ. Technol. 46, 1, 55--59.

[23]

Lampotang, S., Lizdas, D., Liem, E., and Gravenstein J. 2007. The anesthesia patient safety foundation anesthesia machine workbook v1.1a. http://vam.anest.ufl.edu/members/workbook/apsf-workbook-english.html

[24]

Law, A. and Kelton, W. 2000. Simulation Modeling and Analysis. McGraw-Hill Higher Education, New York.

Digital Library

[25]

Looser, J., Billinghurst, M., and Cockburn, A. 2004. Through the looking glass: The use of lenses as an interface tool for augmented reality interfaces. In Proceedings of the 2nd International Conference on Computer Graphics and Interactive Techniques. 204--211.

Digital Library

[26]

Macedonia, M. 2002. Games soldiers play. IEEE Spectrum 39, 3, 32--37.

Digital Library

[27]

Macredie, R., Taylor, S., Yu, X., and Keeble, R. 1996. Virtual reality and simulation: An overview. In Proceedings of the Winter Simulation Conference. 669--674.

Digital Library

[28]

Mattsson, S., Elmqvist, H., and Otter, M. 1998. Physical system modeling with Modelica. Control Engin. Pract. 6, 4, 501--510.

[29]

Milgram, P. and Kishino, F. 1994. A taxonomy of mixed reality visual displays. IEICE Trans. Inform. Syst. 77, 1321--1329.

[30]

Moffat, S., Hampson, E., and Hatzipantelis, M. 1998. Navigation in a virtual maze: Sex differences and correlation with psychometric measures of spatial ability in humans. Evol. Hum. Behav. 19, 73--87.

[31]

Montello, D. R. 1993. Scale and multiple psychologies of space. In Proceedings of the COSIT'93 Conference Spatial Information Theory: A Theoretical Basis for GIS. Lecture Notes in Computer Science, vol. 716. Springer, 312--321.

[32]

Nance, R. E. 1971. On time flow mechanisms for discrete system simulation. Manag. Sci. 18, 1, 59--73.

[33]

Nance, R. E. and Sargent, R. 2002. Perspectives on the evolution of simulation. Oper. Res. 50, 1, 161--172.

Digital Library

[34]

Norman, D. A. 1988. The Psychology of Everyday Things. Basic Books, New York.

[35]

Otter, M., Elmqvistm, H., and Cellier, F. 1996. Modeling of multibody systems with the object-oriented modeling language Dymola. Nonlinear Dynam. 9, 1, 91--112.

[36]

Park, M. and Fishwick, P. 2004. An integrated environment blending dynamic and geometry models. In AI, Simulation and Planning In High Autonomy Systems, Lecture Notes in Computer Science, vol. 3397. Springer, 574--584.

Digital Library

[37]

Park, M. and Fishwick, P. 2005. Integrating dynamic and geometry model components through ontology- based inference. Simul. 81, 12, 795--813.

Digital Library

[38]

Park, M. 2005. Ontology- Based customizable 3D modeling for simulation. Ph. D. thesis, Department of Computer and Information Science and Engineering, University of Florida.

Digital Library

[39]

Pidd, M. 1996. Model development and HCI. In Proceedings of the Winter Simulation Conference. 681--686.

Digital Library

[40]

Pritsker, A. 1974. The GASP IV Simulation Language. John Wiley and Sons, New York, NY.

[41]

Quarles, J., Lampotang, S., Fischler, I., Fishwick, P., and Lok, B. 2008a. A mixed reality approach for merging abstract and concrete knowledge. In Proceedings of the IEEE Conference on Virtual Reality. 27--34.

[42]

Quarles, J., Lampotang, S., Fischler, I., Fishwick, P., and Lok, B. 2008b. Tangible user interfaces compensate for low spatial cognition. In Proceedings of the IEEE Conference on 3D User Interfaces. 11--18.

Digital Library

[43]

Quarles, J., Lampotang, S., Fischler, I., Fishwick, P., and Lok, B. 2008c. Scaffolded learning with mixed reality. Comput. Graph. 33, 1, 34--46.

Digital Library

[44]

Seymour, N. E., Gallagher, A., Roman, S., O'Brien, M., Bansal, V., Andersen, D., and Satava, R. 2002. Virtual reality training improves operating room performance: Results of a randomized, double- blinded study. Ann. Surgery 236, 4, 458--464.

[45]

Shim, H. 2006. An integrative and interactive approach to three-dimensional programming. Ph. D. thesis, Department of Computer and Information Science and Engineering, University of Florida.

Digital Library

[46]

Shim, H. and Fishwick, P. 2007. Enabling the concept of hyperspace by syntax/semantics co-location within a localized 3d visualization space. In Human-Computer Interaction in Cyberspace: Emerging Technologies and Applications.

[47]

Tocher, K. and Owen, D. 2008, 1960. The automatic programming of simulations. J. Simul. 2, 3, 143--152.

[48]

van Rhijn, A. and Mulder, J. 2005. Optical tracking and calibration of tangible interaction devices. In Proceedings of the Immersive Projection Technology and Virtual Environments Workshop.

Digital Library

[49]

Verdaasdonk, E., Stassen, L., Monteny, L., and Dankelman, J. 2006. Validation of a new basic virtual reality simulator for training of basic endoscopic skills. Surgic, Endoscop. 20, 3, 511--518.

[50]

Whitman, L., Jorgensen, M., Hathiyari, H., and Malzahn, D. 2004. Virtual reality: Its usefulness for ergonomic analysis. In Proceedings of the Winter Simulation Conference. 1740--1745.

Digital Library

[51]

Wilson, J. and Goldsman, D. 2001. Alan Pritsker's multifaceted career: Theory, practice, education, entrepreneurship, and service. IIE Trans. 33, 3, 139--147.

Cited By

Keiser N(2024)A systematic review of technology in the after-action review (or debrief)Organizational Psychology Review10.1177/20413866241245314Online publication date: 17-Apr-2024
https://doi.org/10.1177/20413866241245314
Nikolic DFilipovic N(2021)Application of AR Technology in BioengineeringComputational Modeling and Simulation Examples in Bioengineering10.1002/9781119563983.ch6(219-257)Online publication date: 2-Dec-2021
https://doi.org/10.1002/9781119563983.ch6
Akpan IShanker MRazavi R(2019)Improving the success of simulation projects using 3D visualization and virtual realityJournal of the Operational Research Society10.1080/01605682.2019.1641649(1-27)Online publication date: 12-Sep-2019
https://doi.org/10.1080/01605682.2019.1641649
Show More Cited By

Index Terms

A mixed reality approach for interactively blending dynamic models with corresponding physical phenomena
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
  2. Modeling and simulation

Recommendations

Mixed Reality MIDI Keyboard Demonstration
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences

The Mixed Reality MIDI Keyboard is a prototype designed to augment virtual reality experiences through the inclusion of a physical interface which aligns the user's senses with the virtual environment. It also serves as a platform on which the uses of ...
Mixed reality: are two hands better than one?
VRST '06: Proceedings of the ACM symposium on Virtual reality software and technology

For simulating hands-on tasks, the ease of enabling two-handed interaction with virtual objects gives Mixed Reality (MR) an expected advantage over Virtual Reality (VR). A user study examined whether two-handed interaction is critical for simulating ...
MR Coral Sea Evolved Mixed-Reality Aquarium with Physical MR Displays
VRIC '15: Proceedings of the 2015 Virtual Reality International Conference

MR Coral Sea is a mixed-reality (MR) aquarium with which a user can play with virtual fish using Coral Display, which mediates between the real and virtual spaces with multi-sensory physical feedback. We propose physical MR displays such as Coral ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation

ACM Transactions on Modeling and Computer Simulation Volume 20, Issue 4

October 2010

155 pages

ISSN:1049-3301

EISSN:1558-1195

DOI:10.1145/1842722

Issue’s Table of Contents

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 November 2010

Accepted: 01 November 2009

Revised: 01 November 2009

Received: 01 November 2008

Published in TOMACS Volume 20, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
420
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Keiser N(2024)A systematic review of technology in the after-action review (or debrief)Organizational Psychology Review10.1177/20413866241245314Online publication date: 17-Apr-2024
https://doi.org/10.1177/20413866241245314
Nikolic DFilipovic N(2021)Application of AR Technology in BioengineeringComputational Modeling and Simulation Examples in Bioengineering10.1002/9781119563983.ch6(219-257)Online publication date: 2-Dec-2021
https://doi.org/10.1002/9781119563983.ch6
Akpan IShanker MRazavi R(2019)Improving the success of simulation projects using 3D visualization and virtual realityJournal of the Operational Research Society10.1080/01605682.2019.1641649(1-27)Online publication date: 12-Sep-2019
https://doi.org/10.1080/01605682.2019.1641649
Akpan IShanker M(2018)A comparative evaluation of the effectiveness of virtual reality, 3D visualization and 2D visual interactive simulation: an exploratory meta-analysisSIMULATION10.1177/0037549718757039(003754971875703)Online publication date: 27-Feb-2018
https://doi.org/10.1177/0037549718757039
Fan PXu X(2014)AR Technology and its Application in Biomedical EngineeringApplied Mechanics and Materials10.4028/www.scientific.net/AMM.472.421472(421-426)Online publication date: Jan-2014
https://doi.org/10.4028/www.scientific.net/AMM.472.421
Ferrer VPerdomo ARashed-Ali HFies CQuarles J(2013)How Does Usability Impact Motivation in Augmented Reality Serious Games for Education?2013 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES)10.1109/VS-GAMES.2013.6624233(1-8)Online publication date: Sep-2013
https://doi.org/10.1109/VS-GAMES.2013.6624233
Ezzell ZFishwick PCendan J(2011)Linking simulation and visualization construction through interactions with an ontology visualizationProceedings of the Winter Simulation Conference10.5555/2431518.2431866(2926-2937)Online publication date: 11-Dec-2011
https://dl.acm.org/doi/10.5555/2431518.2431866
Ezzell ZFishwick PCendan J(2011)Linking simulation and visualization construction through interactions with an ontology visualizationProceedings of the 2011 Winter Simulation Conference (WSC)10.1109/WSC.2011.6147995(2921-2932)Online publication date: Dec-2011
https://doi.org/10.1109/WSC.2011.6147995

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents