Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content

Advertisement

Springer Nature Link
Log in

AR-enhanced digital twin for human–robot interaction in manufacturing systems

  • Original Article
  • Published:
Energy, Ecology and Environment Aims and scope Submit manuscript

Abstract

The integration of advanced technologies into manufacturing processes is critical for addressing the complexities of modern industrial environments. In particular, the realm of human–robot interaction (HRI) faces the challenge of ensuring that human operators can effectively collaborate with increasingly sophisticated robotic systems. Traditional interfaces often fall short of providing the intuitive, real-time interaction necessary for optimal performance and safety. To address this issue, we introduce a novel system that combines digital twin (DT) technology with augmented reality (AR) to enhance HRI in manufacturing settings. The proposed AR-based DT system creates a dynamic virtual model of robot operations, offering an immersive interface that overlays crucial information onto the user’s field of vision. This approach aims to bridge the gap between human operators and robotic systems, improving spatial awareness, task guidance, and decision-making processes. Our system is designed to operate at three distinct levels of DT functionality: the virtual twin for in-situ monitoring, the hybrid twin for intuitive interaction, and the cognitive twin for optimized operation. By leveraging these levels, the system provides a comprehensive solution that ranges from basic visualization to advanced predictive analytics. The effectiveness of the AR-based DT system is demonstrated through a human-centric user study conducted in manufacturing scenarios. The results show a significant reduction in operational time and errors, alongside an enhancement of the overall user experience. These findings confirm the potential of our system to transform HRI by providing a safer, more efficient, and more adaptable manufacturing environment. Our research contributes to the advancement of smart manufacturing by evidencing the synergistic benefits of integrating DT and AR into HRI.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Amtsberg F, Yang X, Skoury L et al (2021) iHRC: an AR-based interface for intuitive, interactive and coordinated task sharing between humans and robots in building construction. In: ISARC. Proceedings of the international symposium on automation and robotics in construction. IAARC Publications, pp 25–32

  • Arevalo Arboleda S, Rücker F, Dierks T et al (2021) Assisting manipulation and grasping in robot teleoperation with augmented reality visual cues. In: Proceedings of the 2021 CHI conference on human factors in computing systems, pp 1–14

  • Baicun W, Yuan X, Jianlin Y et al (2020) Human-centered intelligent manufacturing: overview and perspectives. Strateg Study CAE 22(04):139

    Article  Google Scholar 

  • Baroroh DK, Chu CH, Wang L (2021) Systematic literature review on augmented reality in smart manufacturing: collaboration between human and computational intelligence. J Manuf Syst 61:696–711

    Article  Google Scholar 

  • Bhatt PM, Malhan RK, Shembekar AV et al (2020) Expanding capabilities of additive manufacturing through use of robotics technologies: a survey. Addit Manuf 31:100933

    Google Scholar 

  • Breque M, De Nul L, Petridis A (2021) Industry 5.0: towards a sustainable, human-centric and resilient European industry. European Commission, Directorate-General for Research and Innovation, Luxembourg

    Google Scholar 

  • Brooke J (1996) SUS—a quick and dirty usability scale. Usab Eval Ind 189(194):4–7

    Google Scholar 

  • Cai Y, Wang Y, Burnett M (2020) Using augmented reality to build digital twin for reconfigurable additive manufacturing system. J Manuf Syst 56:598–604

    Article  Google Scholar 

  • Cao Y, Xu Z, Li F et al (2019) V. ra: an in-situ visual authoring system for robot-IoT task planning with augmented reality. In: Proceedings of the 2019 on designing interactive systems conference, pp 1059–1070

  • Cauchard JR, Tamkin A, Wang CY et al (2019) Drone. io: a gestural and visual interface for human-drone interaction. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 153–162

  • Chen L, Takashima K, Fujita K et al (2021) Pinpointfly: an egocentric position-control drone interface using mobile ar. In: Proceedings of the 2021 CHI conference on human factors in computing systems. Association for Computing Machinery, New York. https://doi.org/10.1145/3411764.3445110

  • Choi S, Cai Y (2014) A virtual prototyping system with reconfigurable actuators for multi-material layered manufacturing. Comput Ind 65(1):37–49

    Article  Google Scholar 

  • Dimitropoulos N, Togias T, Zacharaki N et al (2021) Seamless human–robot collaborative assembly using artificial intelligence and wearable devices. Appl Sci 11(12):5699

    Article  CAS  Google Scholar 

  • Eswaran M, Bahubalendruni MR (2022) Challenges and opportunities on AR/VR technologies for manufacturing systems in the context of industry 4.0: a state of the art review. J Manuf Syst 65:260–278

    Article  Google Scholar 

  • Fuste A, Reynolds B, Hobin J et al (2020) Kinetic ar: a framework for robotic motion systems in spatial computing. In: Extended abstracts of the 2020 CHI conference on human factors in computing systems. Association for Computing Machinery, New York, pp. 1–8. https://doi.org/10.1145/3334480.3382814,

  • Grieves M, Vickers J (2017) Digital twin: mitigating unpredictable, undesirable emergent behavior in complex systems. Transdisciplinary perspectives on complex systems: new findings and approaches, pp 85–113

  • Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv Psychol 52:139–183

    Article  Google Scholar 

  • Hassenzahl M, Burmester M, Koller F (2003) Attrakdiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität. Mensch & Computer 2003: Interaktion in Bewegung, pp 187–196

  • Hedayati H, Walker M, Szafir D (2018) Improving collocated robot teleoperation with augmented reality. In: Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction, pp 78–86

  • Hoang KC, Chan WP, Lay S et al (2022) Virtual barriers in augmented reality for safe and effective human-robot cooperation in manufacturing. In: 2022 31st IEEE international conference on robot and human interactive communication (RO-MAN). IEEE, pp 1174–1180

  • Huang S, Wang B, Li X et al (2022) Industry 5.0 and society 5.0-comparison, complementation and co-evolution. J Manuf Syst 64:424–428

    Article  Google Scholar 

  • ISO I (2020) Dis 23247-1 automation systems and integration-digital twin framework for manufacturing. International Organization for Standardization, Geneva

  • Kuts V, Otto T, Tähemaa T et al (2018) Adaptive industrial robots using machine vision. In: ASME international mechanical engineering congress and exposition. American Society of Mechanical Engineers, p V002T02A093

  • Leng J, Zhu X, Huang Z et al (2023) Manuchain II: blockchained smart contract system as the digital twin of decentralized autonomous manufacturing toward resilience in industry 5.0. IEEE Trans Syst Man Cybern: Syst

  • Li C, Zheng P, Yin Y et al (2023) An AR-assisted Deep Reinforcement Learning-based approach towards mutual-cognitive safe human-robot interaction. Robot Comput-Integr Manuf 80:102471. https://doi.org/10.1016/j.rcim.2022.102471

    Article  Google Scholar 

  • Lindlbauer D, Grønbæk JE, Birk M et al (2016) Combining shape-changing interfaces and spatial augmented reality enables extended object appearance. In: Proceedings of the 2016 CHI conference on human factors in computing systems. Association for Computing Machinery, New York, pp 791–802. https://doi.org/10.1145/2858036.2858457

  • Liu S, Wang XV, Wang L (2022a) Digital twin-enabled advance execution for human–robot collaborative assembly. CIRP Ann 71(1):25–28

  • Liu X, Zheng L, Wang Y et al (2022b) Human-centric collaborative assembly system for large-scale space deployable mechanism driven by digital twins and wearable AR devices. J Manuf Syst 65:720–742

  • Liu YK, Ong SK, Nee AYC (2022c) State-of-the-art survey on digital twin implementations. Adv Manuf 10(1):1–23. https://doi.org/10.1007/s40436-021-00375-w

  • Lu Y, Wang H, Feng N et al (2022) Online interaction method of mobile robot based on single-channel EEG signal and end-to-end CNN with residual block model. Adv Eng Inform 52:101595

    Article  Google Scholar 

  • Mukherjee D, Gupta K, Chang LH et al (2022) A survey of robot learning strategies for human–robot collaboration in industrial settings. Robot Comput-Integr Manuf 73:102231

    Article  Google Scholar 

  • Müller F, Deuerlein C, Koch M (2021) Cyber-physical-system for representing a robot end effector. Procedia CIRP 100:307–312

    Article  Google Scholar 

  • Okegbile SD, Cai J, Yi C et al (2022) Human digital twin for personalized healthcare: vision, architecture and future directions. IEEE Netw 37:262–269

    Article  Google Scholar 

  • Ostanin M, Mikhel S, Evlampiev A, et al (2020) Human-robot interaction for robotic manipulator programming in mixed reality. In: 2020 IEEE international conference on robotics and automation (ICRA), pp 2805–2811. IEEE

  • Özgür A, Lemaignan S, Johal W et al (2017) Cellulo: versatile handheld robots for education. In: Proceedings of the 2017 ACM/IEEE international conference on human–robot interaction, pp 119–127

  • Paripooranan CS, Abishek R, Vivek D et al (2020) An implementation of AR enabled digital twins for 3-d printing. In: 2020 IEEE international symposium on smart electronic systems (iSES) (Formerly iNiS). IEEE, pp 155–160

  • Pedersen EW, Hornbæk K (2011) Tangible bots: interaction with active tangibles in tabletop interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems. Association for Computing Machinery, New York, pp 2975–2984. https://doi.org/10.1145/1978942.1979384

  • Peng H, Briggs J, Wang CY et al (2018) Roma: interactive fabrication with augmented reality and a robotic 3d printer. In: Proceedings of the 2018 CHI conference on human factors in computing systems, pp 1–12

  • Quintero CP, Li S, Pan MK et al (2018) Robot programming through augmented trajectories in augmented reality. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 1838–1844

  • Sifat MMH, Choudhury SM, Das SK et al (2022) Towards electric digital twin grid: technology and framework review. Energy AI 11:100213

    Article  Google Scholar 

  • Sisinni E, Saifullah A, Han S et al (2018) Industrial internet of things: challenges, opportunities, and directions. IEEE Trans Ind Inf 14(11):4724–4734

    Article  Google Scholar 

  • Siu AF, Yuan S, Pham H et al (2018) Investigating tangible collaboration for design towards augmented physical telepresence. In: Design thinking research: making distinctions: collaboration versus cooperation, pp 131–145

  • Suzuki R, Karim A, Xia T et al (2022) Augmented reality and robotics: a survey and taxonomy for AR-enhanced human–robot interaction and robotic interfaces. In: CHI conference on human factors in computing systems. ACM, New Orleans, pp 1–33. https://doi.org/10.1145/3491102.3517719

  • Takashima K, Oyama T, Asari Y et al (2016) Study and design of a shape-shifting wall display. In: Proceedings of the 2016 ACM conference on designing interactive systems, pp 796–806

  • Tao F, Sui F, Liu A et al (2019) Digital twin-driven product design framework. Int J Prod Res 57(12):3935–3953

    Article  Google Scholar 

  • Tuegel EJ, Ingraffea AR, Eason TG et al (2011) Reengineering aircraft structural life prediction using a digital twin. Int J Aerosp Eng 2011:1687–5966

    Article  Google Scholar 

  • Villani V, Pini F, Leali F et al (2018) Survey on human-robot collaboration in industrial settings: safety, intuitive interfaces and applications. Mechatronics 55:248–266

    Article  Google Scholar 

  • Wan J, Li X, Dai HN et al (2020) Artificial-intelligence-driven customized manufacturing factory: key technologies, applications, and challenges. Proc IEEE 109(4):377–398

    Article  Google Scholar 

  • Wang B, Zheng P, Yin Y et al (2022) Toward human-centric smart manufacturing: a human-cyber-physical systems (hcps) perspective. J Manuf Syst 63:471–490

    Article  Google Scholar 

  • Watanabe A, Ikeda T, Morales Y et al (2015) Communicating robotic navigational intentions. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 5763–5769. https://doi.org/10.1109/IROS.2015.7354195

  • Yin Y, Zheng P, Li C et al (2023) A state-of-the-art survey on Augmented Reality-assisted Digital Twin for futuristic human-centric industry transformation. Robot Comput-Integr Manuf 81:102515. https://doi.org/10.1016/j.rcim.2022.102515

    Article  Google Scholar 

  • Yuan L, Reardon C, Warnell G et al (2019) Human gaze-driven spatial tasking of an autonomous MAV. IEEE Robot Autom Lett 4(2):1343–1350

    Article  Google Scholar 

Download references

Acknowledgements

This research is supported by The Hong Kong University of Science and Technology (Guangzhou) and the Department of Science and Technology of Guangdong Province (2021QN02Z112).

Author information

Authors and Affiliations

  1. Division of Emerging Interdisciplinary Areas (EMIA), Academy of Interdisciplinary Studies, The Hong Kong University of Science and Technology, Hong Kong SAR, China

    Zhongyuan Liao & Yi Cai

  2. Smart Manufacturing Thrust, Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

    Yi Cai

  3. Department of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China

    Yi Cai

Authors
  1. Zhongyuan Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yi Cai.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Appendices

Appendix A: DH parameters

See Fig. 14.

Fig. 14
figure 14

The DH parameters for the robot arm (ER myCobot 280 Pi)

Full size image

Appendix B: Questionnaire

See Table 2.

Table 2 The questionnaire for post-experiment survey
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liao, Z., Cai, Y. AR-enhanced digital twin for human–robot interaction in manufacturing systems. Energ. Ecol. Environ. 9, 530–548 (2024). https://doi.org/10.1007/s40974-024-00327-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40974-024-00327-7

Keywords

Search

Navigation