research-article

An Expert Digital Companion for Working Environments

Authors:

Kimberly Garcia,

Simon MayerAuthors Info & Claims

IoT '21: Proceedings of the 11th International Conference on the Internet of Things

Pages 25 - 32

https://doi.org/10.1145/3494322.3494326

Published: 08 March 2022 Publication History

Abstract

Research in proactive agents capable of anticipating users’ needs has been conducted in different application areas, from agents aiming to help users accomplish their next task while using a text editor to digital assistants that provide support to medical professionals in hospitals. Considering these works and following the rapid development in machine learning, the growing adoption of the IoT, the prevalence of pervasive computing, and the various innovative user interfaces that are becoming mainstream, we propose the creation of expert digital companions. Expert digital companions are agents that know the specifics of an environment and the available physical equipment in a space as well as software components reachable in that environment. On their side, personal digital companions know the preferences of a specific user. The separation of concerns between expert and personal companions enables them to evolve independently. Thus, expert and personal digital companions collaborate to provide customized assistance to a user given her environment. In this paper, we present two expert digital companions for employees working in an office and a shopfloor environment that takes advantages of: the decoupling that the W3C WoT Thing Description provides to interact with and control devices and other tools; a Knowledge Graph to provide richer descriptions of elements in the environment; computer vision to perceive the physical world; and Mixed Reality as the medium to deliver assistance to workers.

References

[1]

Sultan Noori Al-karawi. 2014. Comparison of costs of personal protective equipment for all workers to avoid costs of fall accidents. Ph.D. Dissertation. Missouri University of Sciene and Technology, Rolla, Missouri. https://scholarsmine.mst.edu/cgi/viewcontent.cgi?article=8260&context=masters_theses

[2]

Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv:2004.10934 [cs, eess] (April 2020). http://arxiv.org/abs/2004.10934 arXiv:2004.10934.

[3]

P R Boyce, J A Veitch, G R Newsham, C C Jones, J Heerwagen, M Myer, and C M Hunter. 2006. Lighting quality and office work: two field simulation experiments. Lighting Research & Technology 38, 3 (Sept. 2006), 191–223. https://doi.org/10.1191/1365782806lrt161oa

[4]

Hans Chalupsky, Yolanda Gil, Craig A Knoblock, Kristina Lerman, Jean Oh, David V Pynadath, Thomas A Russ, and Milind Tambe. 2002. Electric Elves: Agent technology for supporting human organizations. AI magazine 23, 2 (2002), 11–11.

[5]

Andrei Ciortea, Simon Mayer, Simon Bienz, Fabien Gandon, and Olivier Corby. 2020. Autonomous Search in a Social and Ubiquitous Web. Personal and Ubiquitous Computing (Jun 2020). https://doi.org/10.1007/s00779-020-01415-1

Digital Library

[6]

Angelo Croatti, Manuel Bottazzi, and Alessandro Ricci. 2020. Agent-Based Mixed Reality Environments in Healthcare: The Smart Shock Room Project. In Advances in Practical Applications of Agents, Multi-Agent Systems, and Trustworthiness. The PAAMS Collection, Yves Demazeau, Tom Holvoet, Juan M. Corchado, and Stefania Costantini (Eds.). Springer International Publishing, Cham, 398–402.

[7]

Suisse Secrétariat d’Etat à l’économie (SECO). 2020. Commentaire de l’ordonnance 3 relative à la loi sur le travail, Art. 16. https://www.seco.admin.ch/seco/fr/home/Arbeit/Arbeitsbedingungen/Arbeitsgesetz-und-Verordnungen/Wegleitungen/wegleitung-zur-argv-3.html##-1755811567.

[8]

Lei Fang, David Peter Wyon, Geo Clausen, and Povl Ole Fanger. 2004. Impact of indoor air temperature and humidity in an office on perceived air quality, SBS symptoms and performance. Indoor Air 14, s7 (Aug. 2004), 74–81. https://doi.org/10.1111/j.1600-0668.2004.00276.x

[9]

Kimberly Garcia, Daniel David Schwyn, and Florian Michahelles. 2020. A Digital Companion for Air Travelers. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI ’20). Association for Computing Machinery, New York, NY, USA, Article 35, 6 pages. https://doi.org/10.1145/3406324.3410544

Digital Library

[10]

Kimberly Garcia, Simon Mayer, Alessandro Ricci, and Andrei Ciortea. 2020. Proactive Digital Companions in Pervasive Hypermedia Environments. In 2020 IEEE 6th International Conference on Collaboration and Internet Computing (CIC). 54–59. https://doi.org/10.1109/CIC50333.2020.00017

[11]

Andres Gomez. 2020. On-demand communication with the batteryless MiroCard: demo abstract. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems. 629–630.

Digital Library

[12]

Dominique Guinard, Vlad Trifa, Friedemann Mattern, and Erik Wilde. 2011. From the internet of things to the web of things: Resource-oriented architecture and best practices. In Architecting the Internet of things. Springer, 97–129.

[13]

Barry P. Haynes. 2008. The impact of office comfort on productivity. Journal of Facilities Management 6, 1 (Feb. 2008), 37–51. https://doi.org/10.1108/14725960810847459

[14]

Sumedha Joshi. 2008. The sick building syndrome. Indian journal of occupational and environmental medicine 12 (08 2008), 61–4. https://doi.org/10.4103/0019-5278.43262

[15]

Jung-Sing Jwo, Ching-Sheng Lin, and Cheng-Hsiung Lee. 2021. Smart technology–driven aspects for human-in-the-loop smart manufacturing. The International Journal of Advanced Manufacturing Technology 114, 5-6 (May 2021), 1741–1752. https://doi.org/10.1007/s00170-021-06977-9

[16]

Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, 2018. Advancing the state of the art in open domain dialog systems through the alexa prize. arXiv preprint arXiv:1812.10757(2018).

[17]

Ue-Hwan Kim, Jin-Man Park, Taek-Jin Song, and Jong-Hwan Kim. 2019. 3-d scene graph: A sparse and semantic representation of physical environments for intelligent agents. IEEE transactions on cybernetics 50, 12 (2019), 4921–4933.

[18]

Alan Kohll. 2019. How Your Office Space Impacts Employee Well-Being. https://www.forbes.com/sites/alankohll/2019/01/24/how-your-office-space-impacts-employee-wellbeing/ Section: Leadership Strategy.

[19]

Yikang Li, Wanli Ouyang, Bolei Zhou, Jianping Shi, Chao Zhang, and Xiaogang Wang. 2018. Factorizable net: an efficient subgraph-based framework for scene graph generation. In Proceedings of the European Conference on Computer Vision (ECCV). 335–351.

Digital Library

[20]

Pattie Maes. 1995. Agents that reduce work and information overload. In Readings in human–computer interaction. Elsevier, 811–821.

[21]

William W Nazaroff. 2013. Four principles for achieving good indoor air quality. Indoor Air 23, 5 (Oct. 2013), 353–356. https://doi.org/10.1111/ina.12062

[22]

Guy Newsham, Jennifer Veitch, Chantal Arsenault, and Cara Duval. 2004. Effect of dimming control on office worker satisfaction and performance. In Proceedings of the IESNA annual conference. Citeseer, 19–41.

[23]

Jasmin Niess and Paweł W Woźniak. 2020. Embracing companion technologies. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society. 1–11.

Digital Library

[24]

European Statistics on Accidents at Work (ESAW). 2020. Accidents at work statistics. https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Accidents_at_work_statistics

[25]

Marie-Pierre Pacaux-Lemoine, Damien Trentesaux, Gabriel Zambrano Rey, and Patrick Millot. 2017. Designing intelligent manufacturing systems through Human-Machine Cooperation principles: A human-centered approach. Computers & Industrial Engineering 111 (Sept. 2017), 581–595. https://doi.org/10.1016/j.cie.2017.05.014

Digital Library

[26]

Malcolm Peltu and Yorick Wilks. 2008. Close engagements with artificial companions: Key social, psychological, ethical and design issues. OII Forum Discussion Paper.

[27]

Joseph Redmon and Ali Farhadi. 2017. YOLO9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7263–7271.

[28]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2016. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. arXiv:1506.01497 [cs] (Jan. 2016). http://arxiv.org/abs/1506.01497 arXiv:1506.01497.

[29]

Mihail C Roco and William S Bainbridge. 2002. Converging technologies for improving human performance: Integrating from the nanoscale. Journal of nanoparticle research 4, 4 (2002), 281–295.

[30]

Weiming Shen and Douglas H. Norrie. 1999. Agent-Based Systems for Intelligent Manufacturing: A State-of-the-Art Survey. Knowledge and Information Systems 1, 2 (May 1999), 129–156. https://doi.org/10.1007/BF03325096

Digital Library

[31]

Francesca Stazi, Federica Naspi, Giulia Ulpiani, and Costanzo Di Perna. 2017. Indoor air quality and thermal comfort optimization in classrooms developing an automatic system for windows opening and closing. Energy and Buildings(2017), 15.

[32]

Ja Veitch, Gr Newsham, Pr Boyce, and Cc Jones. 2008. Lighting appraisal, well-being and performance in open-plan offices: A linked mechanisms approach. Lighting Research & Technology 40, 2 (June 2008), 133–151. https://doi.org/10.1177/1477153507086279

[33]

Max von Pettenkofer. 1858. Über den Luftwechsel in Wohngebäuden. Cotta.

[34]

Zijian Wang, Yimin Wu, Lichao Yang, Arjun Thirunavukarasu, Colin Evison, and Yifan Zhao. 2021. Fast Personal Protective Equipment Detection for Real Construction Sites Using Deep Learning Approaches. Sensors 21, 10 (May 2021), 3478. https://doi.org/10.3390/s21103478

[35]

Pawel Wargocki. 2008. Improving Indoor Air Quality Improves The Performance Of Office Work And School Work. (2008), 7.

[36]

Pawel Wargocki, K. W. Tham, S. C. Sekhar, and D. Cheong. 2003. Estimate of an economic benefit from investment in improved indoor air quality in an office building. In Proceedings of Healthy Buildings 2003, Vol. 3. University of Singapore, Department of Buildings, 382–387. http://hb2003.nus.edu.sg/

[37]

Yorick Wilks. 2006. Artificial companions as a new kind of interface to the future internet. (2006).

[38]

David P Wyon and Pawel Wargocki. 2005. Indoor Air Quality Effects On Office Work. (2005), 13.

[39]

Neil Yorke-Smith, Shahin Saadati, Karen L Myers, and David N Morley. 2012. The design of a proactive personal agent for task management. International Journal on Artificial Intelligence Tools 21, 01(2012), 1250004.

[40]

Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. 2020. Bridging knowledge graphs to generate scene graphs. In European Conference on Computer Vision. Springer, 606–623.

Digital Library

Cited By

Fan SShi HWang CMa RWang X(2024)RFL-LSU: A Robust Federated Learning Approach with Localized Stepwise UpdatesACM Transactions on Internet Technology10.1145/369082224:4(1-26)Online publication date: 15-Nov-2024
https://dl.acm.org/doi/10.1145/3690822
Garcia KVontobel JMayer S(2024)A Digital Companion Architecture for Ambient IntelligenceProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596108:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659610
Jha SGarcia KAntille YSolèr MPadua SMayer S(2024)Temporal Scene Understanding using Contextually Unique Identification2024 IEEE 36th International Conference on Tools with Artificial Intelligence (ICTAI)10.1109/ICTAI62512.2024.00145(1016-1023)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ICTAI62512.2024.00145
Show More Cited By

Index Terms

An Expert Digital Companion for Working Environments
1. Computing methodologies
  1. Artificial intelligence
2. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

SOCRAR: Semantic OCR through Augmented Reality
IoT '22: Proceedings of the 12th International Conference on the Internet of Things

To enable people to interact more efficiently with virtual and physical services in their surroundings, it would be beneficial if information could more fluently be passed across digital and non-digital spaces. To this end, we propose to combine ...
HoloWoT: A First Step Towards Mixed Reality Digital Twins for the Industrial Internet of Things
MODELS Companion '24: Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems

The Metaverse has recently gained popularity, bridging physical and digital realms via Extended Reality (XR) technologies. Such technologies provide unprecedented opportunities for visualizing decentralized Internet of Things (IoT) systems. This paper ...
Ubiquitous home simulation using augmented reality
CEA'07: Proceedings of the 2007 annual Conference on International Conference on Computer Engineering and Applications

Computing paradigm is moving toward ubiquitous computing in which devices, software agents, and services are all expected to seamlessly integrate and cooperate in support of human objectives. Augmented reality (AR) can naturally complement ubiquitous ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

IoT '21: Proceedings of the 11th International Conference on the Internet of Things

November 2021

233 pages

ISBN:9781450385664

DOI:10.1145/3494322

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 March 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

IoT '21

IoT '21: 11th International Conference on the Internet of Things

November 8 - 12, 2021

St.Gallen, Switzerland

Acceptance Rates

Overall Acceptance Rate 28 of 84 submissions, 33%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
182
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)4

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fan SShi HWang CMa RWang X(2024)RFL-LSU: A Robust Federated Learning Approach with Localized Stepwise UpdatesACM Transactions on Internet Technology10.1145/369082224:4(1-26)Online publication date: 15-Nov-2024
https://dl.acm.org/doi/10.1145/3690822
Garcia KVontobel JMayer S(2024)A Digital Companion Architecture for Ambient IntelligenceProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596108:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659610
Jha SGarcia KAntille YSolèr MPadua SMayer S(2024)Temporal Scene Understanding using Contextually Unique Identification2024 IEEE 36th International Conference on Tools with Artificial Intelligence (ICTAI)10.1109/ICTAI62512.2024.00145(1016-1023)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ICTAI62512.2024.00145
Bektaş KStrecker JMayer SGarcia K(2024)Gaze-enabled activity recognition for augmented reality feedbackComputers and Graphics10.1016/j.cag.2024.103909119:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.cag.2024.103909
Kilic KWeck SKampik TLindgren H(2023)Argument-based human–AI collaboration for supporting behavior change to improve healthFrontiers in Artificial Intelligence10.3389/frai.2023.10694556Online publication date: 16-Feb-2023
https://doi.org/10.3389/frai.2023.1069455
Strecker JAkhunov KCarbone FGarcía KBektaş KGomez AMayer SYildirim K(2023)MR Object Identification and InteractionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108797:3(1-26)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610879

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten