Due to ever-increasing information overload, human attention has evolved to the most critical par... more Due to ever-increasing information overload, human attention has evolved to the most critical parameter in the successful design of pervasive display systems. This article aims at the validation of physical effort as a suitable descriptor for attentional and perceptional behavior in interactive public display scenarios. For this purpose, we integrated our qualitative effort-based behavior description approach into Wickens' established Saliency-Effort-Expectancy-Value (SEEV) attention model to compare predicted and observed attentional behavior and demonstrate its significance for attention mechanisms. The SEEV attention model is adapted to a public display scenario, with analysis focused on Minimum Required Effort (MRE) for the assessment of information. This attention modeling approach is evaluated based on data collected in an empiric study in an exhibition setting and is based on a database of 188 visitors. We employ different approaches of correlation analysis to evaluate th...
Dynamic changes of pupil dilation represent an established indicator of cognitive load in cogniti... more Dynamic changes of pupil dilation represent an established indicator of cognitive load in cognitive sciences Exploitation of these insights regarding pupil dilation as an indicator of cog-nitive load for attention-aware Information and Communication (ICT) systems has been impeded due to restrictions of pupil analysis to a posteriori processing and exclusion of disturbing environmental factors. To overcome these issues, this paper proposes an algorithm based on Hoeks's pupil response model, enabling online analysis of pupil dilation for the dynamic interpretation of cognitive load as an input for interactive, attention-aware systems, which outperforms state-of-the-art approaches regarding complexity, accuracy, flexibility and computation time. Beyond mathematical pupil modeling, this paper identifies Environment Illumination compensation (IC), Blink Compensation (BC), Reference Baseline computation (RB) and Onset/Offset detection (OO) as crucial fields of research for the transfe...
Abstract—This paper presents a novel approach towards identity management strategies for applicat... more Abstract—This paper presents a novel approach towards identity management strategies for application in large scale interactive systems while separating pure detection from the identity management processes. Detection is achieved by employing a scalable, modal, network-based, real-time multi-camera tracking system in which numerous cameras are used to cover large areas. Objects are detected by employing blob detection and image pre-processing algorithms. The proposed identity handling algorithm covers standard tracking as well as solving Split, Merge, and Handover problems between adjacent cameras. General ID handling strategies are introduced which allow reliable tracking and simple access to high-level movement data for later movement analysis. Results in terms of accuracy and technical feasibility are given, gathered from a pilot installation at a public event. I.
Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments, 2019
Companies are re-focusing and making use of human labor [4, 21] in order to create individualized... more Companies are re-focusing and making use of human labor [4, 21] in order to create individualized lot-size-1 products and not produce the exact same mass product again and again. While human workers can produce with a least with the same quality as machines do, they are not that consistent, so it's better to combine strengths of both men and machines [14]. In this work, we investigate how we can utilize how humans behave in relation to task-required skills levels. To do so we investigate hand-eye coordination on precision tasks, its relation to fine and gross motor skills, in an unconstrained industrial setting. This setting consists of an up to 22 tasks assembly processes of two variants of a high quality product. We establish that there is a high correlation between expected task required skill level and the captured hand eye coordination of expert factory workers and that hand eye coordination can be used to distinguish between fine and gross motor skills. In addition we prov...
With the availability of miniaturized low cost sensors and the general availability and easy appl... more With the availability of miniaturized low cost sensors and the general availability and easy applicability of algorithms for activity recognition, we investigate how various sensors can be deployed in a harsh environment, the industrial shop-floor. We review related work and provide an in-depth review of our own experiences were sensors wer used to enable recognition of activity, task progress and also mental and cognitive states of assembly workers. The recognition process is based on stationary (RGBD cameras, stereo vision depth sensors) and wearable devices (IMUs, GSR, ECG, mobile eye tracker). We describe in detail the used sensors, the challenges of fusing the data from these various sources together in real-time and how to interpret that data semantically.
Proceedings of the Symposium on Spatial User Interaction, 2018
We propose the idea of a powerful mobile eye tracking platform that enables whole new ways of exp... more We propose the idea of a powerful mobile eye tracking platform that enables whole new ways of explicit and implicit human-machine interactions in complex industrial settings. The system is based on two hardware components (NVIDIA Jetson TX2, Pupil labs eye tracker) and a message-oriented framework for real-time processing [1]. The design is described and potential use cases are sketched.
In the context of Industry 4.0, there is a strong focus on man-machine interaction, and a push fo... more In the context of Industry 4.0, there is a strong focus on man-machine interaction, and a push for ICT solutions in industrial applications. One aspect of this are industrial assistance systems, both to aid operators in their work and to train novice workers in complex processes. Addressing the latter purpose, in this work, a training station e-learning concept is detailed, with the purpose to automatically teach a novice worker the necessary steps to assemble an alpine ski without the need for constant human supervision. It is designed to observe and especially model the state of the trainee for optimal support via delivery of instructional material and feedback based on an evaluation of the trainee's needs and behavior. The training station is comprised of a work bench, displays to deliver instructional material, and various sensors to monitor both the trainee's progress and overall state. To enable best possible worker support, a model of worker state (Idle, Flow, Busy, O...
In the recent revival of human labour in industry, and the subsequent push to optimally combine t... more In the recent revival of human labour in industry, and the subsequent push to optimally combine the strengths of man and machine in industrial processes, there is an increased need for methods allowing machines to understand and interpret the actions of their users. An important aspect of this is the understanding and evaluation of the progress of the workflows that are to be executed. Methods for this require both an appropriate choice of sensors, as well as algorithms capable of quickly and efficiently evaluating activity and workflow progress. In this paper we present such an algorithm, which provides activity and workflow recognition using both depth and RGB cameras as input. The algorithm's main purpose is to be used in an industrial training station, allowing novice workers to learn the necessary steps in assembling nordic ski products without the need for human supervision. We will describe how the algorithm recognizes predefined workflows in the sensor data, and present ...
This paper motivates the employment of a cognitive headgear in the context of two concrete human-... more This paper motivates the employment of a cognitive headgear in the context of two concrete human-in-the-loop industrial use cases. However, the distinct design and implementation of such a headgear is strongly task-dependent and no one-fits-all solution exists. To overcome that, a computer-aided design process is proposed that allows the design of a device truly tailored to the individual needs. Further, the features and components required to support workers in the identified scenarios are depicted, including image-based worker localization, gaze-based skill-level detection, workflow recognition using head-worn sensors and multimodal assistance. Finally, early functional prototypes are illustrated.
Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, 2019
Due to the explicit and implicit facets of gaze-based interaction, eye tracking is a major area o... more Due to the explicit and implicit facets of gaze-based interaction, eye tracking is a major area of interest within the field of cognitive industrial assistance systems. In this position paper, we describe a scenario which includes a wearable platform built around a mobile eye tracker, which can support and guide an industrial worker throughout the execution of a maintenance task. The potential benefits of such a solution are discussed and the key components are outlined.
The eyes are a particularly interesting modality for cognitive industrial assistance systems, as ... more The eyes are a particularly interesting modality for cognitive industrial assistance systems, as gaze analysis can reveal cognition- and task-related aspects, while gaze interaction depicts a lightweight and fast method for hands-free machine control. In this paper, we present mobEYEle, a body-worn eye tracking platform that performs the entire computation directly on the user, as opposed to primarily streaming the data to a centralized unit for online processing and hence restricting its pervasiveness. The applicability of the platform is demonstrated throughout extensive performance and battery runtime tests. Moreover, a self-contained calibration method is outlined that enables the usage of mobEYEle without any supervisor nor digital screen.
2019 17th International Conference on Emerging eLearning Technologies and Applications (ICETA), 2019
The Airtention project aims at the realization of a pilot training assistance system which embodi... more The Airtention project aims at the realization of a pilot training assistance system which embodies a truly competence- and evidence-based training approach, with the ambition to replace established skill representations as ’hours of experience’ with actual measures and models of level of (i) automation, (ii) task execution accuracy, and (iii) task execution efficiency. For this purpose, a mobile eyetracker will be employed to assess and estimate procedural, behavioral and psychophysiological indicators such as visual attention, perception, cognitive load, interaction with objects of interest, and progress in the workflow to enable deductions about associated skill levels, and adaption of training plans both during runtime and on the scale of a long-term training schedule. Such a system will allow to create a training process which is flexible, adapting to the individual, while at the same time ensuring the quality of the training results.
A human attention estimation system is proposed, derived from automatic analysis of movement and ... more A human attention estimation system is proposed, derived from automatic analysis of movement and behavior of people in front of public displays. For this purpose, new attention metrics defining detailed levels and types of attention are proposed, based on an established attention model which is adapted to the specific use-case. New movement features are introduced which translate detected movements and behavior of people into the novel metric scales. Data capturing is achieved by depth image analysis for tracking of people and extraction of statistical movement data, whereas head orientation of people in front of the interactive system is estimated from RGB video analysis. Classification is carried out for long and short term attention levels, employing a data-driven approach via Support Vector Machines on experimental data derived from an empirical field study at a public exhibition. The system is capable of real-time feature extraction and classification.
Due to ever-increasing information overload, human attention has evolved to the most critical par... more Due to ever-increasing information overload, human attention has evolved to the most critical parameter in the successful design of pervasive display systems. This article aims at the validation of physical effort as a suitable descriptor for attentional and perceptional behavior in interactive public display scenarios. For this purpose, we integrated our qualitative effort-based behavior description approach into Wickens' established Saliency-Effort-Expectancy-Value (SEEV) attention model to compare predicted and observed attentional behavior and demonstrate its significance for attention mechanisms. The SEEV attention model is adapted to a public display scenario, with analysis focused on Minimum Required Effort (MRE) for the assessment of information. This attention modeling approach is evaluated based on data collected in an empiric study in an exhibition setting and is based on a database of 188 visitors. We employ different approaches of correlation analysis to evaluate th...
Dynamic changes of pupil dilation represent an established indicator of cognitive load in cogniti... more Dynamic changes of pupil dilation represent an established indicator of cognitive load in cognitive sciences Exploitation of these insights regarding pupil dilation as an indicator of cog-nitive load for attention-aware Information and Communication (ICT) systems has been impeded due to restrictions of pupil analysis to a posteriori processing and exclusion of disturbing environmental factors. To overcome these issues, this paper proposes an algorithm based on Hoeks's pupil response model, enabling online analysis of pupil dilation for the dynamic interpretation of cognitive load as an input for interactive, attention-aware systems, which outperforms state-of-the-art approaches regarding complexity, accuracy, flexibility and computation time. Beyond mathematical pupil modeling, this paper identifies Environment Illumination compensation (IC), Blink Compensation (BC), Reference Baseline computation (RB) and Onset/Offset detection (OO) as crucial fields of research for the transfe...
Abstract—This paper presents a novel approach towards identity management strategies for applicat... more Abstract—This paper presents a novel approach towards identity management strategies for application in large scale interactive systems while separating pure detection from the identity management processes. Detection is achieved by employing a scalable, modal, network-based, real-time multi-camera tracking system in which numerous cameras are used to cover large areas. Objects are detected by employing blob detection and image pre-processing algorithms. The proposed identity handling algorithm covers standard tracking as well as solving Split, Merge, and Handover problems between adjacent cameras. General ID handling strategies are introduced which allow reliable tracking and simple access to high-level movement data for later movement analysis. Results in terms of accuracy and technical feasibility are given, gathered from a pilot installation at a public event. I.
Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments, 2019
Companies are re-focusing and making use of human labor [4, 21] in order to create individualized... more Companies are re-focusing and making use of human labor [4, 21] in order to create individualized lot-size-1 products and not produce the exact same mass product again and again. While human workers can produce with a least with the same quality as machines do, they are not that consistent, so it's better to combine strengths of both men and machines [14]. In this work, we investigate how we can utilize how humans behave in relation to task-required skills levels. To do so we investigate hand-eye coordination on precision tasks, its relation to fine and gross motor skills, in an unconstrained industrial setting. This setting consists of an up to 22 tasks assembly processes of two variants of a high quality product. We establish that there is a high correlation between expected task required skill level and the captured hand eye coordination of expert factory workers and that hand eye coordination can be used to distinguish between fine and gross motor skills. In addition we prov...
With the availability of miniaturized low cost sensors and the general availability and easy appl... more With the availability of miniaturized low cost sensors and the general availability and easy applicability of algorithms for activity recognition, we investigate how various sensors can be deployed in a harsh environment, the industrial shop-floor. We review related work and provide an in-depth review of our own experiences were sensors wer used to enable recognition of activity, task progress and also mental and cognitive states of assembly workers. The recognition process is based on stationary (RGBD cameras, stereo vision depth sensors) and wearable devices (IMUs, GSR, ECG, mobile eye tracker). We describe in detail the used sensors, the challenges of fusing the data from these various sources together in real-time and how to interpret that data semantically.
Proceedings of the Symposium on Spatial User Interaction, 2018
We propose the idea of a powerful mobile eye tracking platform that enables whole new ways of exp... more We propose the idea of a powerful mobile eye tracking platform that enables whole new ways of explicit and implicit human-machine interactions in complex industrial settings. The system is based on two hardware components (NVIDIA Jetson TX2, Pupil labs eye tracker) and a message-oriented framework for real-time processing [1]. The design is described and potential use cases are sketched.
In the context of Industry 4.0, there is a strong focus on man-machine interaction, and a push fo... more In the context of Industry 4.0, there is a strong focus on man-machine interaction, and a push for ICT solutions in industrial applications. One aspect of this are industrial assistance systems, both to aid operators in their work and to train novice workers in complex processes. Addressing the latter purpose, in this work, a training station e-learning concept is detailed, with the purpose to automatically teach a novice worker the necessary steps to assemble an alpine ski without the need for constant human supervision. It is designed to observe and especially model the state of the trainee for optimal support via delivery of instructional material and feedback based on an evaluation of the trainee's needs and behavior. The training station is comprised of a work bench, displays to deliver instructional material, and various sensors to monitor both the trainee's progress and overall state. To enable best possible worker support, a model of worker state (Idle, Flow, Busy, O...
In the recent revival of human labour in industry, and the subsequent push to optimally combine t... more In the recent revival of human labour in industry, and the subsequent push to optimally combine the strengths of man and machine in industrial processes, there is an increased need for methods allowing machines to understand and interpret the actions of their users. An important aspect of this is the understanding and evaluation of the progress of the workflows that are to be executed. Methods for this require both an appropriate choice of sensors, as well as algorithms capable of quickly and efficiently evaluating activity and workflow progress. In this paper we present such an algorithm, which provides activity and workflow recognition using both depth and RGB cameras as input. The algorithm's main purpose is to be used in an industrial training station, allowing novice workers to learn the necessary steps in assembling nordic ski products without the need for human supervision. We will describe how the algorithm recognizes predefined workflows in the sensor data, and present ...
This paper motivates the employment of a cognitive headgear in the context of two concrete human-... more This paper motivates the employment of a cognitive headgear in the context of two concrete human-in-the-loop industrial use cases. However, the distinct design and implementation of such a headgear is strongly task-dependent and no one-fits-all solution exists. To overcome that, a computer-aided design process is proposed that allows the design of a device truly tailored to the individual needs. Further, the features and components required to support workers in the identified scenarios are depicted, including image-based worker localization, gaze-based skill-level detection, workflow recognition using head-worn sensors and multimodal assistance. Finally, early functional prototypes are illustrated.
Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, 2019
Due to the explicit and implicit facets of gaze-based interaction, eye tracking is a major area o... more Due to the explicit and implicit facets of gaze-based interaction, eye tracking is a major area of interest within the field of cognitive industrial assistance systems. In this position paper, we describe a scenario which includes a wearable platform built around a mobile eye tracker, which can support and guide an industrial worker throughout the execution of a maintenance task. The potential benefits of such a solution are discussed and the key components are outlined.
The eyes are a particularly interesting modality for cognitive industrial assistance systems, as ... more The eyes are a particularly interesting modality for cognitive industrial assistance systems, as gaze analysis can reveal cognition- and task-related aspects, while gaze interaction depicts a lightweight and fast method for hands-free machine control. In this paper, we present mobEYEle, a body-worn eye tracking platform that performs the entire computation directly on the user, as opposed to primarily streaming the data to a centralized unit for online processing and hence restricting its pervasiveness. The applicability of the platform is demonstrated throughout extensive performance and battery runtime tests. Moreover, a self-contained calibration method is outlined that enables the usage of mobEYEle without any supervisor nor digital screen.
2019 17th International Conference on Emerging eLearning Technologies and Applications (ICETA), 2019
The Airtention project aims at the realization of a pilot training assistance system which embodi... more The Airtention project aims at the realization of a pilot training assistance system which embodies a truly competence- and evidence-based training approach, with the ambition to replace established skill representations as ’hours of experience’ with actual measures and models of level of (i) automation, (ii) task execution accuracy, and (iii) task execution efficiency. For this purpose, a mobile eyetracker will be employed to assess and estimate procedural, behavioral and psychophysiological indicators such as visual attention, perception, cognitive load, interaction with objects of interest, and progress in the workflow to enable deductions about associated skill levels, and adaption of training plans both during runtime and on the scale of a long-term training schedule. Such a system will allow to create a training process which is flexible, adapting to the individual, while at the same time ensuring the quality of the training results.
A human attention estimation system is proposed, derived from automatic analysis of movement and ... more A human attention estimation system is proposed, derived from automatic analysis of movement and behavior of people in front of public displays. For this purpose, new attention metrics defining detailed levels and types of attention are proposed, based on an established attention model which is adapted to the specific use-case. New movement features are introduced which translate detected movements and behavior of people into the novel metric scales. Data capturing is achieved by depth image analysis for tracking of people and extraction of statistical movement data, whereas head orientation of people in front of the interactive system is estimated from RGB video analysis. Classification is carried out for long and short term attention levels, employing a data-driven approach via Support Vector Machines on experimental data derived from an empirical field study at a public exhibition. The system is capable of real-time feature extraction and classification.
Uploads