ECG-Based Stress Detection and Productivity Factors Monitoring: The Real-Time Production Factory System

3.1. Biomedical Subsystem

The biomedical subsystem manages the acquisition of body signals and other vital parameters belonging to the S category for fatigue/stress and well-being detection. Vital parameters are collected using a shared Point-of-Care (PoC) installed in the factory, whereas the acquisition of body signals is completed in real time using a wearable device and a smartphone, which runs an ad hoc developed software application (App). All data collected are automatically sent to the middleware exploiting the dedicated API endpoints.

Concerning the real-time monitoring of body signals, the choice of the wearable device is significantly influenced by the considered types of work, especially in the case of manufacturing activities that require specific PPE usage or more in general are characterized by a high operator motion. This leads to discarding solutions such as wristbands and smartwatches due to the high noise affecting the signals, which is caused by the movement of the limbs during work activity; in addition, they could represent a hindrance for PPE such as gloves, thus introducing risks from their use. Instead, the selected sensorized T-shirt, being perfectly adherent to the body of the user, provides good-quality signals even in the presence of movements. In addition, it can be easily worn under the work uniform without impacting worker’s mobility and safety.

The sensorized T-shirt is a COTS device produced by Smartex srl [55]. It is provided with a Software Development Kit (SDK) to facilitate the integration in custom development. The device measures the ECG signal related to the first lead (right arm—left arm) by means of two dry electrodes, the extension of the ribcage during breathing by means of a piezoelectric band, and finally the movement by means of a 9-axis Inertial Measurement Unit (IMU) platform (accelerometers, gyroscopes and magnetometers). In addition to the embedded acquisition electronics, the device is equipped with Bluetooth connectivity for data exchange and a rechargeable battery capable of powering the circuit for the entire work shift. The on-board processing, based on an Atmel microcontroller, also provides derivative signals such as heart and respiratory rate, which can be optionally requested by the user. The complete list of data acquirable from the device is shown in Table 2. Among the supported signals, only the ECG is currently used for stress detection, while the rest of the data are collected to enable further offline analysis and future extensions such as fall or activity detection.

The sensorized T-shirt needs to be paired to the smartphone used for data acquisition and notification management. The chosen smartphone is a commercial rugged device powered by an Android 9 operating system. It features wireless connectivity (i.e., Bluetooth, Wi-Fi and mobile broadband) and storage capability required for the monitoring activity handled by the App.

The App has a threefold objective:

• To handle data acquisition from the T-shirt;

• To act as connecting node toward the middleware, to send acquired data and receive messages, alerts, or commands;

• To present the user with the abovementioned notifications and/or requests coming from local data processing.

The App requires a one-time initial configuration to obtain the identification code (ID) that will uniquely identify the App instance (and therefore the smartphone and the worker to whom the device is assigned) within the RT-PROFASY system. This identifier will be later associated with all the generated data streams and also exploited to address the notification messages to the specific smartphone. As far as the T-shirt pairing procedure is concerned, the App automatically recognizes it among the Bluetooth devices associated with the smartphone and sets it as the active data source. If none or more devices are found, special messages inform the user about the steps to follow for completing the installation. At the end of the configuration and every time the App is launched, the main screen is shown, from which the user can access all the functionalities and start the monitoring.

The monitoring can be carried out in two ways: either in a continuous and automatic manner or when manually requested by the user. In the continuous monitoring mode, acquisitions are started by a timer at fixed intervals, which are defined by customizable settings (by default 60-second-long acquisition every 30 min). Then, there is a button in the App to immediately start data acquisition if an automated one is not in progress. In either case, the App is able to keep the monitoring running even when the App is in the background. This allows the users to lock the smartphone and keep it aside while working. To stop the monitoring, regardless of the chosen mode, a dedicated button is present on the App’s main screen.

In order to minimize the actions requested to the user, the App automatically performs data forwarding to the middleware and manages temporarily Bluetooth disconnections from the T-Shirt, occurring for example in case the worker moves too far away from the smartphone. The only interaction that could be optionally requested to the user, yet not mandatory or blocking, is to provide feedback about the perceived stress level and ongoing working activity at the end of acquisitions. Moreover, to improve robustness and avoid data loss in case of network failures, all incoming data are temporarily stored in the permanent memory of the smartphone and deleted only after a positive transmission acknowledgment is received from the middleware.

The App is also able to receive two types of messages from the middleware (i.e., push notifications), respectively, to convey information resulting from the data processing (e.g., detection of too high stress levels, presence in an interdicted zone, evacuation order, etc.) and to provide additional features such as requesting log or unsent data or changing App settings remotely. The former is presented to the user in the notification bar and as a Toast within the App, showing the text in the message. They are also stored and presented in a history list for later reference. The latter is completely transparent to the user (i.e., no visible notification is produced). Moreover, visible notifications are also directly produced by the App as a result of local data processing and to require user attention in case of feedback requests or unrecoverable errors.

Finally, a set of user-modifiable settings is made available on a dedicated App screen. These settings include the duration of the acquisitions and the interval between them in continuous monitoring mode, the activation of the perceived stress request to the user, and the selection of the types of data to acquire among those provided by the T-shirt. Furthermore, information related to the specific installation (i.e., device ID, sensor name, and software version) is displayed. Figure 2 shows the App’s screens: customizable settings (A), main view (B) and feedback request (C).

The aforementioned functionalities are implemented using the Android SDK API level 28, which is compatible with devices running operating systems version 9 or higher. Figure 3 shows the architecture of the implemented App.

The schema presents, with reference to the typical Android SDK terminology, the main functional blocks of the application; data interchange modes among them are also highlighted. To lighten the notation, processing flows for notifications internally generated or received from the RT-PROFASY system are not shown. In particular, gray blocks represent the device resources used by the App for interacting with the user (touch screen), scheduling automatic acquisitions (alarm manager), saving data, and communicating with external systems (file system and wireless resources, respectively). The main front-end is implemented by a single multi-fragment Activity, which includes graphical elements for presenting status information and messages, and the buttons to perform actions (e.g., start/end automatic or voluntary acquisition, provide stress feedback, access settings, consult notification history, etc.). Its content changes according to the App current status. Data acquisition, local storage, and transmission are carried out by a Foreground Service and two threads, which are dedicated to communication with the T-shirt and the middleware, respectively. Such a service remains active even when the App loses focus or the smartphone is locked. It displays a permanent notification and is destroyed only when the App is intentionally closed. Device permanent storage is used to write incoming data during acquisitions and to read data to be sent during transmissions for preventing data loss in case of errors or connectivity faults. A SharedPreferences component stores configuration parameters that are only writable from the configuration screen but are available for reading to all other components.

Push notifications are implemented by exploiting Firebase Cloud Messaging (FCM) provided by Google. It acts as a message proxy between the App and the RT-PROFASY middleware and requires the integration of a dedicated software module both in the sending and receiving component. FCM supports unicast, multicast, and broadcast message delivery. In addition, to prevent unauthorized operations, the parts involved must share a credential file released by Firebase itself. Firebase exploits a generated token to address notification messages, which is an alphanumeric string identifying a specific device. Once the token is received by the App (at start-up), it is sent to the middleware together with the device ID (the identifier inserted at installation time). In this way, whenever the middleware has to send a message to a specific App instance, it addresses the notification to the token looked up by device ID.

Finally, in addition to the real-time monitoring with the wearable device described above, the biomedical subsystem includes a Point-of-Care (PoC) module for the acquisition of other vital parameters on a voluntary basis. The PoC consists of an Android 9 tablet and a set of commercial Bluetooth biomedical devices. Through a dedicated App, the PoC allows the workers to take measurements of vital parameters such as blood pressure, temperature, oximetry, and weight. These optional data can contribute to the assessment of the health and well-being of workers but are not used for stress detection. The software application requires the worker to log in with a personal username and password. Once signed in, the user can select the desired measurements and follow the guided procedure to perform the acquisitions. Data acquisition and forwarding to the middleware are handled transparently to the user, which is finally informed of the outcome of the operation.

3.2. Machine Learning

The detection of acute stress conditions is carried out by a binary classification algorithm based on Machine Learning (ML), which uses as input the ECG signals acquired during working activities. Conversely, periodic and chronic stress conditions are highlighted in case acute stress episodes are continuously observed over time.

The ECG signal represents the electrical activity of the heart, and its main components are shown in Figure 4. In particular, the QRS complex reflects the electrical activity within the heart during the ventricular contraction; hence, it serves as the basis for classification schemes of the cardiac cycle [56]. Typically, the QRS complex band ranges from 3 to 40 Hz [57], the P-wave ranges from 0.5 to 10 Hz, and the T-wave band ranges from 1 to 20 Hz.

The developed algorithm has to deal with the noise affecting the ECG signals, which is mainly introduced by worker movements and muscle activity. The movement produces artifacts in the measured signal mainly in the 0–5 Hz spectral range [58] due to the sliding of the electrodes over the subject’s body. Whereas, muscle noise is predominant in the band above 35 Hz [59].

Thus, to make usable the acquired ECG signals, a high-pass filter with a cut-out frequency of 5 Hz and a low-pass filter with a cut-out frequency of 35 Hz are used. This results in a strong attenuation of the noise components while keeping the contribution of the QRS complexes almost unchanged. Given the non-diagnostic nature of the system, the limited information loss due to the filtering (i.e., T and P waves) can be endured to acquire a cleaner signal. A comparison of the raw ECG signal acquired under motion conditions and its final filtered version is shown in Figure 5.

In addition, since noise does not affect the signal with the same intensity throughout its duration, a windowing strategy is applied to further reduce the impact of noise on the final classification. The filtered signal is divided into 12-second-long non-overlapping slices, and classification is computed for each of them using a 1D Convolutional Neural Network (CNN). The final classification outcome is elaborated by a majority voting logic in the last stage of the algorithm. In this way, the decision taken on a very noisy segment can be counterbalanced by decisions taken on clearer signal slices, improving the overall performance. Figure 6 shows the block diagram of the stress detection algorithm, where filtering and classification steps are highlighted.

The structure of the CNN is shown in Figure 7. In order to maintain the figure clearly, and since the central layers repeat themselves, the blocks inside the box appear in the actual network six times. The CNN presents 25 one-dimensional convolutional layers with a filter size of 9, which are followed by a dense layer with 1024 nodes. Batch normalizations and Rectified Linear Unit (ReLu) activation functions are used after each convolutional layer. Every two convolutional layers, apart from the first one, a skip connection connects the n-th layer with the n+2-th layer. In order to progressively reduce the input size, every four convolutional layers, a 1D MaxPooling with stride = 2 is applied, and after each skip connection, the next layer also has a value of strides = 2 to compensate for the increase of input size caused by the connection itself. The final dense layer is preceded by a Global Maxpooling and uses a Softmax activation function and dropout normalization.

The supervised training of the CNN was completed in two distinct phases, which are called pretext and downstream task, respectively. The pretext task allowed the network to learn robust generalized features from ECG signals (i.e., high-level abstract representations and recognition of signal transformations) [60]. In the second task, the network was trained to perform the required stress classification.

During the pretext task, the network learnt ECG spatiotemporal representations by recognizing four different types of ECG signal transformations: unchanged, inverted in time, inverted in space, and inverted in both time and space. The dataset used for this scope is Physionet [61], which provides long-duration ECG traces sampled at 250 Hz concerning 11,000 patients. The original signals were divided into 2-min portions; then, time and space transformations were applied. Out of the whole resulting dataset, 75% was used for training, 15% was used for validation, and 10% was used for testing. The model reaches an accuracy of 98.4% for the pretext classification task evaluated on the testing dataset.

The downstream task trained the network to perform stress classification. It used as its starting point the trained network resulting from the pretext task. According to the transfer learning approach [62], the obtained weights of the convolutional layers were transferred to a second identical network, and the final dense layer was replaced by another one of the same type. This dense layer is the only one trained in this task. To this aim, the dataset was built using ECG signals coming from databases publicly available [63,64], a database available for research purposes through access request [65], and signals taken on the field during a preliminary acquisition campaign performed with the RT-PROFASY system itself. In all cases, each trace is labeled with the corresponding perceived stress level.

As reported in Table 3, the final downstream dataset is composed of 93% ECG signals coming from the literature, and 34% of the total is labeled as signals of stressed people. In the case of the signals collected on the field, it was the workers themselves who labeled the traces through the App, according to the perceived stress. As before, the dataset was divided into three parts: training (75%), validation(15%), and testing (10%).

The final network gained an accuracy of 88.4%, with a F1-score of 0.90 in the binary classification of stress. The model is implemented using the TensorFlow library and has a footprint of 25 MB. Since TensorFlow comes also in a Lite version, it will be possible in the future to integrate the model directly into edge devices (i.e., smartphones), so that stress levels’ classification could be completed without sending significant amounts of data to the middleware.

3.3. Building Automation Subsystem

The P data type collection is handled by the Building Automation subsystem, which measures environmental parameters and acquires operational data from the production site’s industrial machines and plants with the aim of environment, process, and safety conditions monitoring.

The system is implemented with Programmable Logic Controllers (PLCs) produced by IEI Integration Corp. and sets of commercial I/O boards and sensors distributed within the factory. The backbone interconnection uses an industrial RS485 bus. In order to increase fault tolerance and implement recovery procedures, the subsystem architecture exploits redundancy and distributed components. It is composed of the Centralized Master PLC (CM-PLC), the Centralized Backup PLC (CB-PLC), and several Local Backup PLC (LB-PLC), which are placed in proximity of the machinery to be controlled. CM-PLC and CB-PLC are embedded Debian systems featuring specialized hardware and running custom software applications implemented for specific control and communication tasks within the RT-PROFASY system. They both have two RJ45 connectors: one is toward the local network (LAN) where the middleware is hosted to enable bidirectional communication and PLC remote configuration; the other is connected directly to the Ethernet RS485 bus bridge communication board (BCB) to exchange data with the distributed nodes connected to the backbone bus. This bridge automatically switches to the CB-PLC in case the connection with the CM-PLC fails, loading the latest setup and parameters. Finally, several LB-PLCs (Raspberry Pi3 boards with a light version of the embedded Debian system and ad hoc developed software) are located close to each machinery to manage the acquisition and sending of data gathered by the local I/O board. Each I/O board has wired connections to the BCB through a double-wired bus RS485 with proprietary FB NET protocol and with the sensors and actuators installed on the machinery.

To the aim of environmental monitoring, the integrated sensors’ and actuators’ sets provide worker presence detection (using InfraRed technology) and measurements of temperature, brightness, and humidity. All installed sensors are identified with a unique ID, which allows the environmental data to be further correlated with stress, well-being, and process performance analysis. Moreover, automatic procedures or manual interventions (using the Workplace UI) allow the adjustment of environmental variables to improve the worker’s comfort.

As far as process control and safety are concerned, the subsystem directly interfaces production machinery and plants. In this way, it is able to acquire instant and overall power consumption estimation of the production processes as well as detect and broadcast warning events triggered, for example, in case of raw materials shortage or operative anomalies (i.e., level below the specific threshold). Nevertheless, corrective actions are not automated and must be performed directly on the involved machine by human operators. In addition, each workstation is provided with dedicated buttons to signal dangerous situations which, when pressed, produce notifications sent to PLCs and then to other workstations and users via middleware.

More in general, data exchange within the building automation subsystem and toward the RT-PROFASY middleware happens either at fixed sampling rates (for analog quantities whose dynamics change slowly, i.e., temperature, humidity and brightness sensors, power consumption) or as soon as the information/event is detected (for digital quantities, e.g., presence sensors, triggers, warning events, changes of state, etc.).

Finally, some post-processing optimization criteria are implemented in the software routines to balance the quantity of data logged and transmitted with respect to the usefulness of the data collected as well as save storage space and speed up research and analysis.

3.4. RTLS Subsystem

The tracking of objects, assets, and the PPE’s position within the production site is managed by the Real-Time Location System (RTLS) subsystem. The availability of positioning data with sub-meter precision and, at the same time, the virtual segmentation of factory spaces into areas, allowed the implementation of software routines to:

• Verify PPE compliance with respect to area-specific rules;

• Verify accesses made to workstations and restricted areas;

• Search for work tools, product batches, semi-finished products, and raw materials;

• Stand by machinery in case of an unattended workstation.

This subsystem is composed of COTS hardware resources produced by Ubisense: a centralized RTLS server, several receiver antennas placed inside the factory at specific coordinates (called references), and active tags placed onto the items to be localized. The smart antennas compute the position of each tag by measuring Time Distance of Arrival (TDoA), Angle of Arrival (AoA), and the strength of the signal (see Figure 8). The localization process happens in real time, and all the positioning data are processed by the RTLS server and immediately made available to the RT-PROFASY middleware.

The RTLS subsystem uses Ultra-Wideband (UWB) technology with a signal strength of 2.4 GHz, which allows for sub-meter accuracy in real time (less than 1 s) and potential scalability over 1000 tags. In addition, the use of UWB technology enables the possibility of deriving the pin-point 3D location, which helps in the detection of dynamic equipment movements, and the identification of people/assets in critical processes. An alternative technology that could be adopted is based on Bluetooth, whose hardware is less expensive but provides worse performance, especially in 3D positioning.

Given the industrial environment, in which signals can be reflected and/or obstructed, antennas may measure incorrect positioning and/or physically impossible movements. In order to detect and remove these outliers, the RTLS subsystem applies advanced Kalman-based filtering algorithms, which distinguish the direct signal path from its reflections.

Overall, the availability of precise positioning data enables the middleware to raise alarms and send notifications to the workers whenever they enter a restricted area or are not wearing the proper PPE, contributing to the prevention of dangerous situations. In addition, it enables the analysis of the correlation between stressful situations and the specific site location, while production times could benefit from the search for assets capability offered by the subsystem.

3.5. Workplace UI Subsystem

The Workplace User Interface (UI) allows the interaction between the worker and the RT-PROFASY system. It is an ad hoc implemented web-based UI (Figure 9) made available in the production site through different types of hardware resources, the choice of which needs to keep both workers and machinery safe. For example, some machines have an industrial Linux-embedded 10” touchscreen panel placed nearby, but when this is not possible, workers can interact with the UI through the same smartphone used for the Biomedical subsystem and/or a shared 10” industrial rugged Android OS tablet. In either case, the connection to the middleware exploits the Wi-Fi network available in the factory.

This UI also provides useful information to better carry out the job, for instance showing the list of activities, the bill of materials required, tracking the status of single activities, and entire processes by recording their start/end time, etc. It also allows reporting problems that occur during production activities, including interruptions or discarded products. Overall, the information contributes to the “P” data type by monitoring the evolution and outcomes of processes and coordinating subsequent production phases.

In addition, this subsystem optionally allows the visualization of the environmental parameters measured in the proximity of the workstation and, where available, to modify them by acting on the actuators of the Building Automation subsystem.

For the purposes of safety, through the Workplace UI, the workers could receive immediate notification about dangers/fail machinery events raised by the local machine or elsewhere on the production site. Similarly, they can signal emergencies to all their colleagues.

Finally, this subsystem provides functionalities related to RTLS components such as:

• Search for the location of PPE required to operate the associated machine or enter a specific area;

• Search for the location of assets and materials needed to accomplish the assigned activity;

• Receive timely notification if PPE is missing.