Emotions Matter: Towards Personalizing Human-System Interactions Using a Two-layer Multimodal Approach

Monitoring and predicting user task performance is critical as it provides valuable insights for developing personalized human-system interactions. Key factors that impact task performance include cognitive workload, physiological responses, and affective states. However, a lack of consideration of any of these factors could lead to inaccurate task performance prediction because of their interplay. To address this challenge, we developed a novel hierarchical machine learning approach that considers these three factors to predict task performance. We exposed twenty-eight participants to a two-step experimental study. The first step aimed to induce different affective states using a validated video database. The second step required participants to perform validated low and high cognitive workload-inducing tasks. To evaluate the performance, we compared the models developed using the hierarchical approach that uses emotional and physiological information, to models that use only physiological information. We observed that our proposed approach always outperformed the models that only use physiological information to predict task performance by achieving a better average person independent mean absolute error. However, information gained across various models using the hierarchical approach was not linear. Additionally, we found that the top predictors for each model varied, and the model with the highest information gain included emotional features. These findings suggest the importance of choosing the appropriate machine learning model and predictors for building robust models for predicting task performance.