Facial Expression Recognition with the advent of face masks

With the worldwide spread of COVID-19, wearing face masks while interaction in public is becoming a common behavior to protect against infection. Thus, how to improve effectiveness of existing facial expression recognition (FER) technology on masked faces has become an urgent issue. However, there are no publicly available masked facial expression recognition datasets that take facial orientation into consideration. To address this issue, we propose a method that can add face masks to existing FER datasets automatically using differently shaped masks according to facial orientations. The FER models based on VGG19 and MobileNet are trained on public and private FER datasets added with mask. As part of our contribution, we collected real-world masked faces from the Internet using emotional keywords and constructed a masked FER test dataset for a fair performance evaluation. The experimental results show that training an FER model based on a simulated masked FER dataset is feasible.