A Survey of Multimodal Perception Methods for Human–Robot Interaction in Social Environments

Red–green–blue (RGB) [1, 2, 20, 21, 28, 36, 42, 52] and RGB-depth (RGB-D) cameras [29, 44, 47, 80, 109, 120, 121, 150, 154] were the most commonly used visual sensors in surveyed works. Universal Serial Bus webcams (e.g., the Logitech HD Pro C920 [149]) can provide RGB data [76], while other commercial RGB cameras are available in the Microsoft LifeCam Studio HD [123], Point Grey FIREFLY [108], and from Omron [87]. Additionally, many of the social robot platforms surveyed have embedded RGB cameras, such as Pepper [17, 71, 72, 105, 106, 143], Nao [71, 72], Nico [35], and Robotinho [116]. The Microsoft Kinect RGB-D camera was the most commonly used sensor in the entire survey [11, 12, 18, 27, 38, 43, 54, 75, 86, 87, 98, 117, 124, 126, 156, 164, 180, 181, 182]). Other RGB-D sensors in the surveyed literature include the ASUS Xtion Pro and its variants [34, 97, 123, 176], the Intel RealSense D435 [45, 46, 153], the Intel RealSense SR300 [126], and the Orbbec Astra Embedded S. Less common visual sensors include stereo cameras [44, 47, 120] (one of which is embedded in the iCub robot [14, 57]), time-of-flight depth cameras [114], the FLIR Lepton long-wave infrared camera [41], and the LeapMotion LEAP controller—a stereo infrared camera system designed specifically for hand tracking used in [101, 127].

Surveyed works used various commercial, open-source, and custom techniques to process visual data and compute relevant HRI information.

Body, head, and hand pose estimation were common functions in surveyed works. Skeletal pose tracking is offered natively through the Kinect API as in [18, 27, 38, 86, 110, 127, 156, 164, 180, 181, 182] and provides spatial locations of selected joints (torso, shoulders, head). OpenPose [24] is another means of estimating human poses, gestures, and gaze via keypoint extraction. OpenPose is open-source, platform-agnostic and accepts any RGB or RGB-D image as input, as used in [28, 42, 153]. The Pepper robot’s software development kit (SDK) supports human localization and motion estimation and was used for body and head motion estimation in [143]. The OpenNI framework is another open-source pose recognition option and provides skeletal pose tracking and hand point tracking as used in [34, 43, 110, 167]; however, its last official update was in 2012. Banerjee et al. [11] and Foster et al. [45, 46] implement pose recognition using convolutional pose machines (CPMs). Head pose can be computed with the open-source OpenFace framework as in [17, 76, 110] or OpenHeadPose as in [45, 46]. Foster et al. use RGB least squares matching to estimate head position [44, 47, 120]. Ragel et al. use the MediaPipe tracker to get hand pose keypoints from RGB-D data in [127]. Lastly, the LEAP provides hand pose tracking natively as in [101, 127].

Pose information can be used to locate humans, recognize gestures, and infer their focus of attention. Chau et al. [28] use OpenPose to compute spatial pose keypoints of nearby humans from a mono RGB camera input; they compute the human’s 2-dimensional (2D) location by projecting the 3-dimensional (3D) keypoints into a 2D plane around the sensor platform. Likewise, Terreran et al. [150] compute skeletal pose keypoints of an RGB-D image with OpenPose, then use 2D-to-3D lifting and projection to project 2D keypoints onto the depth channel, providing a 3D pose estimate. Notably, OpenPose also provides a confidence measure which can be used to weight the visual pose detection in multimodal fusion applications as in [28]. The Hobbit elderly care robot computes a 3D bounding box around skeletal pose keypoints, then uses this spatial information to detect if the human has fallen [43]. An even simpler use of skeletal pose tracking is to estimate the number of humans in the scene by counting the detected skeletal poses, as Jain et al. did [76]. Simple gesture recognition can be accomplished via geometric modeling. For example, Kollar et al. [86] recognize the “hand raised” gesture if the wrist is above the shoulder, and Whitney et al. [167] compute the shoulder-to-wrist vector to recognize pointing gestures. OpenNI includes basic gesture recognition (swipe left/right, pointing) [43]. Supervised learning techniques can also be used for gesture recognition; Trick et al. use an SVM to classify OpenPose joint trajectories as gestures [153]. Paez et al. use skeletal pose recognition to classify “emotional gestures” such as stretching arms, hands on chin, or rubbing hands together [126]. Chao et al. use a geometric model to compute the angle between a human’s head and shoulders relative to hips, then use this to infer if the human is facing the robot or not [27]. Features extracted from pose data can also infer emotional state; Filntisis et al. use a DNN to extract affective features from children’s skeletal pose data, which is fused with facial features to classify their emotional state [42].

Detection of human centroids is another visual processing technique found in the literature [10, 12, 20, 21, 36, 44, 47, 49, 80, 87, 97, 98, 120, 124, 154, 176] and typically involves supervised learning of labeled datasets while leveraging both depth and color from RGB-D data. Models such as DarkNet53 can fuse RGB and Depth features, then YOLOv3 can be used to detect 3D human centroids or 2D bounding boxes from the fused RGB-D features as in [98]. Fung et al. [48] use a multimodal YOLOv4 (MYOLOv4) implementation to extract and fuse RGB and Depth features to robustly detect 3D human body part positions in offline datasets. Similarly, Martinson et al. use AlexNet to localize humans in RGB, plus a custom depth-based segmentation, geometric feature extraction, and Gaussian mixture model (GMM) classifier to localize humans from the Depth channel [109]. They fuse RGB and Depth detections using a weighted sum and compare the fused performance with individual modes. Segmenting the depth channel into foreground and background, then using a depth-template or histogram of oriented gradients (HoG) upper-body detector on candidate regions was done in [80, 97, 176, 154] for near-field torso detections; far-field detections used a HOG detector on RGB data since depth data is unreliable beyond approximately 5 m [80, 97, 176, 154]. Komatsubara et al. [87] use a custom head-and-shoulders shape detection from RGB-D data; likewise, the JAMES bartending robot [47, 44, 120] uses 2D RGB silhouettes and 3D depth information to estimate 3D shoulder locations.

Nakamura et al. [114] localize humans using thermal and depth data in a Thermal-Distance integrated localization model which uses separate modalities to reduce false positives. First, a binary mask is applied to the thermal points to separate hot (above temperature threshold and presumably human) from cold pixels. Hot pixel regions are then mapped onto the depth image, and the background depth is subtracted. The resultant clusters of 3D points are considered human.

Once detected, body centroid locations can be used to localize and track humans or estimate the person’s focus of attention. For example, Linder et al. [97] and Yan et al. [176] project 3D torso detections onto a 2D plane surrounding a mobile robot, then fuse it with a LiDAR leg detections in a multihypothesis tracker to localize and track multiple humans around the robot. The JAMES bartending robot [44, 47, 120] computes the angle between a human’s torso and the robot to infer if the human is facing the robot.

Facial detection on RGB images is another common function in surveyed literature [54, 122]. Face detection provides a 2D estimate of face locations in the RGB frame and is supported by open-source packages such as OpenCV as in [18, 149, 164], OpenFace as in [42, 76, 102, 110], face\(\_\)recognition⁴ as in [127], or the Pittsburgh Pattern Recognition (PittPatt) SDK as in [108]. The most common facial detection algorithm was the Haar feature-based cascade classifier, also called the Viola-Jones algorithm [12, 35, 96, 116, 123, 165]. Haar-like features are simple geometric classifier patterns which, when cascaded into stages of increasing complexity, can rapidly search an image and detect complex visual patterns like faces. The algorithm is popular for real-time usage on robots and embedded systems because of its speed and simplicity. Other facial detection techniques include deep networks [14, 59] (which can also infer gaze direction as in [11]) or custom blob tracking of facial regions [44, 47, 120]. Both Martinson et al. [108] and Portugal et al. [123] project the 2D facial detection onto a 3D depth image to estimate the 3D location of the detected face. The Pepper robot’s SDK supports facial detection natively [106, 105].

Detected faces can be further processed to infer relevant HRI information such as identity, emotion, gaze, or soft biometrics. Commercial options for identification via face recognition include SoftBank Robotics’ NAOqi recognition module in [71, 72], and OKAO Vision, which computes Gabor wavelet transform features of each face and classifies them using an SVM [52, 87]. Dlib is an offline, open-source facial recognition option [110]. Other surveyed works implemented facial recognition by classification of local phase quantization [149] or local binary pattern (LBP) features [35, 118]. Commercial options for facial emotion recognition include FaceReader from Vicar Vision [99, 152], Microsoft Cognitive Services’ Emotion Recognition API [102], the Affdex SDK [126], and the eMax face analysis toolbox [95], which require a cloud connection. Supervised classifiers can also estimate emotion; Prado et al. [125] use principal component analysis (PCA) to extract features from facial keypoints (eyebrow, cheek, chin, mouth, etc.) and classify emotion via a dynamic Bayesian network (DBN), and Paez et al. use facial gestures (turn head, flexion, close left/right/both eyes) as features to indicate emotional state. Cloud-based age and gender demographic estimation are available through FaceReader, Microsoft Cognitive Services’ Face API [127], Pepper’s SDK [105, 106], and SoftBank Robotics’ NAOqi framework recognition module. OpenVINO has pre-trained models to estimate emotion and age/gender demographics without a cloud connection, as described in [110]. Gaze estimation is available offline via OpenFace [17, 76, 110, 168] and GazeSense [121]. Banerjee et al. use a cascaded deep network to estimate gaze and face location [11]; the SPENCER robot [80, 154] also uses a supervised classifier—a SVM that classifies difference of oriented Gaussians features in an RGB image—to estimate approximate gaze direction (left, right, back, front). The virtual assistant of Bohus et al. [20, 21] uses a gaze model trained on labeled images to infer if nearby humans are looking at or away from the system. Face detections can also be used to infer soft biometrics such as a human’s approximate height as in [71, 72, 108], although the location and orientation of the camera relative to ground plane must be known. Martinson et al. also use color histograms of the face as soft biometrics [108].

Miscellaneous facial image processing in the surveyed literature include facial action unit computation, which can be done by OpenFace [17, 45, 46, 76] and used to classify emotion. Filntisis et al. use ResNet 50 to extract facial emotion/affective features, which they fuse with DNN-extracted features from skeletal pose data and classify via a fully connected neural network (FCNN) layer to estimate child emotional state [42]. Gomez et al. compute mouth movement activity to estimate if a human is speaking [54]. In one of the more unique applications of early fusion, Filippini et al. [41] used a fused RGB-Thermal image to locate a face using an RGB HOG detector and a regression tree ensemble to locate RGB facial keypoints; a FIR filter extracted the associated thermal features of the keypoints, which were then classified using a multi-layer perceptron (MLP) to estimate children’s emotional states.

A less common visual processing technique was hand detection and hand gesture recognition. Abioye et al. use an OpenCV Haar cascade to locate hands visually, then a convex hull to count fingers [1, 2], where each gesture indicates a specific robot command. Jacob et al. use a custom fingertip detector; the trajectory of each fingertip is smoothed using a Kalman Filter, and the trajectory’s speed and curvature are classified using a hidden Markov model (HMM) classifier [75], where each gesture corresponds to a specific command to a robotic assistant. Foster et al. use a custom blob tracker on the JAMES bartending robot to locate nearby humans’ hands, which are used as features to estimate the human’s overall status. Chao et al. compute relative motion of tracked hands and use this to estimate gesture activity (but not specific gestures). Gesture activity is then used to estimate overall human activity and engagement [27].

Aside from the capabilities listed above, visual data features can be classified via supervised learning techniques to recognize gestures and activity or fused with features from other modalities for multimodal classification. Feature extraction techniques in the surveyed works consisted of scene flow [139], space-time interest points [91, 139], or visual embedded vectors [70]. Processed visual data can be classified as a specific gesture using HMM and SVMs [139], LSTM networks, or transfer-learning trained networks [101]. Similarly, multiple surveyed works compute dense trajectory features, using histogram of optical flow and motion boundary histogram descriptors, which are then encoded via a bag-of-visual words or vector of locally aggregated descriptors framework and classified via an SVM [38, 83, 156, 187, 186] to detect gestures for children and the elderly. Liu et al. use attention layers to extract RGB spatial features and LSTM layers to extract skeletal pose features, which are concatenated and classified with an FCNN to estimate human activity [100]. Chu et al. use motion cues as features; motion cues are a simple geometric computation between successive joint positions and angles and are used to detect activity [34]. Islam et al. use ResNet 50 to extract spatial features from RGB and Depth images and LSTMs to embed the temporal features. The features are fused in a custom multimodal self-attention mechanism which classifies the activity seen in the RGB-D image [74]. Similarly, Robinson et al. [134] use a transformer network to extract, fuse, and classify activities of daily living (ADL) of elderly persons in RGB-D video. Reily and Lu compute HoG features for RGB, depth, and thermal images, then fuse them in a bag-of-words representation for each human. The individual and group activities are then inferred based on a weighted linear combination of these features [103, 131]. The mobility robot (MOBOT) uses OpenCV to compute optical flow variables in an RGB image, which is used to detect regions that contain activity [83, 187, 186]. Chen et al. [29] extract local RGB features using an HRNet backbone and global features using a CNN; these features are concatenated with a LiDAR feature vector to robustly estimate human pose in visually obscured conditions.

Supervised classification of visual features can also infer engagement or group affiliation. Vaufreydaz [164] and Benkaouar [18] use skeletal pose data to compute the distance from human to robot and Schegloff metrics—pose features based on computing the relative positions between hips, torso, and shoulders keypoints—then fuse these features with other modalities to classify if the human intends to engage with the robot. Banerjee et al. compute body position and orientation from tracked pose keypoints, then fuse these with head orientation, gaze direction, facial bounding box features, and audio cues. These fused features are input to multiple supervised classifiers that estimate a human’s interruptability, or if they are willing to engage with the robot [11]. Nigam et al. simply use grayscale values from RGB pixels and fuse them with audio features to estimate the overall social context and if humans are willing to engage with the robot [117]. Bohus et al. analyze the variance in the RGB pixel values of each human’s clothing to infer group affiliation; business attire (e.g., suits) resulted in higher RGB variance, and the person was assumed to be a visitor since members of their organization dressed casually [20, 21]. Likewise, RGB color histograms of clothing can be used as soft biometrics to help discern identity as in [108].

Most of the surveyed systems employed multiple visual processing techniques simultaneously. For example, the virtual agent/humanoid robot system of Yumak et al. [180, 181, 182] uses several visual processing techniques to localize humans, identify them, and estimate their gaze direction. For localization, they use Kinect’s skeletal tracking which locates joint keypoints. Similar to Bohus et al. [20, 21], Yumak’s system uses feature extraction and classification of a set of visual pixels to identify humans. LBP and hue saturation value (HSV) color features [166] are extracted for a patch of pixels around each skeletal joint. The LBP and HSV features are concatenated into a single vector (feature-level or intermediate fusion) for each person; a k-nearest neighbor (k-NN) classifier matches the visual features to an identity. To estimate gaze direction, they use a regularized maximum likelihood deformable head model fitting algorithm [23]. The rotation matrix and the translation vector of the head model are iteratively optimized using a method similar to iterative closest point, and then used to estimate the pose of the head. Head pose is then used to infer focus of attention as described in Section 3.9.

In surveyed MMP HRI applications, a computer audition hardware setup is typically comprised of an array of multiple microphones when SSL is performed. This includes commercial products like the Microsoft Kinect’s 4-channel microphone array, used in [12, 18, 38, 44, 47, 124, 156, 164], the 2-channel array in the Asus Xtion Pro Live [123], the 7-channel Microcone [12], or the 8-channel TAMAGO-03 [28]. Other surveyed works used custom arrays of 4 linear microphones [20, 21], custom 6-channel arrays [127], or 6–8 microphones organized into pairs [10, 36, 49], circular arrays [180, 181, 182], or 8-channel linear arrays [83, 187].

Some surveyed social robot platforms contain embedded arrays in support of audition functions. Pepper has a 4-channel array embedded in the head [17, 45, 46, 143], and Curi has two microphones embedded in the body [34], while Honda Research Institute’s Hearbo has a 16-channel array [54, 114]. Binaural microphone arrays are another common configuration and are usually found on humanoid or anthropomorphic robots such as the SIG2, NAO, or iCub systems [14, 57, 148, 179]. Eight to sixteen total audio channels may be desirable if a high degree of noise cancellation or SSL accuracy is required [67, 69, 140, 159]. The largest array in surveyed works was comprised of two 8-channel arrays and two 16-channel arrays, used for precise SSL in an indoor setting [73].

Mono-channel microphone systems were used in surveyed works where SSL was not required, since sound sources cannot be localized using a single audio channel. This configuration was used in applications when the robot is interacting with a single person, such as: the UAV control interface of Abioye et al. [1, 2]; the robot of Churamani et al., which uses a single audio for voice identification and speech recognition; the Curi robot, which uses audio cues from a single human to determine engagement [27]; robots which play joint games with a single human, as in [76, 121, 168]; or Nigam et al., who analyze monochannel features to classify the audio scene [117]. The virtual agent/humanoid robot system of Yumak et al. [180, 181, 182] also used a single standalone microphone. A unique monochannel audio implementation is the Roboceptionist robot [86], which used an Android tablet’s microphone for speech recognition.

SSL was a common computer audition function in surveyed works, and various SSL algorithms exist for practical use on robots and intelligent systems. Several surveyed works [12, 28, 54, 73, 114, 124, 127] used variants of multiple signal classification (MUSIC) techniques (such as generalized eigenvalue decomposition-MUSIC or generalized singular value decomposition-MUSIC) for SSL in their applications. All were implemented using HARK⁵ middleware (Honda Robot Institute—Japan Audition for Robots with Kyoto University) [113]. HARK includes ambient noise filtering via extended minimum variance distortionless response, as in [127]. Note that MUSIC requires an estimate of the transfer function between the audio source and microphones, which becomes increasingly complicated for multidimensional SSL. Hence, several works [20, 28, 114, 124] computed the sound source direction of arrival (i.e., 1-dimensional (1D) SSL) in 5–10° increments. Notable modifications include Nakamura et al. [114], who include sound source identification via a hierarchal Gaussian mixture model to ignore non-speech sources, and Chau et al. [28] who extract features via DNN-based spectral mask estimation to increase noise robustness.

Other SSL techniques include the one used by D’Arca et al. [36], who compute the intra-channel time differences of arrival using generalized cross-correlation with phase transform (GCC-PHAT) and then use beamforming techniques to localize speakers indoors. Beamforming techniques were used for other indoor speech localization applications, such as the child–robot interaction systems in Tsiami et al. [156] and Efthymiou et al. [38] that used a steered response power with phase transform (SRP-PHAT) beamformer, and the MOBOT’s perception platform [83, 187]. Likewise, Yumak et al. [180, 181, 182] use a speech harmonicity beamforming algorithm [170] to compute speech direction-of-arrival from an 8-channel circular microphone array. Beamforming SSL and SSS can be implemented using open embedded audition system, another free, open-source computer audition package [62]. Gebru et al. [49] use a supervised learning approach; they extract the auto- and cross-spectral density features between two binaural channels, then train a localization model using a labeled multimodal dataset. Likewise, Foster et al. incorporate SSL on a Pepper robot by using a supervised neural network [45, 46]. Belgiovine and Gonzalez-Billandon et al. also use a DNN for coarse 1D SSL, classifying the direction-of-arrival (DoA) (or azimuth) into one of three zones in front of the robot [14, 58]. Lastly, the Microsoft Kinect can compute the sound source DoA through its API, as done in [18, 44, 47, 164], or used within a networked array of multiple sensors as in [186]. For additional details on the categories, algorithms, and implementation of SSL, we recommend Rascon and Meza’s survey [130].

ASR is the second most common audio processing function in the surveyed literature, with a range of techniques surveyed. For dialog-heavy, cloud-connected applications, commercial options include Microsoft Cloud speech recognition [20, 21, 121], Loquendo (now Dragon from Nuance) [122, 143, 116], iFlyTek Speech API [102], and Google Cloud’s speech-to-text API [35, 45, 46, 95, 105, 106, 127] (which has a Python software wrapper [149]). Offline, open-source ASR options are available in the Kaldi deep learning model⁶ [99, 152], DeepSpeech⁷ [168], or in the Carnegie Mellon University (CMU)Sphinx framework⁸ [1, 2, 75, 123]. Julius⁹ is another open-source ASR framework which can be used offline and, along with Kaldi, features HARK integration. Audio feature extraction and supervised classification can also be used to generate custom acoustic and language models for ASR. These types of ASR models are typically used on embedded systems where large vocabulary recognition is not required and a smaller set of words or commands is sufficient. The most commonly used audio features are mel-frequency cepstral coefficients (MFCCs), which approximate the nonlinearities of human hearing in the frequency domain. MFCC computation is a well-documented process, included natively by HARK and other open-source audio processing software like Torchaudio. A 2D data tensor with MFCCs in one axis and time in the other can be visualized as a spectrogram and classified using supervised learning methods to infer the phonemes spoken. Sanchez-Riera et al. [139] extract MFCCs from speech, then classify the MFCC vector as a discrete robot command using HMMs and SVMs. Likewise, Liu et al. [101] classify MFCCs as a discrete set of spoken commands using a CNN. Similarly, Trick et al. compute the MFCCs of speech audio, then use the PyTorch-based Honk framework—a supervised CNN classifier—for small vocabulary speech recognition [153]. Tsiami et al. [156] use extracted MFCCs and their first temporal derivative (MFCC+\(\Delta\)) features as inputs to a Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) to phonemes in a speech recognition system. Kardaris and Zlatintsi [83, 186, 187] use a similar approach, extracting MFCC+\(\Delta\) features and classifying them using an N-best grammar-based language model using the HTK toolkit.¹⁰

Audio feature extraction and analysis can also be used to infer emotion, speech activity, personality traits, or identity. Open Emotion and Affect Recognition toolkit provides an online means of classifying emotional content in audio [123]. Offline options include Ma et al. [104], who extract and classify MFCCs to detect emotional content of speech using a trained supervised classifier model. The Praat toolbox can extract signal information like pitch, frequency, intensity/volume, harmonicity, and duration; Prado et al. use Praat-extracted features as input to a DBN which classifies the emotional content of the speech. The openSMILE toolbox can also extract audio features for emotional classification as in [95, 110], or for engagement detection as in [17]. Jain et al. use Praat-extracted audio features to classify engagement level. Sound cues—the difference in amplitude between successive samples—are a relatively simple audio feature which can also be used to infer human engagement as in [34]. Chao et al. use a Pure Data module to compute pitch [27]; Maniscalco et al. perform root mean square analysis of the audio signal [106]; both then use these features to detect speech activity. The Pepper robot used in the MuMMER project uses a supervised neural network to detect speech activity [45, 46]. Shen et al.’s Pepper implementation extracts MFCCs, voice energy and voice pitch, which are fused with visual features to estimate the human’s “Big Five” personality traits. Speaker identification (also called voice identification) through audio data can be accomplished via use of a CNN classifier as in [35], an implementation of i-vector feature classification in the ALIZE 3.0 framework [149], or by MFCC computation and GMM classification, trained on a 60-second speech sample for each speaker [36]. In [35], the authors note that MFCCs may be unreliable in noisy HSEs, so learned CNN features may be more suitable and also avoid the additional preprocessing step of computing MFCCs.

Lesser-used audio processing techniques include social context classification and lexical feature extraction. Nigam et al. use the signal amplitude in the time domain as a feature to estimate social context in HSEs; i.e., if the HSE is used to eat, study, or socialize [117]. Lexical feature extraction can be accomplished with OpenCCG, which provides embeddings of recognized speech as in [44, 47]; embeddings can reduce a large vocabulary of spoken words into the most likely command or words relevant to a service robotics domain.

Model-based fusion techniques are categorized by the type of model used to fuse data streams [9]. Recall from Section 3.2 that this includes Multiple Kernel Fusion, Graphical Model Fusion, and Neural Network Fusion. Despite being derived from well-established ML algorithms,¹² model-based fusion techniques are less common in robotics; model-based fusion techniques often require large amounts of relevant data and onboard compute to be used in real time on embedded systems. However, advances in onboard computing and ML research have resulted in model-based fusion techniques being employed on robots in recent years.

Graphical Model Fusion can combine asynchronous data from different streams, and both training and inference are faster than neural network model fusion [72]. Additionally, graphical models can continue to provide a state estimate based on prior information, even if updated information is sparse. Graphical models can also model states that change temporally. These are all desirable qualities for onboard robotic sensor fusion. Examples of Graphical Model Fusion include Irfan et al., who use a multimodal incremental Bayesian network to fuse facial identity, height, age, and gender information to estimate a human’s identity incrementally over time [71, 72]. Banerjee et al. compare the performance of graphical models such as HMMs, CRFs, hidden CRFs, and latent-dynamic CRFs to categorize interruptability based on pose, gaze, and audio signals [11]. Lastly, Markov decision process (MDP) graph models can be used as dialog management or task planning engines. Lu et al. use a Partially Observable Markov Decision Process (POMDP)-based state estimator and event reasoner to fuse multimodal inputs [102], while the SPENCER robot uses a Mixed Observability Markov Decision Process (MOMDP)-based collaboration planner to estimate human intentions from asynchronous multimodal inputs [80, 154]. Notably, the MOMDP collaborative planner is just one component of SPENCER’s supervision system, an integrated perception and planning system that affects specific actions based on user inputs and perceptual data.

Neural Network Fusion was also used in surveyed works to align and fuse multimodal features. Neural Networks are often used within an end-to-end fusion and classification/prediction framework, where one layer is used to fuse data or features, and another is used to classify or decode the feature representation. Linder et al. use the DarkNet-53 network to fuse RGB and Depth features; the fused feature vector is then classified with a modified YOLOv3 network to compute human centroid location in the Depth image and 2D bounding box for the RGB image [98]. Fung et al. [48] use a custom YOLOv4 variant—MYOLOv4—to individually extract RGB and Depth features and fuse them to robustly detect people and body parts. Robinson et al. [134] use ResNet to extract RGB-D video features, a graph convolutional network (GCN) + Self-Attention network to extract pose features, and a CSPDarknet + PANet + YOLO network to extract object location features. A Spatial Mid-Fusion Module then embeds each modality’s features into a feature vector, which is classified by a Dense Neural Layer to estimate which ADL is occurring in the RGB-D video. Li et al. use a bidirectional gated recurrent unit (BGRU) to fuse audio, lexical, and facial features before classifying the fused feature vector with a FCNN to infer the human’s emotion [95]. Chen et al. [29] use a Transformer integrated module to fuse global RGB image and mm-Wave Radar pointcloud features into a multimodal feature vector. The feature vector is then classified by a fusion Transformer module, which can selectively weight the most salient features for more robust human detection when vision is occluded. G. Liu et al. [100] compute spatial RGB features using cross-modal and self-attention mechanisms and temporal features from skeletal pose data via bidirectional LSTM networks. The spatial RGB features and temporal pose features are fused via a concatenation layer and classified via a FCNN to estimate the human’s activity. Yasar et al.’s IMPRINT network [178] uses Neural Networks and Transformer models within an end-to-end feature extraction, fusion, and decoding network on RGB-D data. Unimodal feature extractors compute spatio-temporal features for skeletal poses in the scene, and a GRU encoder extracts contextual features from the scene. The pose and contextual features are fused with an Interaction Module that also estimates group affiliations. A multimodal context module fuses the processed group, individual, and context features. Lastly, the fused and processed feature embedding is decoded by a Motion Decoder to estimate where each person in the scene will move. In each case, Neural Network Fusion can synchronize asynchronous data streams and leverages mutual information between individual modalities.

Our survey found no instances of Multiple Kernel Learning models used in multimodal fusion.

As discussed in Section 3.2, model-agnostic fusion techniques can be categorized as Early (Data) Fusion, Intermediate (Feature) Fusion, or Late (Decision) Fusion depending on when the data are fused [25, 128, 180].

Early fusion involves the synchronization and alignment of raw, independent data streams into a single structure. Many robots or embedded systems feature asynchronous, heterogeneous data streams, making early fusion challenging; it is unclear how a 2D image can be merged with a 1D time-indexed dataset. As noted in [13], data synchronization and alignment is relatively expensive computationally. As such, early fusion is generally not feasible for real-time applied robotics or embedded systems applications, and we only found one project that employed early fusion: Filippini et al. [41] performed extrinsic calibration to find the correspondence between an RGB and a thermal camera. They generate a fused RGB-thermal image by converting thermal pixels into corresponding RGB pixels and linearly interpolating the sampled value if thermal sample time falls between RGB sample times. Similarly, the PST900 dataset [144] uses an extrinsic calibration technique to align RGB and thermal images.

Intermediate or feature-level fusion is generally used when mutual information is conveyed across multiple heterogeneous modalities simultaneously. Specifically, separate audio and vision sensors may detect mutual information about the same object or event. Hence, a fused audiovisual feature may be more informative than separate unimodal audio and visual features.

Intermediate fusion is typically accomplished through alignment, summation, and/or concatenation of unimodal feature vectors. H. Liu et al. [101] use intermediate fusion to perform multimodal command classification for a collaborative human–robot manufacturing task. They use artificial neural networks (ANNs) to extract features from the human’s speech, gesture, and body movements. The features are fused into a single vector, then classified using a MLP as one of six potential commands. Ma’s emotion recognition network [104] functions similarly; CNNs are used to extract features from audio and visual data, which are then fused into a single tensor via an MLP which is classified using an SVM. Ben-Youssef et al. record asynchronous proxemic, gaze, head, and speech measurements, extract features, and fuse the features via temporal pooling of sliding temporal measurement windows [17]. Benkaouar and Vaufreydaz’ domestic robot [18, 164] fuses various spatial, speech, and skeletal pose features from asynchronous sensors; they use neutral values when one of the features is unavailable, and they use the last known feature value for features that are not computed at the current timestep. The fused feature vector is classified via an ANN and SVM to estimate engagement. Lu and Reily [102, 131] compute HoG features for multiple sensor modalities, then represent the fused features in a bag-of-words for each person in a group. Each human’s fused feature vector is multiplied with a weighted matrix and summed to estimate the activity of the group. Vector concatenation is another intermediate fusion technique, used by Filntisis et al. [42]. They extract facial and pose features using Resnet 50 and DNN, respectively, then concatenate both into a single feature vector that is classified by a FCNN to estimate human emotion. Similarly, Islam et al. [74] extract features from RGB and skeletal pose modalities using a self-attention mechanism; they evaluate intermediate fusion using both concatenation and summation of the feature vectors. The fused vector is then classified by a FCNN to determine the human’s activity. Using PCA and data concatenation, Nigam et al. fuse grayscale values from RGB data and signal intensity from the audio channel; they then use SVMs, decision trees, and naïve Bayes classifiers to infer social context [117]. Shen et al. [143] use linear interpolation and truncation to align variable-sized visual and audio feature vectors temporally, which are then concatenated into a single vector and classified. Wilson et al. [168] employ intermediate fusion to combine gaze and speech features into a single feature vector. Since speech commands are sparse and asynchronous, they use a linear degradation function to progressively reduce speech feature weight over time if speech features are not readily available. The fused feature vector is then classified with a random forest to estimate if the user needs assistance.

Model-agnostic late fusion techniques were the most common category of fusion in the surveyed works. We found examples of late fusion implemented using simple arithmetic or logical comparison, probabilistic and Bayesian methods, action engines and state machines, data association, and supervised classifiers as described below.

Simple late fusion can be accomplished through mathematical operations on unimodal data streams. Weighted summation was one common late fusion technique, which was used for human identification and command/intent recognition. Tan et al. and Churamani et al. compute human identity through a weighted sum of voice and facial identification confidence values [35, 149]. Similarly, Martinson et al. compute how closely a human’s face, clothing, and height match a database of known values; the weighted sum of these similarity metrics is used to select the most likely human identities [108]. Pourmehr et al. [124] independently track humans using Kalman Filters for visual, audio, and LiDAR input data, generating three separate occupancy grids each representing the likelihood of humans present. The fused output is simply a weighted average of the three independent modalities’ occupancy grids. Sanchez-Riera et al. [139] also used weighted averages of individual modes to classify human commands. A visual gesture classifier and a verbal speech SVM classifier each estimate which command was issued (“Hello,” “Bye,” “Yes,” “No,” Stop,” “Turn”); the two modes are weighted and summed to generate a posterior estimate of the command. The MuMMER robot computes an “interest likelihood” from head pose, gaze, and location data, then computes a weighted sum to infer who is likely to interact with the robot [45, 46].

Logical operations on unimodal data streams are another way of affecting simple late data fusion. For example, the speech and gestural interface in [1, 2] will fuse the two command inputs only if they arrive within 0.5 seconds of one another and contain complementary information; redundant commands are discarded. Ishi et al.’s LiDAR and microphone system will only fuse human location measurements if they are within a distance threshold [73]. The audiovisual speaker localization system in [54] takes sound source location, face location, and mouth movement as inputs; the speaker location is only returned if mouth activity is detected near the sound source. Lastly, the speech and gestural interface in [83, 187] computes confidence values for recognized speech and gesture commands, then ranks all received commands and selects the command with the highest overall confidence.

Projection and/or lifting is another means of late fusion and can be used to combine processed data from one modality with raw data from another. Terreran et al. detect 2D pose keypoints of an RGB-D image using OpenPose, then use 2D-to-3D lifting and 2D-to-3D projection to fuse the 2D pose keypoints to the depth channel of the RGB-D image. The result is a 3D pose estimate. Due to the inherent processing time in the OpenPose process, this is best suited for offline operations, and the authors of [150] use this method on datasets.

Action engines and managers are another means of handling processed outputs of multiple modalities and work well for HRI applications when actions, states, and state transitions can be explicitly defined. The simplest case is to execute actions based on any commanded input as they are received—such as processed touch screen, gesture, or verbal commands—as done by the Hobbit service robot in [43]. More involved dialog, action, and service managers implement execution rules and reason about potential courses of action before execution. Roboceptionist’s dialog manager uses rules to determine when to process verbal and gestural commands and when to respond. It also parses the semantic content of human speech to select the most appropriate response given the current dialog state [86]. Based on visual and speech inputs, R3D3 also uses a dialog manager to manage initiative and conversation flow and determine which museum exhibit a human wants information about [99, 152]. The SocialRobot platform allows developers flexibility by allowing them to implement custom action descriptions and behaviors in an XML file that is managed by the robot’s action engine [123]. Efthymiou et al. use a dialog flow manager, which monitors for specific speech or gesture inputs to select the next robot action in a series of child–robot interaction games and scenarios [38]. Chao et al. use a timed Petri net architecture to asynchronously handle visual and audio stimuli to determine when to interact with a nearby human [27]. Many surveyed works implemented action engines or dialog managers using state machines, where state transitions are affected by specific perceptual inputs. For example, Maniscalco et al. implement a behavior manager on the Pepper robot using SMACH¹³ state machine software; Pepper will not interact with a human if they are not engaging with the robot and will not respond to a person unless the person has been identified and initiated interaction with the robot [105, 106]. Churamani et al. also implement a dialog manager in SMACH [35], which uses multisensory cues from nearby humans to select its spoken responses. In surveyed works, state machines were used to generate hotel robot service behaviors [122] and robot social gaze [121]; Jacob et al. use a verbal- and gestural-controlled state machine to pick and pass surgical instruments and enable or disable specific sensory modalities [75]. One advantage of state machines is that they can be programmed to better utilize limited onboard compute resources in mobile robots; for example, Chu et al. do not perform some human perception functions if there are no humans in the scene [34].

Ragel et al. [127] use a sequential data fusion pipeline to combine skeletal pose, face, hand, and speech data into a single Person instance. Each mode is processed individually using a variety of techniques, and the Person instance is incrementally updated as new data from each modality becomes available. The system handles asynchronous messages via standardized robot operating system (ROS) data structures and middleware.

Probabilistic and Bayesian techniques comprise another broad category of late data fusion; in surveyed literature, they were used for human localization and tracking, as well as command, intent, and emotion recognition. Ban [10] and Gebru [49] represent visual and audio localization results as probability distributions, then fuse the two modes via Bayesian estimators to compute the posterior location estimate. Likewise, Bayram and Ince [12] and Nakamura [114] use particle filtering to fuse visual and audio location measurements into a single posterior estimate. Tan et al. use probabilistic hypothesis testing to fuse sound source and face locations and distinguish them from background sound sources [149]. Chau’s Audio-Visual Simultaneous Localization and Mapping algorithm [28] is designed specifically for simultaneous localization of the robot platform and multiple nearby humans. They use a Gaussian Mixture Probability Hypothesis Density filter to fuse audio and visual location measurements,¹⁴ a multitarget tracking filter that models intermittent sensor visibility and human entry/exit from the field of view as random finite sets. Linder [97], Yan [176], and Nieuwenhuisen [116] use multi-hypothesis tracking—another common multiobject tracking method—to fuse visual and LiDAR human position measurements for multiple human tracking in dense social environments. Bohus et al. [20, 21] use probabilistic models to represent quantities like human engagement, activity, and initiating or terminating contact. Both Trick et al. and Whitney et al. fuse speech and gesture to recognize the object that a human is referring to (referring gestures). A prior estimate of the referring gesture can be recursively updated with incoming speech and pointing gestures as in [167], or a categorical probability distribution of the most likely gesture can be computed via an independent opinion pool, as in [153]. Prado et al. use DBNs to classify facial and speech emotional content, and then use an additional DBN to fuse the unimodal results [125].

Data association techniques can also be employed for late fusion. NN association can be used as a late fusion technique, suitable for assigning information from an imprecise spatial measurement to a second, more precise, spatial measurement. For example, [49] use NN fusion to attribute coarsely localized speech activity to precise visual human location; the fused result contained precise location and speech information. Similarly, D’Arca et al. [36] used NN fusion to attribute speaker identity from audio data to a more precise, fused audio-visual human location. Yumak et al. [180, 181, 182] use NN fusion to assign speech to a visually localized human. They compute the speech direction-of-arrival vector, then fuse the associated speech content to the person nearest to the vector. Komatsubara et al. [87], Glas et al. [52], and the Robot Operating System for Human-Robot Interaction (ROS4HRI) fusion module [110] use NN association to fuse facial identity with a detected body. NN and extended NN association can also be used to assign position measurements to humans that are currently tracked by Kalman Filtering as in [97]. The Hungarian algorithm (also called the Munkres algorithm) is another data association technique, used in the literature for associating multiple measurements with multiple existing detections in late fusion. Belgiovine and Gonzalez-Billandon et al. use the Hungarian algorithm to assign multiple face detections to existing faces that are tracked via Kalman Filtering [14, 57], while Nieuwenhuisen et al. use the Hungarian algorithm to assign visual and LiDAR position measurements to humans tracked via multi-hypothesis tracking [116].

The last category of late fusion found in our survey is supervised classification, which was used for several HRI functions that infer multilabel quantities given features or labels from multiple unimodal streams. (Note that this is strictly different from model-based fusion techniques discussed in Section 3.9.1, which use trained models to fuse data instead of classify it.) Engagement and status recognition were common use-cases for supervised classification in the literature, and projects typically evaluated several supervised classification techniques; this is recommended because the results of supervised classifiers can vary significantly based on the application and available data. For the JAMES bartending robot, Foster et al. compute head and hand positions, torso and head poses, and speech activity, then classify them to infer the bar patron’s current state. They evaluate binary regression, logistic regression, multinomial logistic regression with ridge estimator, k-NN, decision tree, naïve Bayes, SVM, and propositional rule classifiers [44, 47]. Ben-Youssef et al. compute gaze, head motion, facial expression, gestures, and speech features from separate perception modules then evaluate logistic regression, DNN, LSTM, and GRUs to classify engagement [17]. Vaufreydaz et al. compute position, velocity, speech, sound source, and pose features via separate perception modules, then classify engagement using an SVM and ANN [164]. Jain et al. compute facial features and audio signal features while measuring performance in a joint child–robot game; these are used to classify the child’s engagement using naïve Bayes, KNN, SVM, NN, logistic regression, random forest, and gradient boosted decision trees [76]. Banerjee et al. also compute head orientation, gaze, body position, body orientation, and audio signals, then classify the human’s interruptability (i.e., willingness to engage) using SVMs and random forest classifiers. They also compute engagement using model-based fusion techniques like CRFs, as discussed previously [11]. Paez et al. estimate a human’s emotional state using k-means classification of recognized body pose, facial gestures, and facial emotion [126]. Lastly, Terreran et al. [150] individually classify pose keypoints of the body and hands with the Shift-GCN architecture, then use ensemble averaging to estimate the person’s overall activity.

More complex HRI systems employed combinations of intermediate and late fusion, known as hybrid fusion. Yumak et al.’s system [180, 181, 182] used intermediate fusion to identify humans by combining and classifying LBP and HSV visual features. They also used NN late fusion to assign speech content to localized humans and infer their focus of attention. They also employ a world model and a rule-based dynamic user entrance/leave mechanism in their fusion module to infer when humans enter or exit the field of regard. The final, fused result contains each nearby human’s identity, location, speech content, and when they entered or left the area.

D’Arca et al. [36] used a hybrid of feature-level and decision-level fusion methods along with state estimation. Kalman and particle filters fuse audio and visual localization results to track speakers in the field of regard. Speaker detection is accomplished by canonical correlation analysis, a feature-level fusion technique that combines audio MFCCs and visual optical flow features to detect who is speaking. The final, fused result contains speaker location and identity and is robust to visual occlusions. The resulting system is comparatively complex—it is comprised of an overall fusion network and subnetworks for tracking, speaker recognition, and audiovisual feature extraction—and is used in a static, indoor setting instead of being implemented on a mobile platform.

While HRI in HSEs has many common characteristics, specific applications and environments can vary significantly from one another. A robot’s tasks and perceptual requirements are often highly application-specific, and an algorithm or component that is ideal for one setting may not work well in another. This has significant implications for the design, evaluation, and usage of MMP systems.

Ideally, HRI researchers should evaluate several components for usage in HSEs. Many surveyed works [11, 17, 18, 44, 47, 97, 120] evaluated various fusion techniques or ML classifiers, for example, and selected the best overall method for their application. Likewise, a ready-made solution may not exist for the specific perception or fusion function, meaning that researchers may need to implement their own custom solutions. Some surveyed works developed custom SSL [49], hand pose tracking [75], head pose estimation [80, 154] or facial recognition algorithms [12], for example, despite the open-source availability of versions of these algorithms, and Yumak et al. developed a custom Integrated Interaction Platform [180, 181, 182]. Note that this is especially evident in earlier surveyed works that pre-date open-source middleware and modern ML frameworks.

Because HRI applications in HSEs can vary, this makes it difficult to provide a solid set of community guidelines or standards for MMP system design; indeed, the HRI community does not appear to have coalesced around a specific set of tools or framework for HRI development and integration. These factors culminate in an increased development time for multimodal systems compared to unimodal perception systems and require increased effort by the developer.

Related to Section 4.2.1, the added overhead of multimodal system design, evaluation, and integration means that state-of-the-art ML and unimodal perception capabilities may take several years before they are used in a MMP system in an HSE. This posed a challenge when conducting this survey: many surveyed fusion and perception techniques (especially in the ML domain) can be more effectively accomplished by using more modern techniques. On the other hand, unimodal ML and perception techniques are rapidly advancing and many methods have not yet been published in a peer-reviewed journal, let alone incorporated into a multimodal system. As we discuss in Sections 4.3 and 4.4, readers are encouraged to evaluate novel perception and fusion capabilities—perhaps that are not yet published—rather than reproduce systems used in past works.

As mentioned in Section 4.2, it is difficult to define a single set of design guidelines for MMP systems in HSEs. However, Table 8 can provide a rough starting point for a system design based on the application. Additionally, we encourage developers to consider the following factors when designing and implementing a MMP system for use in HSEs:

Here, we summarize software and hardware resources discussed in the survey and introduce additional resources for MMP system development.

State-of-the-art HRI perception techniques are continually advancing, and many were outside of the scope of this survey because they were in pre-print, unpublished status or have not yet been incorporated into a MMP system. In addition to the datasets discussed in Section 3.8, the following websites provide resources for training novel HRI perception models:

Integration of MMP systems poses another challenge, and the following software frameworks may be relevant for some HRI applications:

Several surveyed works used social robot platforms for research and development. Additionally, our initial literature search found many publications that—while they did not meet final inclusion criteria for the survey—used common robot hardware platforms for HRI research, and therefore may be of interest for MMP HRI system development. Some are commercially available development platforms with APIs, while others offer case studies in robot system development or were used to conduct HRI studies.

A robot’s ability to sense and interact with humans is a subset of its overall perception capability, and robot perception has advanced considerably in the past several years. These advances are largely driven by new software frameworks, ML architectures, and improvements in commercially available hardware. For example, the Transformer architecture [161] has seen widespread usage in ML tasks since its inception in 2017. Specifically, Transformer components like encoders, decoders, and attention mechanisms can be used to perform time-series fusion and classification tasks that are crucial to multimodal HRI perception. Many unimodal datasets and pre-trained Transformer models exist for baseline HRI functions like gesture recognition, speech recognition, emotion recognition, and VAD.

Although we found several examples of Transformers used for multimodal HRI within the past 3 years [74, 95 100], this research effort has primarily been performed on labeled datasets. As such, we believe that there are many more opportunities to use Transformer-based approaches to accomplish multimodal HRI functions mentioned in this survey, specifically: deictic/referring gesture fusion, multimodal emotion classification, intent recognition, and asynchronous/sparse data fusion. Additionally, deploying these models within multimodal embedded systems in HSEs appears to be relatively unexplored research area. An evaluation of Transformer perception and fusion model efficacy in HSEs could further guide multimodal HRI research.

In the same way that interaction policies can be personalized to improve HRI outcomes [14, 17, 40, 52, 57, 123], we believe that HRI perception systems can be similarly adjusted to fit the current interaction needs. This is partially motivated by surveyed works that used state machine architectures to affect actions based on perceptual stimuli [34, 35, 75, 105, 106, 121, 122] but could be extended to other types of robots operating in HSEs. In particular, a state machine or behavior tree architecture could use cues about the social context, human engagement, and human emotion to select which perception resources and modalities to use, and when. In the same way that a human does not deeply examine the facial expressions of every person they pass on the street, a robot could selectively choose when and how to perceive humans by employing methods found in this survey. For example, if no humans are present, a robot could simply perform human detection using vision and sound event detection. While operating in crowded, public HSEs, human tracking and social navigation using vision and LiDAR would likely be sufficient. If the robot detects that a person wishes to engage, resource-intensive HRI functions like facial recognition, emotional recognition, and speech recognition could be activated as appropriate. This has the added benefit of computational resource management [34], since many perception methods—especially those that use large models for real-time inference—may use a disproportionate share of onboard processing capability for mobile robots and embedded systems.

Many surveyed works employed MMP systems in support of long-term, personalized interaction [14, 52, 57, 72]. This typically involved perceiving soft biometrics, face, and voice identification for open-set identification and fusing these observations with Bayesian or Graph Fusion techniques. One study [52] also recorded metadata about each person, such as when, how long, how often they visited a particular location. These systems required specialized data storage, learning, and inference/retrieval mechanisms which we did not cover in this survey. However, understanding the type of HRI data to record, and how to store, process, and retrieve it, is crucial for long-term interaction and we believe this is an area worthy of further investigation. The references included in this survey can provide a starting point for further long-term HRI perception system development. While there are surveys of long-term learning for neural networks [119], robots [94, 141], and robot perception and manipulation [84], we found no surveys of perception systems or cognitive architectures specific to long-term HRI.

In HSEs, interaction is situated within a broader environmental context and the most ideal course of action is tightly coupled with the current environmental state [22]. Architectures that can fuse multimodal stimuli to simultaneously perceive the status of nearby humans and plan a suitable action may be especially desirable for HRI in HSEs, however we only found two examples in the survey: the POMDP dialog manager of Lu et al. for the Kejia robot, and the POMDP collaborative planner used in the SPENCER project [80, 154]. Both employ POMDP graphs on mobile robots in public environments to fuse multimodal inputs and select a suitable mode of interaction.

Additionally, the Workshop on Integrated Perception, Planning, and Control for Physically and Contextually Aware Robot Autonomy²⁶ addresses this topic more generally. Contributions in 2023 address multimodal semantic segmentation and robust perception, techniques that could be employed for MMP in HRI.

The surveyed research in perceptual interfaces is largely dominated by audiovisual systems [10, 12, 20, 21, 28, 36, 49, 139, 156, 180, 181, 182], but introducing new sensor modalities could increase the accuracy or robustness of perceptual interfaces while offering new semantic information.

Although 3D LiDAR detection has seen widespread use in autonomous vehicle perception, it was relatively uncommon for HRI perception; Yan et al.’s 3D LiDAR human tracker was the only example in this survey [176]. 3D LiDAR detection [174] could provide valuable complementary information to a MMP HRI systems beyond existing 2D LiDAR leg detectors. Namely, a 3D system could detect humans even while they are not standing and would reduce false positives from furniture legs [175]. Several pre-trained 3D LiDAR human detectors are available as mentioned in Section 4.3 and could be implemented on a robot platform with 3D LiDAR.

Thermal cameras and mmWave-radar also offer promising sensor modalities to complement HRI but were relatively uncommon in surveyed works. Mainly, these modes were used in applications where vision may be degraded [29, 103, 131], but they offer many functional benefits: a human’s thermal signature can help segment human and non-human clusters [114] and identify emotional state [41], while mm-Wave Radar can penetrate visual obscurants like fog, smoke, precipitation, and low-light conditions. Using thermal or radar data in a multimodal human perception system could be especially useful for segmenting humans in cluttered social scenes, potentially improving robustness or accuracy even when vision is not degraded.

Finally, adopting community standards for HRI researchers could reduce duplication of effort and lower the barrier to entry for HRI researchers. HRI in HSEs can often be unstructured and difficult to control, hence reproducibility can also be a concern for HRI experiments and systems. Gunes et al. [65] provide useful guidelines for conducting HRI experiments and for HRI system design. They recommend that HRI researchers be explicit about SDKs, middleware, and datasets used, and note that the usage of common frameworks aids in reproducibility, commonality, and development for HRI systems and experiments.

At time of writing, ROS4HRI appears to be the most concentrated effort toward creating an HRI-specific community developer standard and includes a protocol and ROS message types. Implementation tutorials are under active development. Developers are invited to integrate additional hardware and software components into ROS4HRI in accordance with REP-155 standard.²⁷ Currently, ROS4HRI features facial detection, skeletal pose recognition, and facial body fusion via assignment, which is a subset of the many techniques discussed in this survey. Incorporating more perceptual and fusion techniques into the ROS4HRI framework would enable more rapid development and evaluation of HRI perception systems. Similarly, PSI’s website²⁸ allows external collaborators to develop components for the PSI framework in accordance with the software’s API.