Multi-modal Open World User Identification

One of the first algorithms applied to open world recognition was Nearest-Non Outlier (NNO) [8], which modified Nearest Class Mean (NCM) [64] for open-set classification and incremental learning. Another approach is Extreme Value Machine (EVM) [73] based on Extreme Value Theory, which outperformed NNO on the open world ImageNet benchmark [8, 73]. However, both of these methods work with incrementally adding a batch of new classes (e.g., 100 at a time), as opposed to incremental learning of classes (one at a time). Similarly, the approach proposed in Reference [29] is based on a center-based similarity space learning method and 1-vs-rest strategy of Support Vector Machines (SVM) for object classification. However, none of these methods has been evaluated on user recognition.

Several online learning methods exist for various application areas [34]. In video-based recognition, Lee and Kriegman [55] proposed an online learning algorithm of probabilistic appearances, but a prior generic model is necessary for this approach. Boucenna et al. [13] used online and incremental learning in two neural networks for facial expression recognition and face/non-face discrimination in an HRI imitation game. The former neural network uses a k-means variant SAW (Self Adaptive Winner takes all) [49] to categorise focus points in the image, whereas the latter predicts the interaction rhythm [3] (i.e., timing for interaction) to detect whether the user is interacting with the robot. While the face discrimination method was shown to generalise to new users successfully, the facial expression recognition achieved low success rates for generalisation. In addition, both approaches required preliminary training and were evaluated on a low number of users (20). De Rosa et al. [26] used online learning in open world (object) recognition for incremental learning of a classification metric, the threshold for novelty detection, and describing the space of classes. The approach was applied to three existing algorithms, namely, NCM, NNO, and Nearest Ball Classifier (NBC) [27]. Their results showed that online learning increases classification performance.

In a multi-modal biometric system, information from different identifiers, such as face recognition or gender identification, is fused via prior or post classification [44]. Prior classification requires access to the features or sensor values of the identifiers, which are generally not available for proprietary algorithms.

For post-classification, two approaches exist: classification and combination of confidence scores. Classification methods, such as neural networks and SVM, combine non-homogeneous data from individual classifiers into a feature vector for further classification without the need for preprocessing. In the combination approach, individual matching scores from the identifiers are combined into a scalar score in three steps: (1) normalisation of scores into a common domain, (2) combination of scores based on Bayes decision rule and posterior probabilities, e.g., sum or product rule, and (3) thresholding for classification. The performance of these approaches depends on the chosen method and threshold.

Bayesian approaches have been widely used for combining primary biometrics, such as face and speaker recognition [10, 17, 82], as well as combining soft biometrics [25, 45, 46, 67, 78, 86]. For instance, Jain et al. [45] proposed a Bayesian network for combining fingerprints with soft biometric traits, namely, gender, ethnicity, and height. They used a fixed weighting scheme, where the biometrics with smaller variability and more substantial distinguishing capability were given more weight and achieved slight improvement in recognition. Similarly, Scheirer et al. [78] used a Bayesian network with Noisy-OR weighting that combines face recognition with ethnicity, hair colour, gender, age, eyebrow type, and non-soft biometric contextual information, such as the occupation and location of the person. Contrary to the work in Reference [45] and our approach, they used the accuracy of estimators to adjust the FR match score.

Similar to biometric recognition, the most common approach for user recognition in HRI is through FR [4, 5, 23, 32, 38]. However, robots can take advantage of multi-modal recognition due to the variety of different sensors they carry. Soft biometrics are especially important, because they allow non-intrusive recognition, but only a few studies use soft biometrics. Martinson et al. [61] used a weighted summation of soft biometrics (clothing, complexion, and height) to identify users within a short-term interaction from a group of only three users. Boucenna et al. [12] gathered extensive data (100 images per person) during a game and later evaluated the recognition offline using a Hebbian rule-based neural network. Ouellet et al. [65] combined face recognition, speaker identification, and human metrology through Hampel estimators in closed-set identification using a substantial time for training (3.5 minutes) and a small number of participants (pretraining on 22, test on 7). Al-Qaderi and Rad [1] combined face, body, and speech information using a spiking neural network in closed-set identification and have evaluated on a simulated dataset. These approaches do not apply to open world recognition, hence, their methods are not easily comparable to ours.

Our previous work [43] introduced a multi-modal weighted Bayesian network, which is the first approach in combining multi-modal biometric information for sequential and incremental learning of new users for open world recognition that allows starting from a state without any known users. It is also the first approach in combining soft biometrics (gender, age, height, and time of interaction) with a primary biometric (FR) to identify a user in real-time HRI. Online learning was used for learning the likelihoods of the network from sequential data to improve the recognition over long-term interactions. The weights of the network were optimised to minimise the number of incorrect recognitions. The quality of the estimation measure was introduced to decrease the number of incorrect recognitions for unknown users. The results obtained in a user study with 14 participants over four weeks showed a slight improvement in identification rate (up to 1.4% in open-set and 4.4% in closed-set recognition) compared to 90.3% of FR. The optimised weights suggested that age is the least effective soft biometric parameter, whereas height is the most effective one. Moreover, the Bayesian network performed worse with online learning. However, we concluded that the dataset might be biased towards the participants’ characteristics due to the low number of participants and limited age range, and an evaluation with a bigger dataset is necessary to understand the capabilities of the system entirely.

This article extends the work in Reference [43] for evaluating the approach within the Multi-modal Long-term User Recognition Dataset and two other real-world HRI experiments, and optimising the weights of the Bayesian network through a long-term recognition performance loss criterion with hybrid normalisation.

The number of states for each node depends on the modality: F and \(I\) nodes have \(n_e\)+1 states, where \(n_e\) is the number of enrolled (known) users. \(A\) and \(H\) nodes are restricted to the available range of the identifier, such as \([0,75]\) for \(A\) and \([50,240]\) (cm) for \(H\). \(G\) has “female” and “male” states. \(T\) is defined by the day of the week and the time, through time slots. For example, if each minute corresponds to a time slot (i.e., time period, \(t_p\), is 1 min), then there will be 10,080 \(T\) states (there are 10,080 minutes in a week).

When a user is encountered, the corresponding multi-modal biometric evidence is collected from the identifiers. An example for the biometric evidence from the identifiers and the transformed (weighted and normalised) evidence is shown in Figure 16 B in Appendix A. FR provides similarity scores, which give the percentage of similarity of the user to the known faces in the database. Age, height, and time are assumed to be discrete random variables with a discretised and normalised normal distribution of probabilities, \(N(\mu , \sigma ^2)\), defined by (1), where \(V\) is the estimated value, \(Z\) is the standard score, and \(C\) is the confidence of the biometric indicator for the estimated value.

The time period and its standard deviation (\(\sigma _t\) in the normal distribution of \(T\)) can be set depending on the precision required in the application. A smaller time period and standard deviation ensure higher precision, however, this would increase the complexity of the Bayesian network, thereby increasing the time to identify the user. In addition, a higher precision carries the risks of decreasing the recognition rate if the users are not encountered near the time slot that they were previously seen.

For example, if users in the application scenario will change every 5 minutes, then \(t_p=5\) min and \(\sigma _t=15\) min would be reasonable. However, in an HRI scenario, \(t_p=30\) min with \(\sigma _t=60\) min can allow better identification, because it is less likely to encounter users around the same time every day. Hence, we use the latter in this article.

Soft biometric traits are characteristics that are not suited to identify an individual uniquely. We can assume that the population will have similar characteristics, but the distribution is unknown. However, some soft biometric features may contain more information about an individual than others, e.g., age is often more informative than gender. This can be modelled by using different weights for the parameters in a Bayesian network [45].

Weights (\(w_i\)) are used as the exponential to the likelihoods of the child nodes (\(X_i\)), similar to the work in Reference [88]. In contrast to our previous work [43], we optimise the weights of soft biometric features (gender, age, height, and time of interaction) through Bayesian optimisation, as described in Appendix C.6, while the weight of the face node (\(w_F\)) is set to be 1, as it is the only primary biometric in our system. The posterior probability \(P(I^j| X_1, \ldots , X_n)\) is approximated as in Equation (2). \(I^j\) stands for the \(j\)th user (\(I=j\)), where \(I\) is the identity node.

As in Reference [45], we assume that the identifiers perform equally well on all users. Therefore, the accuracy of an identifier is independent of the user and equal priors are assumed for each of the identifiers.

The posterior probability simplifies to Equation (3):

Because the distribution of users over time is not known, one approach for determining \(P(I^j)\) is to use adaptive priors using frequencies of user appearance, however, this can create a bias in the system towards the most frequently observed user, as it affects the posterior probability directly, thus, may result in a decrease in the identification rate.

Therefore, we assume that the probability of encountering user \(j\) is equally likely as encountering user \(m\), hence, we assume equal priors for \(P(I)\), as shown in Equation (4), where \(n_{e}\) is the number of enrolled users, which is updated whenever a new user is enrolled, as presented in Figure 16 in Appendix A.

Algorithms for open-set problems generally use a threshold (e.g., over the highest probability/score) to determine if the user is already enrolled or “unknown.” However, the resulting posterior probabilities in a Bayesian network can be low due to the multiplication of the conditionally independent modalities and vary depending on the number of states. Hence, we use the two-step ad hoc mechanism introduced in Reference [43] to transform the Bayesian network to allow open-set recognition: (1) An “Unknown” (\(U\)) state is used in both \(F\) and \(I\) nodes. The similarity score in FR of \(U\) is set to the FR threshold (\(\theta _{FR}\)), such that when normalised, scores below/above the threshold will have lower/higher probabilities than \(U\). This allows maintaining the threshold for the FR system in use. (2) We use the confidence measure called the quality of the estimation (\(Q\)). Given the evidence \(y_t\) at time \(t\), it compares the highest posterior probability (\(P_w\)) to the second highest (\(P_s\)), as shown in Equation (5). The difference between the probabilities decreases, as the number of enrolled users (\(n_e\)) increases, since \(\sum _{j} P(I^j|y_t)=1.0\). A similar method was used in Reference [31] for estimating the quality of localisation based on different images.

Using the quality of the estimation enables decreasing misidentifications. For example, the highest posterior score can be very high, but if the second highest posterior is very close to it, then it means that there are two possible strong candidates for the current user. If the system were to identify the user in this case, then the resulting misidentification could cause adverse effects on the current user especially in the case of different genders or age differences between the two users, as well as security issues. Thus, it is more preferable to identify the user as unknown if the quality is zero or below a predetermined threshold (\(\theta _Q\)) or if \(U\) has the highest posterior probability. Otherwise, the identity is estimated with a maximum a posteriori (MAP) estimation, given in Equation (6).

For personalisation in long-term HRI applications, new users may often need to be enrolled in a system to allow recognition in subsequent encounters, such as for admitting a new patient to personalised robot therapy. However, in such applications, the intermediary (e.g., clinical staff) and end-users (e.g., patients) are often non-experts, hence, systems that require the least amount of technical knowledge, effort, and time are desirable, especially those that allow users to enrol themselves. Thus, we developed an incremental learning system for the weighted multi-modal Bayesian network, which expands the network upon new user enrolment. When the MMIBN detects that the user is new, the robot requests to meet the user, and (verbally) asks for their name, gender, birth year, and height, which the user can enter through a tablet interface, after which a photo of the user is taken by the robot (step 9 in Figure 15).

This information, along with the time of interaction, is gathered to have the ground truth values for recognition and for setting the initial likelihoods of the MMIBN.

Initially, the system starts from a “tabula rasa” state, where there are no known users. Bayesian network is formed when the first user is enrolled: one state for the new user and one for the “Unknown” (\(U\)) state. Figure 16 A (in Appendix A) illustrates an example for the initial MMIBN after the enrolment of the first user, e.g., a 25-years-old female who is 168 cm tall and encountered at 11:00 am on a Monday. The initial likelihood for \(F\) is set to be much higher for the true values, as shown in Equation (7), where \(w_F\) is the weight of the face variable, and \(n_e\) is the number of enrolled users. The value was found based on preliminary experiments.

The remaining likelihoods are set using the prior knowledge that the user entered in a similar structure to the evidence for age, height, and time variables with a discretised and normalised normal distribution, \(N(\mu , \sigma ^2)\), where \(\mu\) is the true value (e.g., age of the person), and \(\sigma\) is the standard deviation of the identifier. Gender is set at \([0.99^{w_G}, 0.01^{w_G}]\) ratio, which is experimentally found. For the unknown state, \(P(X_i^k|I^U)\) is set to be uniformly distributed, as an unknown user can be of any age, height, and be recognised at any time of the day, except for the face node, which follows Equation (7).

When a new user is enrolled, the Bayesian network is expanded by adding a new state to \(I\) and \(F\) nodes. \(P(F^k|I^j)\) for each previous state in \(I\) (including \(U\)) is updated by appending the value corresponding to \(k \ne j\) condition in Equation (7), and then probabilities are re-normalised. The likelihoods of \(G\), \(A\), \(H,\) and \(T\) nodes for the previously enrolled users remain the same. An example of the MMIBN likelihoods during incremental learning of a new user, e.g., a 37-years-old male, 173 cm tall, and encountered on a Wednesday at 8:00 pm, is illustrated in Figure 16 E in Appendix A.

The scalability feature removes the need to retrain the network when a new user is introduced, hence, the time complexity is decreased, which can be crucial if the new user is introduced at a later step (e.g., after 1,000 users). More precisely, if each image corresponding to \(\hspace{0.83328pt}\overline{\hspace{-0.83328pt}n_o\hspace{-0.83328pt}}\hspace{0.83328pt}\) average number of observations per user was to be recognised again after a new user is added to the face database, then it would take a significant amount of time to expand the network compared to scaling, since \(n_e*\hspace{0.83328pt}\overline{\hspace{-0.83328pt}n_o\hspace{-0.83328pt}}\hspace{0.83328pt}*\mathcal {O}(FR) \gg n_e*\mathcal {O}(1)\) updates, where \(\mathcal {O}(FR)\) is the time complexity of the FR algorithm, and \(n_e\) is the number of enrolled users.

To reduce the risk of confusing new users with known users, it is preferable to have sufficient data within the MMIBN prior to making reliable estimations, hence, in the first few recognitions (here, we chose \(N\lt N_{min}=5\) recognitions, i.e., the first 4 recognitions),⁶ the identity is declared as unknown, regardless of the estimated identity, as illustrated in Figure 16 C (Appendix A).

Bayesian network parameters are generally determined by expert opinion or by learning from data [51]. The former can cause incorrect estimations if the set probabilities are not accurate enough. The latter, for which Maximum Likelihood (ML) estimation is commonly used, is not possible when the Bayesian network is constructed with incomplete data. One solution is to use offline batch learning, however, it requires storing data that can cause memory problems in long-term interactions. Another approach is to update the parameters as the data arrive, which is termed online learning. Variants of Expectation Maximization (EM) algorithm with a learning rate (EM(\(\eta\))) [6, 20, 57, 59] have been proposed for online learning in Bayesian networks.

We use a Bayesian network where the likelihoods are updated through EM(\(\eta\)) with an adaptive \(\eta\) (learning rate) based on ML estimation, similar to Voting EM [20]. Adopting the notation in Reference [6], the formulation is given in Equation (8). \(\theta _{ijk}^{t}\) represents the likelihood of the modality \(X_i\) at time \(t\), \(P(X_i=x_i^k|I^j)\). \(P_{\theta ^t}(x_{i}^{k}|y_{t}, I^{j})\) represents the posterior probability of the modality \(X_i\) at time \(t\) given the current evidence \(y_t\) and the actual identity of the user \(I^j\). The difference between Voting EM and our approach is that we work with continuous probabilities due to uncertainties in the identifiers. We will refer to the proposed multi-modal incremental Bayesian network with online learning as MMIBN:OL.

Combining ML estimate to achieve an adaptive learning rate (given in Equation (9)) allows the learning rate to depend on the observation of the user \(j\) (\(n_{oj}\)), which is more reliable than using a fixed rate for all users. Also, each observation of the user creates a progressively smaller update on the likelihoods, such that the effect of a new observation decreases as the number of recognitions of the user increases.

Supervised learning is necessary to achieve accurate online learning. The identity of the user should be known for updating the corresponding likelihoods, which can be achieved in HRI by asking for a confirmation of the estimated identity.

If the user \(j\) is previously enrolled in the system, then the likelihoods are only updated for user \(j\), as shown in Figure 16 D (in Appendix A) based on the evidence in Figure 16 B. However, if the user \(j\) is a new user, then online learning is applied on the face likelihood for the unknown state (\(P(F^k|I^U)\)), followed by incremental learning by expanding the MMIBN (as described in Section 3.4), and finally by applying online learning for the new user, as illustrated in steps 8–18 in Figure 15 and in Figure 16 F. The likelihoods of gender, age, height, and time remain the same for \(U\) to ensure uniform distribution.

The standard metrics for open-set identification are Detection and Identification Rate (DIR) and False Alarm Rate (FAR) [69]. DIR is the fraction of correctly classified probes (samples) within the probes of the enrolled users (\(\mathscr{P}_\mathscr{E}\)), given in Equation (10). FAR is the fraction of incorrectly classified probes within the probes of unknown users (\(\mathscr{P}_\mathscr{U}\)), given in Equation (11).

In other words, DIR represents the “true positive” (TP) of enrolled users, in which the current probe (referring to the multi-modal biometric sample) belongs to a previously enrolled user and identified correctly. FAR serves as a “false positive” (FP) for unknown users, that is, the probe belongs to an unknown user, but he/she is identified as an enrolled user. However, TP and FP are notions of verification problems, in which the probe is compared against a claimed identity, thus, are generally not applicable to open-set identification. Instead, the tradeoff between DIR and FAR that depends on the threshold of the identifier is generally represented by a Receiver Operating Characteristic (ROC) curve. The standard practice in biometric identification is to determine the desired FAR, which would then set the threshold and DIR.

Depending on the biometric application, the cost of incorrectly identifying a user as known may be very different from the cost of incorrect identification of the enrolled user [47]. For short-term interactions, in which a user will be encountered 1–2 times, FAR is as important or more important than DIR. However, for long-term interactions, users will be encountered a greater number of times. Thus, correctly identifying a user (in a closed-set) becomes more important than correctly identifying an unknown user (open-set). Hence, we introduce the long-term recognition performance loss (\(L\)) that creates a balance between DIR and FAR based on the average number of observations per user (\(\hspace{0.83328pt}\overline{\hspace{-0.83328pt}n_o\hspace{-0.83328pt}}\hspace{0.83328pt}\)), as presented in Equation (12), where \(\alpha\) is the ratio of importance of \(DIR\) compared to \(FAR\).

Weights of MMIBN are optimised through this loss function, for gender, age, height, and time in \([0, 1]\) range, along with quality (\(Q\)) that can change within \([0, 0.5]\) range. Ideally \(L=0\), where all unknown users are identified as such (FAR\(=0.0\)) and the known users are correctly identified (DIR\(=1.0\)).

The scores from each modality must be normalised into a common range (e.g., \([0,1]\)) to ensure a meaningful combination. It is important to choose a method that is insensitive to outliers and provides a good estimate of the distribution [44], such as, minmax, tanh [37], softmax [11], and normsum (dividing each value by the sum of values). We introduce hybrid normalisation, which combines the methods that achieve the lowest loss for each modality. In other words, hybrid normalisation uses the best-performing normalisation method for each modality. Extensive tests were made on the dataset obtained from our previous work in Reference [43] to get the optimal methods for each modality (\(F\), \(G\), \(A\), \(H,\) and \(T\)). The long-term recognition performance loss was compared for each combination of the individual modality with face recognition (\(F\), \(F\)-\(G\), \(F\)-\(A\), \(F\)-\(H\), \(F\)-\(T\)) by optimising the weights for each of the combinations. The resulting hybrid normalisation uses normsum for face, gender, and height; tanh for age; softmax for time of interaction.

In the scope of this work, only one user is assumed to be present in each image, hence, the cropped faces of IMDB dataset is used. To simulate an open world HRI scenario, where the users will be met in consecutive days or weeks, we chose images of users that are from the same year. Furthermore, we assume that the average number of times a user will be observed is \(\hspace{0.83328pt}\overline{\hspace{-0.83328pt}n_o\hspace{-0.83328pt}}\hspace{0.83328pt}\ge 10\), which is a reasonable assumption for long-term HRI. Hence, we choose celebrities who have more than 10 images each corresponding to the same age. Moreover, to assess the incremental learning capabilities of our model with a user database that is more realistic for HRI (i.e., sufficiently large with 100 to 200 users instead of thousands of users), we (randomly) sampled 200 users out of 20K celebrities.

To create a diverse set of ages in the dataset, the images that correspond to an age that is within the five most common ages (25, 26, 28, 30, 31) in the set were randomly rejected (with 50% probability) during the selection. For instance, Anne Hathaway has sufficient images corresponding to 25 and 27 years old in the IMDB-WIKI dataset. However, 25 is among the five most common ages, thus, with a 50% chance, this set of images was excluded from the selection, hence, the images of Anne Hathaway corresponding to 27 years old were chosen instead. This also resulted in some celebrities who only have images corresponding to a certain age in the dataset to be excluded from the selection.

The resulting age range is 10–63, with the mean age of 33.04 (SD\(=9.28\)).

Subsequently, the dataset is cleaned in three steps: by removing (1) images with a resolution lower than 150 \(\times\) 150, (2) images without a face detected by NAOqi, (3) images that erroneously correspond to another person. Furthermore, in order of user appearance (as detailed further in Section 6.2), NAOqi identifiers are applied on the selected images to obtain face similarity scores, gender, and age estimations. If the user has not been previously encountered, then the same image is used to identify the user before and after enrolment to the face database in NAOqi.

Height was found to be the most important soft biometric in determining the identity in Reference [43]. To validate whether this finding persists for a large number of users with diverse characteristics and optimise its weight for applying it to real-world HRI experiments, we artificially created height data for each user.

To keep the data realistic and model the differences between the estimated heights, Gaussian noise with \(\sigma =6.3\) cm (as found in Reference [43] for NAOqi height estimation) is added to the actual heights of the users obtained from the web.

Given our assumption that the users will be encountered at least 10 times in long-term HRI, we created two datasets: (1) D-Ten, where each user is observed precisely 10 times, e.g., 10 return visits to a robot therapist, and (2) D-All, in which each user is encountered a different amount of times (10 to 41 times). Two types of distribution are considered for the time of interaction: (1) patterned interaction times in a week modelled through a Gaussian mixture model, where the user will be encountered certain times on specific days, which applies to HRI in rehabilitation and education areas, and (2) random interaction times represented by uniform distribution, such as in domestic applications with companion robots, where the user can be seen at any time of the day in the week. As a result, we created four (sub)datasets as part of the Multi-modal Long-term User Recognition Dataset: D-Ten_Uniform, D-Ten_Gaussian, D-All_Uniform, D-All_Gaussian.

These hypotheses will be validated with various analyses, as provided in Table 1.

Repeated k-fold cross-validation is used to evaluate the model stability and performance. The procedure is described in Algorithm 1 in Appendix B. Two methods for creating validation folds are used, namely, OrderedKFold and ShuffledKFold. OrderedKFold is the case where users are introduced one-by-one to the system without any repetitions of previous users during the enrolment. The order of repeated interactions is random after the enrolment. In ShuffledKFold, there can be repetitions of the previous user(s) before another user is introduced, because the order of overall samples is random. OrderedKFold is similar to batch learning in an incremental learning sense, whereas the iteration (repeat) created by ShuffledKFold is more similar to a real-world scenario. Our aim is to evaluate if there are any performance differences between the two cases and to prove that the model is stable across several repeats. A stratified random bin order is used for having a different initial bin and final bin in each fold to ensure a different enrolment order of users and a different test set, respectively. We chose K\(=5\) folds and R\(=11\) repeats.

Each dataset (D-Ten and D-All) is divided into two with 100 users each. The first set is then divided through cross-validation procedure with 80%–20% ratio of data to the training set (first four bins, corresponding to 800 samples in D-Ten and 2,308 in D-All) and closed-set (training) (final bin, corresponding to 200 samples in D-Ten, 578 in D-All). The open-set is created from the remaining 100 users (800 samples in D-Ten, 2,280 in D-All). The closed-set (open) is similar to the closed-set (training), which corresponds to the final bin in each fold (200 in D-Ten, 569 in D-All). The open-set evaluation is made by introducing the open-set samples after the training set, that is, 100 users are enrolled in the system and recognised multiple times before the introduction of 100 new users. However, the results for the open-set do not include the results for training.

The only difference between Gaussian and uniform datasets is the time of the interaction for each sample; that is, the order of the samples is the same.

For online learning, the likelihoods are learned during the training phase (training and open-set cases), and the learned likelihoods are used without online learning for the closed-set cases.

Given our datasets and the parameters of our model, we have four independent variables and three dependent variables for analysing the results on the evaluation sets: training, open-set, closed-set (training), closed-set (open). The dependent variables are DIR in Equation (10), FAR in Equation (11) and long-term recognition performance loss (shortly, loss) in Equation (12). The independent variables are as follows:

The parameters of the MMIBN need to be optimised to achieve the best recognition results. Correspondingly, we conducted several evaluations on the Multi-modal Long-term User Recognition Dataset as described in detail in Appendix C. Here, we summarise our findings for reasons of perspicuity.

Initially, the loss parameter \(\alpha\) is set as 0.9, based on our average number of observations assumption (\(\hspace{0.83328pt}\overline{\hspace{-0.83328pt}n_o\hspace{-0.83328pt}}\hspace{0.83328pt}=10\)) for long-term interaction (Appendix C.1). Subsequently, the optimum face recognition threshold with the lowest loss for (NAOqi) FR is found to be 0.4 (Appendix C.2).

MMIBN relies on the assumption that the multi-modal biometric information (face, gender, age, height, and time of interaction) are conditionally independent given the identity of the user, since the individual identifiers do not affect each other’s results. Accordingly, we assumed that the NAOqi identifiers (face, gender, and age) are conditionally independent of each other, despite relying on the same visual input (2D image). Structural learning of the Bayesian network on the Multi-modal Long-term User Recognition Dataset (in Appendix C.3) confirmed this assumption, showing that the naive Bayes classifier model is sufficient and suitable for multi-modal user identification, even when the modalities use the same input. Moreover, the average learned likelihoods in online learning are very close to the initially assumed network parameters in Section 3.4.

Bayesian optimisation¹⁴ is applied with these parameters to minimise the loss for each combination of the independent variables (40 conditions) by optimising the weights for soft biometrics and the threshold for the quality of the estimation (see Appendix C.6). Figure 4 shows how the loss decreases during the optimisation, which results in an increase in DIR at the cost of an increase in FAR. The resulting loss of MMIBN is much lower than that of FR, and correspondingly DIR and FAR are much higher. Note that \(\alpha\) can be adjusted to give more importance to FAR or a FAR can be set prior to optimisation, which may lead to a different set of optimised parameters.

While the average standard deviation of NAOqi age estimation is found to be higher (11.0) than in Reference [43] (9.3), the age is found to be the most important parameter and height the least (see Appendix C.6), in contrast with the findings in Reference [43]. Due to the higher number of users (200) and the diverse age range (10–63) in the Multi-modal Long-term User Recognition Dataset, these results are more generalisable than our prior work. Moreover, when the ground truths are not taken into account, the standard deviation of age within the estimations is found to be 8.2, which is less than the average. This is due to the appearance of users (e.g., a 30-year-old person may look like 25), which suggests that online learning of likelihoods (MMIBN:OL) may provide better recognition performance over time, as the identifiers will get better at identifying users based on their own estimations instead of ground truth values. In addition, NAOqi gender recognition is found to be equally accurate for males and females with 0.9 as the recognition rate (i.e., users’ genders are correctly recognised 90% of the time). Furthermore, using the confidence of the estimations instead of exclusively the estimated biometric data (e.g., estimated gender or age, as described in Section 3.1) allows overcoming deviations in the estimations.

With these optimised parameters, 11 repeats of 5-fold cross-validation were applied for each of the conditions (Appendix C.4), which showed that MMIBN models are stable across repeats (i.e., no significant difference in loss between repeats), and the models perform equally well for learning new users incrementally sequentially (OrderedKFold, similar to batch learning) and at random intervals (ShuffledKFold, similar to a real-world scenario). However, the size of the dataset, timing of interaction, and normalisation method are found to have significant effects on the performance of the model, however, the non-adaptive model and the model with online learning performed equally well.

Hybrid normalisation is found to outperform the other normalisation methods in all conditions (Appendix C.5), supporting our hypothesis H3. The models achieved lower loss in D-All than in D-Ten, which showed that the proposed model gets better with the increasing number of recognitions. However, hybrid normalisation with online learning (MMIBN:OL) is found to perform worse than the non-adaptive model (MMIBN), in contrast with our hypothesis H2. Moreover, most methods are found to perform significantly worse when there is no interaction pattern (uniform timing of interaction), as compared to patterned (Gaussian) interactions, supporting our hypothesis H4.

On the grounds that the optimised parameters of our proposed MMIBN are found, we can compare its results to face recognition (FR) and soft biometrics (SB). FR results are obtained from the NAOqi estimations by setting FR threshold (\(\theta _{FR}\)) to 0.4. SB results are obtained by giving zero weight to FR, that is, only gender and age estimates from NAOqi, artificial height estimates and time of interaction are used for identifying a user. The weights of these modalities in SB are the same as MMIBN, as shown in Figure 19 (Appendix C.6). Similarly, the weights of SB:OL are the same as those of MMIBN:OL.

We transformed a state-of-the-art open world recognition method, Extreme Value Machine¹⁵ [73] (EVM), to accept sequential and incremental data for online learning by adjusting its hyperparameters to use it as a baseline, as described in Appendix D. In the original work, batch learning of 50 classes was used with an average of 63,806 data points at each update, instead of a single data point that we used in this work. We compared our methods with the performance of two EVM models: (a) EVM:FR, using NAOqi face recognition similarity scores as data, (b) EVM:MM, using multi-modal information in the same format as it is used for our methods.

Section 6.5.1 compares the long-term recognition performance loss (shortly, loss) between the models. Appendix C.4 provides evidence that there is a significant correlation between loss and DIR, and loss and FAR, but no significant correlation is found between DIR and FAR. Hence, the analysis of loss is sufficient to determine how the model performs in comparison to others. Nevertheless, we will report the results of FAR and DIR of the models in Section 6.5.2 to further observe how the open-set recognition metrics are affected.

In general, the loss is lower, DIR and FAR are higher in D-All than in D-Ten. The increase in DIR and the decrease in loss can be explained by the higher number of recognitions, which increases the performance over time. The increase in FAR can be due to different optimised weights for each dataset (see Figure 19 in Appendix C.6). However, both datasets show similar patterns in differences between FR, SB, and MMIBN models. Even though the number of samples per user is not the same in D-All, the fact that it performs equally well as D-Ten for the same number of recognitions shows that our equal priors assumption (Equation (4)), which states that each user is equally likely to be seen, does not have any adverse effect on our proposed models.

While the weights of the biometric information differ based on the dataset size and the model, their positive values indicate that each modality is beneficial and effective in identifying users, and conditionally independent of each other, as supported by the learned structure (Appendix C.3). We suggest using the optimisation parameters (weights and quality threshold) that are optimised for D-All datasets, since this dataset contains more samples. If the application is based on users appearing at specified times during a week (e.g., long-term therapy in a hospital), then the optimised parameters for D-All_Gaussian should be used; otherwise, it is better to use that of D-All_Uniform (e.g., for companion robots). These optimised parameters generally perform significantly equivalent in both timing conditions in D-All for both models, as shown in Figure 20, even though the timing of interaction does not provide enough information in the uniform timing case. Nonetheless, using different (or more accurate) identifiers for soft biometrics may result in a different set of weights and better recognition performance.

High FAR of the models is due to the tradeoff between recognition and spotting unknown people, which is visible in Figure 4. The value of \(\alpha\) determines the importance of this tradeoff in the loss function to ensure a higher number of correct recognitions in a long-term interaction. We found \(\alpha =0.9\) based on our assumption, that the average number of interactions is 10. Using a varying amount of samples (D-All) did not change the overall performance in terms of long-term recognition loss for the same number of total samples when we compared D-All and D-Ten at the same amount of samples (800 for training and open-set and 200 for closed-sets). In Figure 5, 71% of the users had less than 10 recognitions and 20% had more than 10, before the 800th recognition in D-All dataset. This finding shows that our choice of \(\alpha\) did not negatively affect the results. Thus, instead of changing \(\alpha\) for decreasing FAR, we would suggest using a variable threshold of quality (\(\theta _{Q}\)) based on the number of users in the dataset to ensure that the quality is higher when the number of users is low.

The presented results are dependent on the noise level of the identifiers and the characteristics of the population (e.g., the distribution of parameters within the population). By using other algorithms for the identifiers or by setting a desired FAR depending on the application from Figure 4, a different set of weights can be achieved with lower/higher FAR and consequently lower/higher DIR.

We initially assumed that all identifiers work equally well on all users based on the work in Reference [45]. However, there can be changes in the person’s appearance, the similarity between users, as well as changes in time of interaction, which could negatively affect the visual identifiers and the time component of our models, respectively. We claimed that our online learning approach would adjust to these changes and perform better than the non-adaptive model (H2), but the second part of the hypothesis is not supported, because online learning (MMIBN:OL) performed significantly worse or at the same significance as the non-adaptive MMIBN.

The underlying reason might be the accumulating noise in the identifiers. We suggest three possible solutions for improving online learning: (a) identifiers with lower noise can be used, which can be difficult to achieve in real-world scenarios; (b) similar to the work in References [20, 59], the learning rate \(\eta\) can be increased when there is a large error between the estimated parameter and its mean value, and decreased when convergence is reached; (c) confidence value of the identifiers or the quality of the estimation can be used to determine if the likelihoods should be updated at each iteration to avoid updating when the noise is high. However, the average learned likelihoods in online learning showed that the initial parameter assumptions in Section 3.4 hold valid. Online learning can also be applied to the weights of the MMIBN nodes to improve recognition performance over time based on the identifier accuracy, through decreasing or increasing the weights of the identifiers that are less or more accurate based on the data. We suggest applying online learning, similar to References [40] or [56], on top of the optimised weights found in this work, which would allow adapting the MMIBN to work equally well (or better) with any (i.e., NAOqi or other) identifiers. However, a simpler approach is to apply Bayesian optimisation (of the weights and the quality of the estimation) on the Multi-modal Long-term User Recognition Dataset before deploying the MMIBN with other identifier algorithms to the real-world HRI applications.

FR does not perform equally well on users, as shown in Appendix E.3. Our proposed MMIBN models decrease the recognition bias in the system using multi-modal information. This finding is also confirmed for the uniform timing of interaction. Moreover, the first part of our hypothesis that online learning will adjust to these changes is supported, which allowed decreasing the bias of FR further. We can conclude that for long-term recognition our multi-modal incremental Bayesian networks not only perform better than FR alone in all datasets but also increases performance on each user to identify them equally well.

In our prior work [43], we proposed and applied a multi-modal weighted Bayesian network with online learning (MMIBN:OL) to a long-term HRI scenario (through the recognition architectures in Appendix A), where 14 participants (4 female, 10 male, of age range 24–40) interacted with the robot for four weeks in an office at the University of Plymouth (Figure 8). The video showing the interaction for a known user is available online.¹⁷ The study showed that our proposed approach enables and facilitates incremental identification in a real-world HRI scenario. Moreover, the optimised parameters on the real-world data showed an improvement (1.4% increase in DIR for closed-set and 4.4% in open-set) over face recognition (DIR\(=0.903\)). Furthermore, MMIBN using minmax normalisation (MMIBN_Minmax) and MMIBN:OL with softmax (MMIBN:OL_Softmax) were the best-performing methods on the data using zero weights on age and time of interaction. However, the resulting dataset was limited in terms of the number of participants and the characteristics of the participants, hence, the results and the optimisation parameters could not be generalised.

Correspondingly, we created the Multi-modal Long-term User Recognition Dataset to optimise the parameters of our models and validate them on a large number of users in varying conditions with a high variability of subject age and heights, which are highly challenging to obtain in an HRI experiment. The previous sections provided conclusive evidence that our proposed models are suitable for long-term user recognition, generalise well to new users and provide significantly more reliable identification than the state-of-the-art open world recognition model (Extreme Value Machine) and (NAOqi) face recognition alone. This section evaluates how the baselines and the optimised models in this work performed on the raw HRI data in comparison to the models in Reference [43].

McNemar test is the best statistical method for comparing two classification algorithms that are run only once [28]. Cochran’s Q test is an extension of the McNemar test for more than two groups. Thus, Cochran’s Q test is applied to compare the identification of enrolled users (i.e., DIR) and new users (FAR) of all methods separately, and pairwise McNemar using Benjamini-Hochberg adjustment for multiple comparisons is applied as the post hoc test [60]. The results show that there is a significant difference between all models (\(p\lt .001\), \(Q=161.44\), df\(=6\)) for DIR and the pairwise comparisons are shown in Table 2. MMIBN models with optimised parameters on the D-All_Uniform dataset are used as the users were randomly encountered, however, no significant differences are observed between the models that were trained on the D-All_Gaussian dataset (not shown for brevity). The results confirm that the optimised models in this work perform equally well as those that were optimised on the real-world data when the learning method is the same (e.g., comparing online learning models). Moreover, all MMIBN models significantly outperform FR (DIR\(=0.881\), L\(=0.127\)) (supporting hypothesis H1) and EVM:MM (DIR\(=0.858\)). Furthermore, the losses of MMIBN models are less than FR after only 39 recognitions. While the DIR of MMIBN_Minmax is slightly higher (DIR\(=0.932\), L\(=0.135\)) than MMIBN_Uniform (DIR\(=0.929\), L\(=0.117\)), MMIBN_Uniform has the lowest loss. Similar to the previous results, online learning does not outperform the non-adaptive model, in contrast to our hypothesis H2. EVM:FR does not perform significantly different than the MMIBN models, however, it does not reach their performance over time (Figure 9). Moreover, EVM models take substantially higher time to identify users (0.12 s for EVM:FR and 0.13 s for EVM:MM) than the MMIBN models (0.01 s for MMIBN and 0.03 s for MMIBN:OL). While there does not exist significant differences between the models in terms of FAR due to the low number of enrolled users, FR performs best (FAR\(=0.2\)), followed by MMIBN_Uniform and EVM:MM (FAR\(=0.53\)).

In a typical coffeehouse, baristas serve hundreds of customers per day and would not be able to recognise return customers or recall their preferences. A personalised robot could recognise a high number of customers, refer to them by name and recall and recommend their favourite orders, which could improve the customer experience and reduce the order time. In such an application, the customers will arrive sequentially at random times, and they need to be autonomously and incrementally added to the system with minimum time and effort from the customer. MMIBN corresponds to these requirements for incremental long-term user recognition in real-time. Consequently, the non-adaptive MMIBN with the optimised parameters on the D-All_Uniform dataset was applied for identifying customers with a personalised barista robot (using the Adapted Pepper¹⁸ robot) that recalls customer preferences [42].

A five-day HRI study with a generic (non-personalised) and a personalised barista robot was conducted in the coffee bar of an international student campus, Cité Internationale Universitaire de Paris (France), with 18 non-native English speakers (11 male, 7 female) within the age range of 22–47 (Figure 10). Speech recognition was used to make the interaction more natural, and the confirmation of the estimated identity was implicitly taken through the dialogue (i.e., if the user does not oppose the estimated identity, the identity was assumed to be correct), in contrast to Reference [43] where the user needed to explicitly confirm the identity through the tablet interface (as shown in Figure 8). Also, ground truth values (gender, age, height, and an explicitly taken image) were not requested to reduce the effort required by the customer (i.e., step nine in Figure 15 was not used), thus, only the estimated values were used for enrolling users. However, users either did not realise that the estimated identity was incorrect or the identity was incorrectly confirmed due to speech recognition errors,¹⁹ which resulted in a high FAR (FAR\(=0.786\) for MMIBN, FAR\(=0.286\) for FR) and prevented some of the new users to be enrolled, showing the necessity of explicit user confirmation. Nonetheless, MMIBN performed better (DIR\(=0.75\), L\(=0.304\)) than NAOqi FR (DIR\(=0.5\), L\(=0.479\) for 12 known user recognitions), supporting our hypothesis H1. Moreover, personalisation was found to mitigate the negative user experience, which suggests that user recognition plays an important role in long-term HRI. On average, 3.1 seconds (SD\(=0.9\)) were taken to recognise users, which includes the time for user detection and the recognition module (Figure 13) to obtain the biometric samples and the time for MMIBN to identify the user (0.01 s).

Another area where personalisation can have an impact on long-term HRI is rehabilitation. Previous research shows that personalising the therapy improves user motivation and engagement, helps clinical staff in monitoring the progress of the patient, and facilitates rapport and trust over long-term interactions [19, 70, 75, 84]. Such improvements are desirable to improve adherence in cardiac rehabilitation, which is a long-term programme offered to those who suffered a cardiovascular event to accelerate recovery and reduce the risk of suffering recurrent events through structured exercise, education, and risk factor modification [35, 52]. Thus, in collaboration with medical specialists, a personalised socially assistive robot and a sensor interface [14, 15, 41, 54] were designed and deployed for long-term (18 weeks) cardiac rehabilitation programme at the Fundación Cardioinfantil-Instituto de Cardiología (Bogotá, Colombia), as shown in Figure 11, for five months before the outbreak of COVID-19 (which halted the programme at the clinic in March 2020). Because the robot is deployed in rehabilitation with non-expert users (e.g., doctors, nurses, patients), it should be autonomous and require minimal effort from users and medical staff [30]. Accordingly, an incremental user recognition system that does not need preliminary training is necessary for personalisation of the interaction, thus, MMIBN was chosen as the user recognition method. However, because the users will be generally encountered at patterned times (i.e., at their appointments twice per week), MMIBN with online learning with the optimised parameters on the D-All_Gaussian dataset (MMIBN:OL_Gaussian) was used to evaluate its performance in a real-world interaction.

In contrast to the previous experiment [42], we used explicit confirmation of identity, in addition to the ground truth values for user enrolment, to avoid errors. The average recognition response time, which includes user detection, estimation of biometrics and identity, request of identity confirmation, the confirmation by the user on the tablet interface, and the updating of the model parameters (steps 1 to 10 in Figure 15), was 24.8 seconds (SD\(=15.5\)) for known user recognition, and 83.6 s (SD\(=39.3\)) for new user enrolment, including the user to enter the ground truth values on the tablet (steps 1 to 18). Considering that the system is used by non-experts (patients), the time required is not substantial, especially because the patients take on average 9.39 s (SD\(=17.46\)) to give a response to the tablet. MMIBN:OL took 0.04 s (SD\(=0.01\)) for recognition.

Figure 12 shows the performance of MMIBN:OL over time (with the increasing number of recognitions), and the performance of the other models on the real-world data is presented for comparison. Thirteen patients participated in the cardiac rehabilitation programme with the personalised robot, however, as observable from the figure, 30 enrolments were made to the system. The reason was a recurrent NAOqi face recognition failure that was never experienced in any of the prior studies, which resulted in erroneous user enrolments without registering the user’s image to the face recognition database, thus, DIR dropped considerably. The experimenters at the hospital addressed the issue by re-enrolling some of the patients as new users, and the issue was resolved completely after the study by adding a threshold (e.g., 0.4) on NAOqi face recognition confidence. Nonetheless, Cochran’s Q test shows significant differences between all models (\(p\lt .001\), \(Q=21.49\), df\(=4\)) for identifying enrolled users. Table 3 shows that there are significant differences between the MMIBN models and FR (DIR\(=0.34\), L=0.61), in addition to FR and EVM:MM. In contrast to our results in Section 6.5, MMIBN:OL performed slightly better than MMIBN in identifying known users (DIR\(=0.38\) for MMIBN:OL, DIR\(=0.36\) for MMIBN), notably better in identifying new users (FAR\(=0.56\) for MMIBN:OL, FAR\(=0.67\) for MMIBN), and achieved lower loss (L\(=0.62\) for MMIBN:OL, L\(=0.64\) for MMIBN), supporting our hypothesis H2, however, no significant differences are observed between the models. However, FR performed significantly better in FAR (FAR\(=0.13\), \(p\lt .001\)) than all baselines, because it identified most (63%) of known users as new. Because of the lower FAR and improving FR with re-enrolments, FR achieved a slightly lower loss than MMIBN:OL after 260 recognitions, thus, providing only partial support for our first hypothesis (H1).

While EVM:MM performs best overall in DIR and loss (DIR\(=0.42\), FAR=0.67, L=0.57), EVM:FR performs the worst of all models (DIR\(=0.36\), FAR\(=0.8\), L\(=0.66\)), which is in contrast with the findings in Sections 6.5 and 8.1. Moreover, users were not recognised for the first 29 recognitions with EVM because of its tail size parameter that was optimised on the multi-modal dataset, and lowering it gave erroneous results. In contrast, only the first four estimations of MMIBN are discarded (i.e., users were identified as new, regardless of the model estimation), as in the multi-modal dataset. Furthermore, EVM models take 0.12 s for user recognition, which is substantially higher than MMIBN models (0.01 for non-adaptive model and 0.04 with online learning). These findings further support that MMIBN models are the most reliable state-of-the-art open world user recognition method for HRI.

Overall, our findings on the Multi-modal Long-term User Recognition Dataset and the real-world HRI experiments show that both of our proposed approaches perform better in recognising users than the state-of-the-art open world recognition method (Extreme Value Machine) and the NAOqi face recognition alone, supporting that our proposed user recognition models are suitable for incremental user identification in real-world HRI, and that they improve the recognition even when the identifiers are malfunctioning.

The loss parameter \(\alpha\) (Equation (12)) should be set to find the optimum FR threshold (\(\theta _{FR}\)) and optimise the parameters in our network. As \(\alpha\) increases, the fraction of correct recognitions of enrolled users (DIR) increases, but the fraction of the incorrect recognitions of unknown users (FAR) will increase. Based on our average number of observations assumption for long-term interaction (\(\hspace{0.83328pt}\overline{\hspace{-0.83328pt}n_o\hspace{-0.83328pt}}\hspace{0.83328pt}=10\)), \(\alpha\) becomes 0.9.

For applications with fewer observations per user, \(\alpha\) can be set accordingly.

In FR, if the highest similarity score is below the face recognition threshold, \(\theta _{FR}\), then the identity is classified as unknown. We examined how \(\theta _{FR}\) influences the long-term recognition performance loss for the NAOqi FR in both D-Ten and D-All datasets and noticed a decrease in performance (i.e., increase in loss) for \(\theta _{FR}\gt 0.4\). Hence, we chose \(\theta _{FR}=0.4,\) because it is the highest threshold giving the lowest loss to decrease FAR in our model, in agreement with our previous work in Reference [43].

To determine whether the conditional independence of the modalities (face, gender, age, height, and time of interaction) given the identity of the user holds when the same input (i.e., image) is used to obtain multi-modal data, we applied structural learning of the Bayesian network on the Multi-modal Long-term User Recognition Dataset using the pyAgrum [36] library. We used the all samples dataset with Gaussian times (D-All_Gaussian) with the identification estimations obtained from NAOqi proprietary algorithms (for face, gender, and age estimations) and the artificially generated height estimations and time of interactions, as described in Section 5. Based on the requirements of the pyAgrum library, the multi-modal data is “simplified” by taking the best match evidence for modalities (i.e., confidence scores are not used) to allow structural learning. For instance, the most similar user (or unknown) is taken as the face recognition estimate by taking into account the face recognition threshold, and the evidence for gender, age, and height are taken as the estimated values. Mandatory arcs (e.g., \(I\) -> \(F\)) between the identity node and the modalities are provided as prior structural knowledge, since the multi-modal information is used to determine the identity. Based on the Bayesian Dirichlet equivalent uniform (BDeu) score [39], all three methods available in the library (K2 algorithm [22], greedy hill-climbing search, and local search with tabu list) found no other dependencies between the modalities, confirming the conditional independence.

We initially set the likelihoods to have much higher values for the true values, such as 0.9 for the face node (Equation (7)) corresponding to the actual user and \(0.99^{w_G}\) for the true gender. Average learned likelihoods in online learning for 200 users confirm this assumption, with the mean for face node as 0.913 (SD=0.126), and the mean for the gender likelihood as 0.978 (SD=0.058).

Levene’s test on the loss reveals (\(F(10,2189)=0.026, p=1\)) that there is no significant difference in variances between the repeats, which indicates that our models are stable across repeats. ANOVA (Type-I) supports that there is no significant difference between repeats (\(F(10,2189)=0.044, p=1\)), which shows that there is no significant difference between the ordered k-fold cross-validation and the shuffled k-fold, indicating that the model performs equally well for learning new users incrementally sequentially (similar to batch learning) and at random intervals (similar to a real-world scenario). Hence, we will only analyse the results of a single randomly selected repeat of 5-fold cross-validation. Since the model is stable across repeats, using a single repeat of cross-fold validation instead of independent test sets does not violate ANOVA assumption [7].

Due to the linear relation of loss with DIR and FAR in Equation (12), there will be a correlation between the parameters. Pearson’s product-moment partial correlation coefficient was computed to assess their relationships. The results show that there is a negative correlation between loss and FAR, \(r(200)=-0.18, p=.009\), a positive correlation between loss and DIR, \(r(200)=0.99, p\lt .001\), but no significant correlation between FAR and DIR, \(r(200)=0.08, p=.25\).

A factorial ANOVA is conducted for analysing the primary and interaction effects of our independent variables. The results show that there are no significant primary effects for the model, \(F(1,160)=1.50, p=.22\), and no significant interaction effects are found between the dataset size, timing of interaction, and model combination, \(F(1, 160)=0.01, p=.91\). Every other independent variable and their interactions are found to be significant (\(p\lt .001\)). This shows that the size of the dataset, timing of interaction, and normalisation method have significant effects on the performance of the model, but online learning by itself does not provide significant improvement.

A post hoc analysis using Tukey’s Honestly Significant Differences (HSD) test was conducted in which D-All and D-Ten datasets have been analysed separately for clarity, however, the results show similar patterns in both datasets. The corresponding Tukey’s HSD test plots are presented in Figures 17 and 18 (see Appendix E for the description of significance levels).

In both of D-All and D-Ten datasets, hybrid normalisation provides significantly lower loss (\(p\lt .05\)) in all conditions except for online learning in Gaussian timing for D-All (\(p=.78\) in D-All_Gaussian), in which case it still provides the lowest mean for loss. Hence, our hypothesis H3 is strongly supported, and the hybrid normalisation method is chosen for the remaining analyses.

While no significant differences are found in the primary effect of the learning method, there are significant differences between online learning and the non-adaptive model for hybrid normalisation. Online learning results in a higher loss for both datasets, which is in contrast with our hypothesis H2. The other methods do not show a stable pattern across conditions or datasets.

Most methods perform significantly worse in uniform timing of interaction (random interaction times), as compared to patterned interactions (Gaussian times), supporting our hypothesis H4. Softmax performs equally well on both models for D-All, but performs worse in uniform timing for D-Ten. Hybrid normalisation performs equally well for MMIBN in D-All but performs significantly worse in other conditions.

Hybrid normalisation performs better in all conditions and shows stability across varying conditions compared to the other methods. It achieves lower loss in D-All than in D-Ten, as a result of a higher number of samples in D-All (2,280 in open-set) as compared to D-Ten (800 samples), which shows that the proposed model gets better with the increasing number of recognitions.

It seems to be self-evident that in the case of uniformly distributed time of interaction, online learning would provide worse results, because the information provided by time will be unreliable. Hence, Bayesian optimisation should find a lower weight for the time parameter. The parameters corresponding to the optimum loss, presented in Figure 19, show otherwise. Weight for the uniform time is higher than that of the Gaussian for online learning in both datasets.

While the average standard deviation of NAOqi age estimation from the true age of the users²⁰ (i.e., the average standard deviation of error) is found to be 11.0 (which was 9.3 in Reference [43]), age is found to be the most important parameter and height the least. This is in contrast with our findings in Reference [43]. However, the results on the Multi-modal Long-term User Recognition Dataset are more generalisable to larger populations, because of the higher number of users (200) and the diverse age range (10–63), in comparison to the limited number of users (14) and the narrow age range (24–40) in our prior work. Note that in the multi-modal dataset, we used the same standard deviation of height estimations (6.3 cm) as Reference [43]. The standard deviation within age estimation (i.e., without using ground truths) is found to be 8.2, which is less than the standard deviation of error. NAOqi gender recognition rate²¹ is found to be 0.9, and no difference is found between genders, that is, females and males are recognised equally accurately. The optimised threshold for the quality of the estimation (\(\theta _Q\)) is found to be less than 0.1 in each condition. The underlying reason is the disagreement of the modalities, which can decrease the differences in posterior probabilities, because the results are combined through the product rule in the Bayesian network. When the modalities agree with high confidences (probabilities), the quality can be very high, such as \(Q=7.44,\) as shown in Figure 7 in Section 6.5 for the second user.

Figure 20 presents Tukey’s HSD test results on the training, open-set, closed-set (training), closed-set (open) evaluation sets for D-All datasets with Gaussian and uniform timing of interaction. The results show that the proposed MMIBN model significantly outperforms FR and EVM in all of the datasets.

Tukey’s HSD test results for DIR and FAR are presented in Figures 21 and 22, respectively. The plot for DIR resembles highly that of Figure 20 in a reversed direction, because of \(\alpha *(1-DIR)\) component of loss, whereby, \(\alpha =0.9\). DIR of MMIBN is significantly higher than FR and EVM in all datasets. The detailed analysis is presented in Section 6.5.2.

FAR \(=0\) in closed-sets, because all the users are previously enrolled. FR has a very low FAR in large datasets, because it predominantly identifies users as unknown. The combination of several modalities increases the probability to mistake a user for another user, which increase FAR in MMIBN.

Figure 23 presents the significant differences between the identification of users within the all samples dataset with patterned times. FR significantly performs better or worse for some of the users, whereas the combination of multi-modalities through our proposed model decreases the bias of FR. Online learning (EVM and MMIBN:OL) further mitigates the user recognition bias, in exchange for the performance, due to the accumulating noise in the identifiers.

The authors would like to thank the reviewers for their valuable suggestions for improving the presentation and analyses of the results, Valerio Biscione for his valuable suggestions in the design of the MMIBN, Pierre-Henri Wuillemin for his substantial help with the pyAgrum library, Ethan Rudd for sharing the Extreme Value Machine code, Jonathan Casas and Nathalia Céspedes Gómez for their help in integrating MMIBN within the personalised patient-robot interface for the cardiac rehabilitation programme, Mehdi Hellou for integrating MMIBN into the barista robot, the participants in our experiments for their time and efforts, and Hoang-Long Cao for his contribution of artwork in the human-robot interaction diagram (Figure 1).