At first sight: robots’ subtle eye movement parameters affect human attentional engagement, spontaneous attunement and perceived human-likeness

1.1 Aims

In this work, we investigate how different configurations of the same robot controller may affect cognition and attentional engagement of the user, as well as subjective impression of the robot’s human-likeness while maintaining the same task for all the conditions. We aim to provide roboticists with novel methods, grounded in cognitive psychology, for developing customizable controllers and for using effective strategies to configure the existing ones. To address the aims of our study, we filmed an iCub robot [8, 9] that was systematically manipulated to display two specific parameters of eye movements in the iCub controller: Trajectory Time (rTT) and Fixation Duration (rFD). rTT refers to the time required for the robot’s pupil to shift from one fixed position to one other fixed position in space. rFD refers to the amount of time that the robot’s pupil spends on a given target before moving again. We administered a rating scale to examine how these manipulations affected subjective attributions of human- likeness. Furthermore, to tease apart how human cognitive and attentional mechanisms are affected by these manipulations, we tracked participants’ eye movements, as eye movement patterns are closely related to attention and cognition [10]. We expected variations in rTT and rFD to affect subjective attributions of human-likeness as well as characteristic features of attentional engagement and cognitive resources related to observing the robot behavior.

2.1 Participants

Thirty-four participants were recruited for this experiment (mean age = 25; S.D. = 3.9 years; 21 females). All participants reported normal or corrected-to-normal vision and reported no history of psychiatric or neurological diagnosis, substance abuse or psychiatric medication. Our experimental protocols followed the ethical standards laid down in the Declaration of Helsinki and were approved by the local Ethics Committee (Comitato Etico Regione Liguria). Each participant provided written informed consent to participate in the experiment. Participants were not informed regarding the purpose of the study before the experiment but were debriefed upon completion.

2.2 Stimuli

In the current study, we estimated an average human-like TT by applying the model proposed by Baloh et al. [11], using the following formula: TT = 37ms + 2.7 * |α|, where α represents the visual angle (in degrees) between two targets. For small angles, the calculated trajectory times would be beyond the iCub physical constraints, thus we selected a visual angle of 60 degrees between the two joint positions. Based on this angle, an average TT of 200 ms was estimated. In the present study, we manipulated the velocity profile and the periodic state of TT. While the first refers to the speed of the eye movement, the latter refers to the variability displayed during the trial. Overall, we considered the implementation of the following behaviors in the iCub robot:

1. A fixed, behavior, showing no variability, calculated as an “average human behavior”.

2. A “human-range variable” behavior based on literature and human’s eye models, showing variability.

3. A “slow-range variable” behavior, designed to be considerably slower than the human-range behavior.

This design allowed us to define the effect of periodic state manipulation by comparing behavior (1) and (2), and the effect of velocity profile by comparing behavior (2) and (3). Consequently, we defined three conditions for rTT: a fixed behavior (FTT during which rTT was constant, a human-range variable (HRVTT) behavior and a slow-range variable behavior (SRVTT). With regard to rFD, previous studies suggested that for humans, the typical pause time between two subsequent eye movements (i.e. fixation duration) is approximately 200 ms [12]. We decided to refer to this lower bound and to explore the same variability range adopted for rTT. Therefore, we adopted the same approach used for rTT, defining three conditions for rFD: a fixed behavior (FFD) during which rFD was constant, a human-range (HRVFD) behavior during which rFD was variable [13], and a slow-range variable behavior (SRVFD). Then, we combined these two factors in a 3 X 3 repeated-measures design, generating 9 final conditions (Table 1).

Based on our experimental conditions, we implemented the nine different behaviors in the iCub robot. The behaviors were filmed using a 4K Handycam FDR-AX53 by Sony (Minato, Tokyo, Japan). Each video started with the robot looking straight-ahead (2 sec). Then, the eyes started moving from the initial position (I) to either position A (right) or B (left). Then, the eyes moved from one position to the other, for another ten seconds. Immediately after, the robot eyes returned to the I position. Subsequently, the head of the robot turned either to the right or to the left with a 70 degrees amplitude. Eventually, the head and the eyes returned to the initial position. The manipulation of occasional head movement was introduced in order to test whether participants would engage in a spontaneous attentional following (measured by fixations that would land laterally with respect to the face, in the same direction as the head movement) and whether their spontaneous attentional following would depend on the robot behavior. Importantly, for the measures of the spontaneous attentional following, the stimuli remained identical across conditions (i.e., the head movement was always the same). Therefore, any differential effects would be due to some sort of “priming” by preceding robot behavior (fixed, variable, human-range or slow-range). Each behavior was filmed twice (one starting from I to A and one starting from I to B). Consequently, we filmed 18 videos to be used for the experiment.

2.3 The iCub’s gaze controller

In this study, we used the Cartesian 6-DoF gaze controller developed for the iCub robot [14]. We implemented this to control the trajectories of the neck (TN) and of the eyes (TE) of the robot for looking at 3D Cartesian fixation points in space. This controller fits well with our requirements since it allows specifying the point-to-point execution time for the neck (TN) and the eyes (TE). Therefore, we implemented the robot’s eye movements simply by tuning the TE parameter, considering the nine different conditions shown in Table 1. We implemented a Python script interfacing with the IGazeController Yarp class [15]. The functions for controlling both the eyes and the neck are shown in Listing 1. In the moveEyes method, we controlled the robot to move the eyes between two pre-defined fixation points for a total duration of 10 seconds. The fixation points are provided in relative angles (azimuth, elevation, and vergence) according to the controller’s specifications. Once 10 seconds had elapsed, the moveNeck method was called for shifting the gaze in a pre-defined location (to the left or right). In this case, we released the block on the neck to let the robot rotate the head along the yaw angles.

2.4 Apparatus

The experimental session took place in a dimly-lit room. Stimuli were presented on a 22” LCD screen (resolution: 1366 x 768). A chinrest was mounted on the edge of a table, in order to maintain a distance of 63 cm between participants’ eyes and the screen for the entire duration of the experiment. Consequently, the forward-looking robot’s face subtended 5.5o_ by 7.1o_ of visual angle. Whereas a range of eye-tracking techniques has previously been used in robotics research (see [16] for review), infrared / near-infrared videobased eye-tracking was deemed most suitable for our application [17, 18]. Therefore, we used a screen-mounted SMI RED500 eye-tracker by iMotion (SensoMotoric Instruments GmbH, Teltow, Germany) with a sampling rate of 500 Hz and spatial accuracy of 0.4o_ to record binocular gaze data. The experiment was programmed in and presented with OpenSesame 3.1.8 [19] using the Legacy backend and the PyGaze library [20].

2.5 Procedure

We instructed participants to carefully watch the videos and to evaluate, on a 6-point scale how much the behavior displayed by the robot was human-like (0=not at all, 5=extremely). Each video was repeated six-times across the whole experiment (108 trials in total) in a random order of presentation. The experiment consisted of four blocks of 27 trials to allow for self-paced breaks. During each trial, log messages were sent to the eye-tracker at the video onset, the onset of the robot head movement, and the video offset. For details on the trial structure, see Figure 1. Prior to the task and before starting the second half of the experiment, a 9-point calibration and 4-point validation thereof were carried out (mean accuracy = 0.89°; S.D. = 0.70°). Additionally, participants were recalibrated when deemed necessary (e.g. when a participant moved their head from the chinrest).

Figure 1

A planar view of the iCub robot used in this study. The picture depicts the typical trial sequence, illustrating the iCub’s behavior (staggered panels) and the subsequent human-likeness rating screen (bottom-right). In this example, the robot moves from I to A and turns the head to the right.

2.6 Analyses

In order to explore the potential effects of rTT and rFD on participants’ ratings, a mixed model was applied in R Studio [21]. We considered participants’ responses as the dependent variable and intercept as a random factor. Then, we computed rTT and rFD as fixed factors of the model. This procedure allowed us to explore the main effects of the single factors and the effect of interaction between the two. Pairwise post hoc comparisons were estimated using the Tukey method. For the purpose of investigating whether the eye movements implemented in the iCub controller affected participants’ eye movements during the videos, analyses on the eye-tracking data were performed. Three participants (mean age = 24.67; S.D. = 1.53; 1 female) were excluded from these analyses due to technical errors in the eye-tracker recordings. In order to facilitate processing of the eye-tracking data, we defined an Area of Interest (AoI): the eye region of the robot. (see Figure 2 for details). For each participant, we calculated proportional dwell time (the amount of recorded gaze samples within the AoI regardless of eye movement type) and total fixation count to investigate where our participants attended. We examined the average fixation duration to underpin the temporal characteristics of these mechanisms on the AoI per condition between the onset of the video and the iCub’s head movement. Finally, we investigated whether a fixation lateral to the face occurred within 2500 ms after the robot head movement in each trial as a measure of spontaneous gaze following. We then processed the landing position -the horizontal vector- of the first lateral fixation in the same direction as the head movement. Considering the skewed distribution of our data, we computed these metrics as the dependent variables of different Mixed Models. Each model’s output provided us with predicted values of all the metrics input as dependent variables. Such predicted values derive from raw data and were corrected based on the effects taken into account in the Mixed Models. In order to maximize comprehensibility and graphical rendering of the effects, we plotted predictive values instead of the raw data. Subsequent pairwise post hoc comparisons were estimated using the Tukey method.

Figure 2

A frame of the trial sequence depicting the iCub robot. The figure indicates the Area of Interest (AoI) used for the analyses, defined as the yellow rectangle.

3.1 Subjective reports

For the human-likeness ratings, we found that the slow-range variable (SRV) condition was evaluated as most human-like. We observed a main effect of condition both on rTT (F=7.67, p<0.001, Figure 3 left) and on rFD (F=34.61, p<0.001, Figure 3 right). Post hoc comparisons revealed that our participants evaluated the SRV condition significantly more human-like than the other two conditions. No significant differences were found between the fixed and human-range conditions (see Table 2 for detailed comparisons).

Figure 3

Human-likeness ratings across the study. Predicted values are plotted on the y-axes in order to facilitate the interpretation of results. Vertical bars denote +/- 1 standard error.Horizontal bars denote differences surviving post hoc comparison. Figure on the left shows Robot’s Trajectory Time (rTT) main effect on participants’ ratings (F: Mean=2.08, S.D.=1.30; HRV: Mean=2.12., S.D.=1.29; SRV: Mean=2.25, S.D.=1.32). Figure on the right shows Robot’s Fixation Duration (rFD) main effects on participants’ ratings (F: Mean=2.02, S.D.=1.12; HRV: Mean=2.08., S.D.=1.35; SRV: Mean=2.35, S.D.=1.33).

3.2 Objective measures: eye-tracking data

The eye-tracking data is visualized in Figure 4. Heat maps suggested the presence of differences in fixation patterns due to TT manipulation. Our results in the objective implicit eye-tracking measures showed a somewhat different pattern than the subjective explicit reports.

Figure 4

Fixation heat maps for each rTT condition across the experiment. On the left: trial onset until head movement; on the right: head movement until the end of the trial).

3.2.1 Human-range variability condition (HRV)

Importantly, the eye-tracking data showed that at the implicit level of processing, human-range variability (HRV) engaged participants’ attention more than the slow-range variability (SRV) – a differential effect on fixation durations, and evoked higher degree of spontaneous joint attention than the other two conditions – an effect on the range of lateral fixations and speed of attentional following.In more detail, our results showed that participants fixated on the eye region longer for the HRV condition, as compared to SRV, as evidenced by the significant main effect of rTT for the eye region (F=4.84, p=0.01) in the average fixation duration, and significant difference between HRV and SRV, (z=3.03, p=0.01), planned comparison. No significant differences were found between F and HRV (z=-2.13, p=0.08) or between SRV and F (z=0.93, p=0.63) (Figure 5, left). Furthermore, participants showed a higher degree of spontaneous attentional following in the HRV condition, relative to the other conditions: analyses on lateral fixations recorded after the robot head movement revealed a significant main effect of rFD on the horizontal vector of the first fixation location (F=4.41, p=0.01). Planned comparisons revealed a significant difference between F and HRV condition (t=-2.81, p=0.01). A marginal difference was found between HRV and SRV condition (t=2.24, p=0.07), while no differences were found between F and SRV (t=-0.58, p=0.83) (Figure 5, right).

Figure 5

Fixation patterns across the study. Predicted values are plotted on the y-axes in order to facilitate the interpretation of results. Vertical bars denote +/- 1 standard error. Horizontal bars denote differences surviving post hoc comparison. Figure on the left shows Robot’s Trajectory Time (rTT) main effect on participants’ fixation duration (F: Mean=439ms, S.D.=309ms; HRV: Mean=460ms., S.D.=321ms; SRV: Mean=428ms, S.D.=275ms). Log-transformed (Ln) predicted values are plotted on the y-axis in order to facilitate the interpretation of results. Figure on the right shows Robot’s Fixation Duration (rFD) main effects on the absolute horizontal vector of the first fixation. (F: Mean=2.6°, S.D.=1.8°; HRV: Mean=4.2°., S.D.=3.0°; SRV: Mean=3.8°, S.D.=2.4°). Location was estimated as the distance from the center of the iCub face. Predicted Blom inverse normal transformed values (INT) are plotted on the y-axis in order to facilitate the interpretation of results.

3.2.2 Fixed-behavior condition (F)

Interestingly, also the fixed behavior condition elicited distinctive gaze patterns, compared to other conditions, which was also – similarly to the HRV condition – not in line with the subjective ratings. Specifically, we found that participants’ gaze was directed toward the eye region significantly more (in terms of dwell times) during the Fixed-behavior condition, than during the other two conditions, as evidenced by a main effect of rTT on participants’ proportion of dwell times on the eye region of the iCub (F=4.01, p=0.02), and the significant differences between SRV (z=2.48, p=0.04) and HRV (z=2.43, p=0.04) conditions (Figure 6, left). Analyses did not reveal any effects of rFD on eye region dwell time or of either TT or FD on the dwell time on the whole face. Furthermore, a significant main effect of rTT was found on the number of fixations that occurred in the eye region (F=4.46, p=0.01). In the Fixed-behavior condition, the number of fixations was larger than in the HRV condition (z=2.96, p=0.01). No significant differences were found between SRV and F (z=1.12, p=0.50) or HRV and SRV (z=-1.82, p=0.16) (Figure 6, right).

Figure 6

Fixation patterns across the study. Predicted values are plotted on the y-axes in order to facilitate the interpretation of results.Vertical bars denote +/- 1 standard error. Horizontal bars denote differences surviving post hoc comparison. Figure on the left shows trajectory Time (rTT) main effect on participants’ proportional dwell time (eye region) (F: Mean=0.82, S.D.=0.27; HRV: Mean=0.80., S.D.=0.29; SRV: Mean=0.80, S.D.=0.28). Normalized predicted values of dwell time are plotted on the y-axis in order to facilitate the interpretation of results. Figure on the right shows trajectory Time (rTT) main effects on participants’ amount of fixations (F: Mean=20.41, S.D.=9.89; HRV: Mean=19.51, S.D.=10.11; SRV: Mean=20.03, S.D.=9.83). Predicted values of fixation count are plotted on the y-axis in order to facilitate the interpretation of results. Vertical bars denote +/- 1 standard error. Horizontal bars denote differences surviving post hoc comparison.

3.2.3 Slow-range variable condition (SRV)

The slow-range variable condition elicited a distinctive pattern only in the frequency of instances of attentional following, and this is the only result from the implicit measures that follow the explicit subjective reports. The generalized linear model revealed a significant main effect of rTT on the occurrence of spontaneous gaze following (χ² (2)=6.82, p=0.03). Specifically, planned comparisons revealed a significant difference between SRV and HRV (z=-2.46, p=0.04). No difference were found between F and either HRV (z=0.48, p=0.88) and SRV (z=-1.97, p=0.11) (Figure 7).

Figure 7

Spontaneous gaze following of participants during the study. Robot Trajectory Time (rTT) main effect on participants’ on the proportion of participants’ occurrence of gaze following behavior (F: 35.7% of total F events, HRV: 35.7% of total HRV events, SRV: 39.6% of total SRV events). Logit transformed predicted proportion is plotted on the y-axis. Vertical bars denote +/- 1 standard error. Horizontal bars denote differences surviving post hoc comparison.