Beyond Basic Tuning: Exploring Discrepancies in User and Setup Calibration for Gaze Estimation

Gaze is vital in interpersonal communication, transcending linguistic and cultural barriers. Interpreting gaze direction enriches understanding and facilitates the transmission of emotions, intentions, and cognitive states [Abele 1986; Kleinke 1986]. This holds significance in diverse fields from psychology to artificial intelligence, including human-computer interaction (HCI) [Hessels 2020; Ladouce et al. 2017]. For widespread technology adoption, success hinges not only on technical merits but also on user experience. Comfort and accessibility are crucial, especially beyond specialized environments [Godoi et al. 2020; Li and Liu 2021].

Accurate gaze estimation can be achieved under certain conditions: specialized optical elements; controlled illumination (including infrared sources), and precise image focusing on the ocular region. However, challenges in guaranteeing these conditions in uncontrolled environments and the cost of specialized hardware limit these solutions [Hansen and Ji 2010].

In this context, there is a need to improve existing solutions for what we will refer to as general-camera environments, which make use of off-the-self devices, such as webcams, camcorders or smartphone cameras, for example.

Machine learning has revolutionized computer vision, with neural networks becoming the de facto standard [Bulat et al. 2020; Dosovitskiy et al. 2021; Karras et al. 2020]. Gaze estimation, facilitated by neural networks, democratizes technology use in everyday scenarios [Ansari et al. 2021; Bulat et al. 2020; Dosovitskiy et al. 2021; Guo et al. 2019; He et al. 2019; Karras et al. 2020; Yu and Odobez 2020]. Some of these solutions make use of off-the-self devices such as webcams, camcorders, or smartphone cameras, that seek to bring technology closer to the end user and day-to-day life [Ansari et al. 2021; Guo et al. 2019; He et al. 2019; Yu and Odobez 2020].

Regardless of the nature of the model, calibration enhances its performance. This procedure improves the accuracy addressing two aspects: user calibration and device/environmental calibration (Equation 1). User calibration customizes the system to accommodate individual visual traits like kappa angle, ocular morphology, accessories (e.g., glasses), etc. In contrast, environmental calibration addresses changes in the physical conditions of the testing system, including alterations in lighting, camera position relative to the scene (e.g., screen), typical working distance and more.

Although the importance of calibrations is not questionable in high-performance (infrared) eye tracking, it is still unexplored in general-camera solutions scenarios, and due to the paradigm shift of orienting solutions towards this reality, we consider that is necessary to conduct specific studies on the calibration process and its impact in this framework. Existing works that made use of calibration on neural network solutions focus on how to perform it, but not on its specific analysis [Chen and Shi 2020; Gudi et al. 2021; Linden et al. 2019; Liu et al. 2019; Park et al. 2019; Yu and Odobez 2020].

In this work we will focus on distinguishing the role of personalisation of the system according to the user and to the system's physical configuration for general-camera solutions. We make the following hypotheses.

• H1: Given the dual nature of gaze estimation system calibration, in a general-camera solution isolating one of the calibration components (user or setup calibration) and calibrating solely over it still has a significantly positive impact on the system's performance, albeit not at the same level as a classical calibration.
  
• H2: The impact of each calibration component, i.e. setup calibration and user calibration, is differentiable from one another.
  
• H3: Calibration is not generalizable across users and setups.
  

In this section we will detail the elements and methodologies adopted during the development of this work.

2.2 Database for calibration

The models constructed using pretraining databases will be tested using a dataset built within this work. The setup used 6 cameras distributed around a screen, on the same plane, maintaining the same depth with respect to the user, i.e. 60 centimeters approximately. Figure 1 shows a representation of the setup and the distribution of the cameras.

The recording involves displaying two videos of a calibration grid of 4x4 points while the cameras record the scene. Each point appears randomly without replacement. Two corresponding grids and point sequence are assigned to each session so different users undergoing the same session are presented with the same sequences.

To help with the synchronization of the cameras, a LED visible from all the cameras activates when the user clicks the mouse. During video capture, the user is instructed to click the mouse when looking at each point, so that the LED lights up momentarily.

25 users participate in the experiments. Of these users, 14 were men and 11 women, in the age range between 50 and 20 years old. Each user performed 10 sessions and each session consisting of 2 calibration videos, giving a total of 500 videos recorded by each of the 6 cameras. For each calibration point, 3 frames were extracted per video. In Figure 2 we find a representation of some of the captured images and users.

2.2.1 Privacy and ethics regarding personal data. The databases used in eye-tracking contain sensitive information, as visual interaction is a personal and unique aspect of each individual. Therefore, special attention must be given to the storage of these databases and the models trained with them. Participants in the study were informed about potential risks and, in accordance with data protection legislation, provided their consent. Sharing the database will be considered upon prior contact with the authors, who are the custodians of the data.

2.3 Model architecture

We propose a workflow aligned to the development of an application: a base model, trained extensively over pre-training databases, as described in 2.1, would then be calibrated by the user, to adjust it to its specific conditions.

Due to neural networks’ opaque behavior, particularly with abstract inputs [Fisher et al. 2019; Simonyan et al. 2014], we opt for features instead of images for better control. Our focus is on measuring the impact of calibration, not finding the perfect model or feature set. Simplicity aids in understanding how calibration tweaks influence performance metrics.

Two feature-based models are used, pre-trained with features from the U2Eyes synthetic database [Porta et al. 2019] and the EVE real database [Park et al. 2020]. This accounts for potential biases between synthetic and realistic models. Figure 3 illustrates the features, automatically extracted using a purpose-trained model [Larumbe-Bergera et al. 2021].

The employed model is a fully connected layers-based model with the following architecture:

• An input of 24 features. It consists of a vector of the pixel coordinates (x, y) of the 12 features extracted from the images (Figure 3). The values are normalized relative to the size of the images in each dimension (1920x1080).
  
• Intermediate stages following the pattern: N-neuron layer, N-neuron layer, dropout (0.2), with the values of N being [512, 128, 32, 16].
  
• An output of 2 values corresponding to the (x, y) components of the Look-At-Point (LAP). The values are normalized between -1 and 1 based on the LAPs present in the training dataset (within a maximum range of 0-1920 horizontally and 0-1080 vertically).
  

The design of the experiments is inspired by usage scenarios that we believe may occur during the real-world application of general-camera solutions. We outline the model's pre-training, considering two scenarios: synthetic database pre-training (Section 2.1.1) and real-world database pre-training (Section 2.1.2). This results in the two different baseline models upon which various calibrations will later be performed. Three different experiments are devised: classic calibration, user calibration and setup calibration. Each of the experiments is repeated for each of the two baseline models and, within these, for each user and each camera.

The same base procedure is followed in the case of the three proposed experiments: we start from a pre-trained base model; then the pre-trained base model is calibrated (fine-tuned with calibration data) and, finally, that calibrated model is tested. The calibration and testing sets are defined by the conditions of each experiment, which will determine which combination of users and setups are used in each case. The calibration and test stages are carried out using users belonging to our dataset as explained in section 2.2.

The same fitting conditions are maintained for the three experiments: we use a batch size of 32 inputs, Adam optimiser, MSE as training and validation loss, an initial learning rate of 0.01 and we fit the model during two phases of 100 epochs, applying the early stopping technique in case there is no improvement in the validation loss during 15 epochs. In the second phase, we reduce the learning rate to 0.001. To prevent overfitting, we employ ridge regression or L2 regularisation, with a value of 0.0005.

In addition to the verbal descriptions of the experiments, visual representations of all three can be found in the Appendix A, offering a supplementary graphical insight into the procedures and outcomes discussed.

The difference in degrees between the estimated and the actual LAP will be used as figure of merit. The way to calculate the corresponding angular error follows the formula below:

Table 1 presents the angle results before any calibration, to serve as reference for the results obtained from the experiments. In Tables 2, 3 and 4 we find a summary of the results obtained for the three experiments, but addressing different analysis. Table 2 aggregates all test cases, without distinguishing between user and/or camera. Table 3 groups the results of each experiment according to test user, while Table 4 groups by the test camera. Similarly, Figure 4 is presented encompassing the different analyses. Subfigure (a) represents a comprehensive analysis considering all tests conducted. Subfigure (b) performs a similar analysis, but this time grouping results based on the test user, utilizing information from Table 3. Finally, subfigure (c) displays grouping based on the testing camera, corresponding to Table 4.

The experiments for this paper were guided by two goals:

• To determine the continued relevance of calibration in general-camera gaze estimation systems.
  
• Study the influence of capture hardware and user on calibration.
  

A custom multi-camera setup was developed to capture diverse views of the same user during the calibration process. Subsequently, data from multiple users were utilized to train a feature-based model, simulating both classical calibration and variations isolating user and setup conditions.

The first conclusion is that calibration remains a crucial process for achieving accurate gaze estimation results, surpassing the capabilities of calibration-free systems. Developing a model capable of generalizing across all setups and individual characteristics, a prerequisite for calibration-free remote gaze estimation, proves highly challenging

Among the studied setup and individual conditions, components related to the individual have a slightly more pronounced impact on calibration compared to those related to the setup. Challenges arising from individual characteristics are more difficult to address during general model training. In a multi-user work environment, the emphasis on user adaptation can potentially limit the effectiveness of these solutions, particularly when compared to specialized device-based alternatives.

Regarding physical components, increasing the variety of capture setups, incorporating multiple views, or transitioning to 3D gaze vector estimation can potentially mitigate limitations. However, for user-related components, leveraging their personal characteristics appears necessary, with uncertainties about the effectiveness of multiple views or 3D training in this context.