THÖR-MAGNI: A Large-scale Indoor Motion Capture Recording of Human Movement and Robot Interaction

The presence of goal-directed human agents is crucial in the field of human motion prediction (Dendorfer et al., 2021; Zhao and Wildes, 2021; Chiara et al., 2022). Traditional approaches often depict human agents as rational entities, acting logically and moving towards specific goals or destinations (Ziebart et al., 2009). Real-world recordings commonly show this directional traffic flow, characterized by distinct goal points, often resulting in a consistent and linear motion with limited diversity. In our dataset, we include scenes with seven distinct goal points distributed over a larger spatial volume, and scenes where they are arranged in a more compact space (see Section 4). Goal points and static obstacles are positioned strategically to ensure that recorded trajectories are not only sufficiently long but also topologically diverse, i.e. covering a range of spatial arrangements and configurations. This approach allows for the inclusion of frequent interactions between the moving agents, contributing to a more comprehensive understanding of human motion dynamics.

Heterogeneous agents are dynamic entities that navigate with distinct motion patterns. This heterogeneity stems from various factors that affect the motion such as tasks and ongoing activities performed by the agent (de Almeida et al., 2023). For instance, several works have studied how humans move individually or as part of a social group (Moussaïd et al., 2010; Rudenko et al., 2018; Wang et al., 2022). It has been shown that humans can coordinate their movements as a group by following simple rules based on the visual perception of local motion (Boos et al., 2014). Previous research on the anatomy of leadership in collective behavior (Garland et al., 2018) describes human collective behavior as optimal coordination and leadership dynamics in various group scenarios. In particular, crowd dynamics are not only determined by physical constraints, but also significantly influenced by communicative and social interactions among individuals (Moussaïd et al., 2010). Autonomous driving datasets often highlight the motion of heterogeneous agents in mixed traffic (Chandra et al., 2019; Salzmann et al., 2020). In our dataset, we introduce roles for participants tailored for industrial tasks, such as navigating alone or in groups of different sizes, transporting various objects and interacting with a robot. This heterogeneous social setting provides a novel way to study how specific industrial roles influence human motion, aligning with the work conducted by de Almeida et al. (2023).

Human-aware robot motion planning is crucial for safe navigation in shared spaces, especially in the narrow and crowded indoor environments (Cancelli et al., 2023). Understanding human interaction with robots of different driving styles promotes the design of socially acceptable motion planners (Möller et al., 2021). Analyzing participant behavior with robots of varied movement patterns reveals insights into how robot motion style affects human expectations (Karnan et al., 2022), guiding the development of robots that interact safely and are well-received by people (Shah et al., 2023). Our dataset features scenarios with a mobile robot in teleoperated and semi-autonomous modes and two driving styles: differential drive (forward, backward and turning) and omnidirectional mode (allowing the robot to drive in any direction while keeping its heading). This variety of motion modes (detailed in Section 4.2.2) extends the state-of-the-art datasets of teleoperated navigation which feature a single driving style (Karnan et al., 2022).

The concept of Industry 5.0 aims to prioritize human well-being in manufacturing systems (Leng et al., 2022). This requires enhancing the quality of human-machine and human-robot interactions in these environments. Designing robots that can clearly express their intentions to human collaborators is a crucial step towards fostering mutual understanding and enhancing the well-being of human workers who regularly interact with robots (Pascher et al., 2023). Furthermore, intuitive human-robot interaction (HRI) not only improves well-being but also enhances safety and efficiency in collaborative settings (Haddadin et al., 2011).

Spatial HRI (sHRI) and navigation in shared environments are research areas that have an adherent need for accurate datasets of human motion tracking and prediction (Rudenko et al., 2020a; Chen et al., 2022) and for robots that understand the underlying physical interactions between nearby agents and objects (Castri et al., 2022). Our dataset contains recordings of explicit interactions between a mobile robot and individuals in shared workplace settings. THÖR-MAGNI is a valuable resource for studying human responses to robotic approach and assistance initiatives, enabling researchers to analyze goal-oriented interactions between humans and robots.

Eye tracking is a powerful method to study various aspects of human behavior, including attention, emotion, cognition, and decision-making, with applications spanning education, marketing, gaming, and healthcare (Duchowski, 2017). Eye tracking provides objective data about eye movements and positions and enables researchers to quantify visual information processing through various metrics (Duchowski, 2017; Mahanama et al., 2022). In HRI applications, human eye-gaze is an important nonverbal signal (Admoni and Scassellati, 2017). In our dataset, we align human gaze data with human motion trajectories, providing an opportunity to study human gaze during visual exploration across dynamic tasks, activities, and scenarios.

Complementary to gaze, head orientation provides another essential modality of human behavior, closely related to gaze direction and attentional focus. Head orientation plays a vital role in joint attention, i.e. the coordination of attention between individuals focusing on the same point of interest (Tomasello, 2014). Furthermore, it is valuable for detecting interpersonal dynamics in multi-party interactions (Stiefelhagen and Zhu, 2002). Beyond its social implications, head orientation becomes a predictive indicator of walking motion goals (Holman et al., 2021) and can enhance human motion prediction through vision-based features (Salzmann et al., 2023). Using a state-of-the-art motion capture system and eye-tracking devices, our dataset provides highly accurate head poses and orientations aligned with the eye tracking data.

Crucial environmental information, conveyed by semantic cues such as doors, stairs, floor markings, and signs, plays an important role in guiding humans and robots within a given space. These cues, combined with obstacle configurations, influence human interactions with the environment, leading to actions like detouring, bypassing, overtaking, and avoiding specific areas. In our dataset, we include different semantic cues like markings on the floor indicating areas to be cautious of the environment or one way passages that limit the flow of motion in one direction. In this way, we enable the exploration of navigation and interactions in semantically-rich environments. For instance, leveraging Maps of Dynamics (Kucner et al., 2023) allows the quantification of motion patterns changes around these cues. This information, in turn, can be utilized to predict long-term human motion dynamics, as demonstrated by Zhu et al. (2023).

We conducted the data acquisition in a laboratory at Örebro University, the same as in the THÖR dataset (Rudenko et al., 2020a). There are two different configurations for the laboratory. One features a small but free-space environment(see Figure 2 left). The other resembles an industrial logistics setting and promotes frequent interactions between human and robotic co-workers (see Section  4.3). Both room configurations have seven goal positions, to drive purposeful human navigation through the available space, generating frequent interactions in the center. Additionally, we include several environmental layouts (i.e., obstacle maps) in the THÖR-MAGNI dataset, which vary the placement of static obstacles (robotic manipulators and tables) in the room to prevent walking between goals in a straight path. Apart from static obstacles, two robots are in the room: a static robotic arm near the podium and an omnidirectional mobile robot with a robotic arm on top (see Section 4.2.2).

The interaction and navigation design in THÖR-MAGNI extends the weakly-scripted motion recording procedure introduced in the THÖR dataset (Rudenko et al., 2020a). This procedure facilitates realistic motion in controlled settings, in which accurate ground truth motion capture and eye tracking data are collected using specialized equipment (see Figure 2 on the right). Our key idea is to assign meaningful activities and tasks to the recording’s participants, allowing them to concentrate on their continuous activity during which they freely move in the room shared with other people and robots. To generate a diverse range of interactions, we developed several scenes that vary in the composition of tasks, robot operation, and other contextual cues, as discussed in Section 3.

We address the context of agent movement by including both humans and robots, as previously discussed, in five specifically designed scenes we call “scenarios”. Scenario 1 captures the dynamics of motion because of semantic attributes of the environment and sets up a baseline for goal-directed social navigation. Scenario 2 adds role-specific motion for some participants navigating the environment. Subsequently, Scenario 3 explores the impact of different robot motion styles on these role-specific patterns. Figure 5 depicts a detailed overview of the room configuration and varying environmental layouts for Scenarios 1–3. Scenario 1’s conditions A and B capture regular social behavior in a static environment with and without additional floor markings and a one-way passage. Scenario 2 maintains the same layout as Scenario 1A but introduces individuals performing tasks, emulating industrial activities. Scenario 3 explores human-robot interactions by varying the driving modes of the mobile robot teleoperated by experimenters on a podium.

Transitioning to a smaller room configuration, we present two scenarios to explore human motion and intended interactions between humans and robots: Scenarios 4 and 5. Scenario 4’s participants engaged in intermittent interaction with a mobile robot. This robot communicated in two interaction styles through another entity to mediate joint navigation with participants toward goal points. In Scenario 5, the robots and a human co-worker collaborated actively in transporting small storage bins. For a comprehensive overview of roles and scenarios, see Figure 6.

We recorded multiple runs for each condition in Scenarios 1–5. Specifically, we recorded two runs per condition for Scenarios 1 and 3, two for Scenario 2, four per condition for Scenario 4, and four runs for Scenario 5. To counterbalance learning-based effects, we randomized the recording order of conditions for Scenarios 3 and 5. We implemented this methodical approach to ensure a broad and impartial exploration of the scenarios, capturing subtle interactions and behaviors in each setting.

The average age of the participants was 30.18 years, with a standard deviation of 6.73, indicating a relatively homogeneous age group. The dataset contains a balanced gender distribution with 40 participants, of which 21 are male and 19 female. Geographically, 23 participants are from Sweden. From other European countries, there are 10, including the Czech Republic, Spain, Germany, and Italy, reflecting a diverse European representation. The remaining 7 participants come from countries on other continents like Asia, Africa, and South America, providing a broader international scope. We recruited the participants from different areas of the campus. Their backgrounds varied considerably, including differences in their highest academic degree and primary subjects. At the beginning of each recording day, participants completed a demographic questionnaire. We used this information to create diverse group compositions, aiming for optimal allocation of eye tracking devices across different roles (see Figure 10). For example, we ensured that groups of two or three participants contained only one participant equipped with an eye tracker and the equipment of at least one of the carriers with an eye tracker.

During the data collection procedure, we guided the participants through a series of runs with specific instructions tailored to each scenario. Between successive runs, participants complete questionnaires while logistical preparations are made, such as removing floor markings, configuring a phone for voice chat using Discord (before Scenarios 2 and 3), monitoring and, if necessary, changing the batteries of eye trackers, and preparing the robots for Scenarios 3–5. After completing the questionnaire, participants are assigned new roles in Scenarios 2 and 3. We gave each group a new starting point for the next run, from which they drew their first card. Participants unfamiliar with their roles got a brief recap of their task-related responsibilities. In Scenario 3, we informed participants that an experimenter monitored the robot’s motion for safety and teleoperated the robot. In Scenarios 4 and 5, participants were first briefed about their roles in the scenario (see Section 4.3) and then introduced to the ARMoD and the DARKO robot as co-workers in the room, with the ARMoD acting as a communicator on behalf of the DARKO robot.

After each run, participants completed the raw version of the NASA Task Load Index (RTLX) (Hart and Staveland, 1988; Hart, 2006). The scale consisted of a 21-point set of subscales [1 = low; 21 = high], each of which assessed the mental demand, physical demand, temporal demand, and frustration produced by the task as reported by the participant, as well as their self-perceived performance and frustration. After each session of the last run of Scenarios 3 or 5, participants complete two additional mobile robot questionnaires. First, they complete the Godspeed Questionnaire Series (Bartneck et al., 2009), a semantic differential set of subscales [5-point] that measures participants’ perceptions of the robot in terms of anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety. Second, they complete a 5-point Likert scale [1 = strongly disagree; 5 = strongly agree] to assess trust in the robot in industrial human-robot collaborations (Charalambous et al., 2016). Participants complete all questionnaires on paper.

Multi-modal data synchronization was necessary in our data collection. We used ROS and custom Python scripts to align the data streams while maintaining temporal integrity. To achieve synchronicity between the motion capture and eye tracking data, we strategically placed custom events associated with precise timestamps in the two data streams using the respective software of the eye tracking devices such as Tobii Pro Lab (Tobii AB, Accessed: 2024-02-02c) and Pupil Player (Pupil Labs AB, Accessed: 2024-02-02a) as well as the Qualisys Track Manager (QTM) (Qualisys AB, Accessed: 2024-02-02) for the motion capture system. This procedure resulted in CSV files where all modalities’ timestamps are synchronized on the motion capture system’s timestamp. Within these files, eye tracking data is available for frames where all markers of the rigid body are tracked by the motion capture system, as it is a prerequisite to determine the 3D gaze vector using a correct head orientation. The frame numbers for each respective eye tracker’s scene recording are indexed in the column named “SceneFNr” in the corresponding CSV file.

To facilitate a thorough analysis of the eye tracking data in our study, we offer access to the raw data from the Tobii glasses, along with essential synchronization details. To ensure data protection, the scene recordings are provided in a blurred format and removed audio data. Access to the raw data from the Pupil Invisible glasses can be granted upon individual request, ensuring careful and ethical distribution of sensitive data.

An extensive post-processing stage, including the above mentioned synchronization and alignment, followed the data acquisitions. Its aim was to refine and validate the collected data as well as to ensure the protection of sensitive data. This stage involved several key procedures, such as eliminating artifacts and noise caused by marker occlusion, lighting variations, and camera disruptions. We also rectified misidentified trajectories through a combination of spatial and temporal consistency evaluations, with manual adjustments applied when needed.

For the purpose of dissemination, the dataset has been categorized into five recording scenarios (see Section 4 for a detailed description), aligning with the respective days of data collection. Each scenario’s data is organized into separate folders. Within each folder, multiple acquisitions conducted over the five days of recording are stored. The folders corresponding to the first three scenarios (1–3) contain acquisitions from four days (in May 2022), while the folders representing the last two scenarios (4 and 5) encompass recordings from one day (in September 2022). To enhance the diversity of motion data in the recordings and mitigate random artifacts, we record multiple runs for each scenario and condition. It is essential to note that all files are intended to be extracted into a common directory. In this way, the arrangement preserves the temporal structure of the recorded data.

Each run’s data includes a CSV file and up to two .mp4 videos representing the recordings from the scene cameras of the Tobii eye trackers and if the robot was in motion during the Scenarios 3–5 continuous 3D point clouds from the Ouster lidar as well as the RGB videos from one of the fish-eye cameras. The structure of the recorded data is shown in the Tables 3, 4, and 5 in the Appendix. In the following subsections we will provide more specific details on the usage and processing of the individual files.

The majority of existing datasets in the field lack a dedicated toolbox for streamlined visualization and preprocessing. Addressing this gap, we contribute a set of data visualization tools, including a dashboard, and introduce a specialized Python package named thor-magni-tools. This package is designed to facilitate the filtering and preprocessing of raw trajectory data, enhancing the accessibility and usability of the THÖR-MAGNI dataset. By making available these resources, we aim to provide researchers with versatile and fast means to navigate, analyze, and extract valuable insights from the dataset.

To evaluate the trajectory data of our dataset in comparison to previous data collections, we employ metrics proposed by Rudenko et al. (2020a) and Amirian et al. (2021):

• •

  Tracking Duration ($\text{\,}\mathrm{s}$): this metric represents the average duration of continuous tracking for all human agents. A higher value indicates longer tracking, which is favorable for long-term human motion prediction methods.

• •

  Minimal Distance Between People ($\text{\,}\mathrm{m}$): this metric measures the minimum distance observed between individuals in the dataset. It provides insights into the proximity of human agents during their interactions, offering valuable data for studies related to personal space (proxemics) and social dynamics.

• •

  Number of 8-second Tracklets: this metric counts the non-overlapping tracklets of 8-second duration after downsampling to $0.4\text{\,}\mathrm{s}$ and applying a moving average filter. These choices align with current trajectory prediction benchmarks such as those outlined in Kothari et al. (2022). These tracklets offer discrete temporal segments for analysis, ensuring compatibility with existing evaluation standards in the field of trajectory prediction.

• •

  Motion Speed ($\text{\,}\mathrm{m}$/s): motion speed represents the velocity of all human agents. A higher standard deviation in motion speed indicates a diverse range of behaviors within the dataset. This diversity is essential for capturing various movement patterns and is crucial for robustness in trajectory prediction models. This metric is computed in the 8-second tracklets.

• •

  Path Efficiency: path efficiency quantifies the linearity of trajectories in the dataset, ranging between 0 and 1 (Amirian et al., 2021). It is calculated by dividing the distance between the first and last points by the cumulative distance traveled. A lower coefficient suggests more complex and non-linear trajectories, providing valuable insights into intricate human movement patterns. This metric is computed in the 8-second tracklets.

We compare our dataset with the THÖR dataset and the ETH/UCY trajectory prediction benchmark. The THÖR dataset encompasses three distinct scenarios, each featuring participants performing different tasks such as individual and group movement, as well as box transportation, different amounts of obstacles, and a mobile robot in the environment. In THÖR Scenario 1 (THÖR-S1), participants navigate the environment with one static obstacle. THÖR Scenario 2 (THÖR-S2) introduces a mobile robot navigating around the static obstacle, while participants continue their tasks. Finally, in THÖR Scenario 3 (THÖR-S3), the mobile robot becomes a static obstacle and an additional obstacle is added to the scene. The ETH/UCY trajectory prediction benchmark consists of five scenes: ETH, HOTEL, UNIV, ZARA1, and ZARA2. These scenes represent five outdoor public spaces that capture natural human motion patterns, resulting in a benchmark widely used by the human trajectory prediction community (Salzmann et al., 2020; Dendorfer et al., 2021; Yue et al., 2022; de Almeida and Mozos, 2023).

Firstly, we show the tracking durations in Figure 14. THÖR presents consistent average tracking durations around 15.5 to 17.6 seconds across the three scenarios. In contrast, THÖR-MAGNI shows wider variations, for instance, Scenario 4 features longer tracking durations (averaging 41.3 seconds), whereas Scenario 2 has the shortest durations (averaging 17.1 seconds). This variability can be attributed to the density of participants; that Scenarios 4 and 5, involved fewer human agents in a smaller space, may contribute to higher quality tracking. Nevertheless, THÖR-MAGNI has comparable or higher tracking time than THÖR. Furthermore, compared to the ETH/UCY benchmark (i.e., ETH, HOTEL, UNIV, ZARA1, and ZARA2 scenes), THÖR-MAGNI offers comparable or significantly longer tracking durations. This makes our dataset more useful than its predecessors for tasks such as long-term human motion prediction and human-robot interactions.

Secondly, we compare the minimal distance between people in Figure 15. Again, human density plays an important role: THÖR-MAGNI Scenarios 1–3 show low values comparable to those in ZARA1/ZARA2, while Scenario 4 and 5 reach values similar to THÖR, ETH, and HOTEL. The higher participant density in THÖR-MAGNI Scenarios 1–3 results in reduced spatial navigational freedom, leading to increased interactions and decreased social distances between individuals.

Thirdly, the motion speed statistics are shown in Figure 16. Despite the higher participant density in Scenarios 1–3 of THÖR-MAGNI, these datasets feature faster human agent navigation than THÖR and akin to those in ETH, HOTEL, and ZARA1 scenes, possibly influenced by the task of object transportation, impacting their velocity profiles. Participants in Scenarios 4 and 5 of THÖR-MAGNI have an average velocity similar to those in THÖR, UNIV and ZARA2. Also, generally, THÖR-MAGNI shows comparable standard deviations in motion speeds, indicating diverse and varied movement patterns among human agents. The similarity of the velocity profiles to previous datasets suggests that our dataset is also natural and diverse.

Finally, we show the comparison of path efficiency and number of tracklets in Figure 17. In terms of trajectory linearity, Scenarios 1–3 are aligned with the THÖR and HOTEL datasets, while the other datasets from the ETH/UCY benchmark contain more linear and less complex trajectories. It is also worth noting that THÖR-MAGNI Scenario 4 and 5 display the lowest average metrics ($0.78$ and $0.75$, respectively). The presence of a moving robot might influence these scenarios, prompting human agents to navigate cautiously and align their motion with the robot’s motion profile. Furthermore, THÖR-MAGNI presents a much higher number of non-overlapping tracklets than the other datasets.

In summary, these distinctive features make our dataset uniquely challenging, diverse, and valuable as a benchmark for evaluating human trajectory prediction methods. The heightened complexity and diverse range of trajectories in THÖR-MAGNI can provide a robust platform for evaluating the effectiveness of trajectory prediction methods, thereby increasing the breadth and depth of research in this area.