Enabling Social Robots to Perceive and Join Socially Interacting Groups Using F-formation: A Comprehensive Overview

Human group [146] and activity detection [2] has been a hot topic for computer/machine vision, Artificial Intelligence (AI), and robotics research. When humans interact with each other in a group (two or more people), they use implicit social intuition to position themselves concerning each other which facilitates easy interaction in the situation. The same thing is also applied when a new person wants to join a group. That person also assesses (implicitly) which place is best for joining so that people in the group do not face any inconvenience. The person also considers her role in that group according to the organization they are in right now to position herself [46]. Nowadays, robots are widely used in our daily surroundings [164] for many purposes. One such popular application is to use a robot to attend meetings/conferences/discussions remotely as a telepresence medium [21, 30, 121, 139, 140, 142]. In such scenarios, the robot has to join a group of people. For the robot to fluidly participate in groups, they must need to know how the groups are formed, how they are shaped, and how they have evolved [84, 85, 146]. There can be many kinds of groups that defer in dimension, situation, organization, and so on, and they are generally referred to as “f-formation.”

Facing formation (f-formation) is defined as the set of patterns that are formed during social interactions of two or more people. A robot can join an existing group or go to a single person and form a new group [15, 16, 54, 63, 114, 116, 136, 138, 182]. There are three social spaces related to f-formation, which are O-space, P-space, and R-space. O-space is known as the joint transaction space which is the interaction space between participants. P-space is the space where active participants are standing. R-space is the area that surrounds the participants and is outside the interaction radius as shown in Figure 1 (details in Section 2). According to social science, Kendon (1990) proposed four formations as standard formations. They are vis-a-vis/face-to-face, side-by-side, L-shaped, and circular. Apart from these, there are also many other kinds of formations such as semi-circular, rectangle, triangular, v-shaped, and spooning [112, 161]. By categorizing a formation to any of these formation types, a robot can understand how people are standing in the discussion and decide a position for itself to join the group accordingly. In joining, the robot should take care of the fact that already standing people are not disturbed and obstructed by itself. In detecting groups, several methods exist such as determining the position and orientation of people, graph-cuts methods, and Hough-voting system. One major problem in f-formation detection is occlusion [47]. People in a group may stand in such positions that some of them may occlude the others, or the viewing camera angle is placed in such a location that the complete group is not visible. In such cases, a robot will not know the formation as it may not be able to detect the bodies of some people. So, in this kind of situation, we have to think about what kind of formation we will assume for the robot to continue its work. For the sake of this article, we use group, interaction, and f-formation synonymously as we consider a group of interacting people who follow a particular formation to be detected for various applications as the primary focus area.

Motivation and Research Objective. Social group and interaction detection is a non-trivial task in computer vision and is of importance to the social robotics research community. Many research groups across the globe have concentrated their studies in this area. The idea of social groups and interactions was first proposed in 1990 by Kendon [84]. However, the first mention of human proxemics dates back to 1963 [63]. These concepts of f-formation and proxemic behaviors of humans have formed the fundamentals to the study and detection of physically situated interaction and group of people or robots or both. The research has gained pace since 2010 (see Table 3), delivering many rule-based and learning-based methods and techniques for detecting interaction and f-formation in social setups for various applications, mostly robotics and vision. But not many surveys, reviews, or tutorials have been found in this domain which provide a good overall impression of the research, the state of the art, and future opportunities. This survey article is a 360\({}^{\circ}\) view into the problem of social group and interaction detection using f-formation covering almost all the concerns and aspects comprehensively. The aim is to discuss the domain in a comprehensive manner and facilitate the scientists, researchers, and computer engineers to get a fair idea of the area and conduct fruitful research in the future. We also put forward a few related surveys in Table 1 and have compared them with our work based on various concern areas of our survey framework (discussed in Section 4.1 and Figure 8).

Uniqueness of the Survey. To the best of our knowledge, this survey is the first of its kind in this subject area. Our survey puts forward the idea of social groups with the perspective of f-formation with some comprehensive details. We also discuss the natural joining position for a robot to enable human–robot interaction (HRI), after successful detection of the formation using computer vision techniques. Additionally, we propose a holistic framework to signify the various concern areas in the detection and prediction of social groups. Various taxonomies regarding camera view of the environment for collecting scenes for detection, datasets for training machine/deep learning (ML/DL) models, detection capability, and scale and evaluation methods are discussed. We discuss and categorize all the detection methods, particularly rule-based and ML-based. Furthermore, we also deliberate the application areas of such detection and recognition giving primary focus to robotics. We also attempt to include some discussion on ethical concerns and considerations [73] in data collection and distribution as well as application. We also detailed the challenges, limitations, and future research directions in this area.

Organization of the Survey. This survey article is organized into the following sections. Section 2 gives a comprehensive perspective about social spaces involved in group interaction. The questions like the meaning of f-formation, types of f-formation, and evolution of f-formation from one type to another when a new member joins a group have been answered along with pictorial depiction for readers’ understanding. Then, we present a year-wise compilation of research performed in this domain with analysis in Section 3. In Section 4.1, we propose a generic and holistic framework for group and interaction detection using formations which also becomes the basis of categorization of the literature in the survey based on the various concern areas and modules. Section 4.2 shows the article discovery and selection methodology along with inclusion/exclusion criteria in Section 4.3. Section 5 discusses the various input methods for detection such as cameras and other sensors. It also puts focus on the various camera views and positions. Section 6 summarizes the methods, techniques, and algorithms for detection focusing on both rule-based static AI methods and learning-based (data-driven) methods. In Section 7, we discuss briefly the various datasets available for training and testing purposes. Then, we talk about detection capabilities and scale in Section 8. Section 9 presents the various evaluation strategy and methodology from the perspective of algorithmic computational complexity and application areas like robotics and vision. The various application areas are stated in Section 10. Finally, we discuss the limitations and challenges in the existing state-of-the-art literature and methods as well as propose some future research directions and prospects in each of the modules (of the survey framework) in Section 11. We conclude the survey in Section 12.

In this section, we describe the theory of f-formation, leveraging the theory to study the groups of interacting people, and how a robot can utilize this to perceive and display social awareness during interaction with a group of people. Apart from that we also identify a set of formations which does not fall into the exact definition of f-formation but still are prevalent in social gatherings and environments [112].

In 1990, Kendon [84] proposed the f-formation theory for group interaction by participating people on the basis of proxemics behavior. Human proxemics is the study of how space is used between humans or how it is set between humans and others during interactions or performing an activity. A computer system to detect human proxemics behavior was first studied by Hall almost six decades ago [63, 64]. This section is a survey on the literature collected on f-formation, using static and learning-based AI approaches. Figure 7 shows the various specified distance ranges for different designated interaction types on the basis of intimacy level between the participating people. The distance ranges specified in green-colored boxes are relevant to group/interaction and f-formation detection perspective. The blue-colored boxes signify distance ranges that are not generally seen in any f-formation. After the early studies by Hall, it was until 2009 when computer vision-based approaches using static AI methods or ML/DL techniques have emerged. From 2010, the research in this field started gaining pace. From 2010 to 2013, rule-based fixed AI methods were prevalent. Learning-based or data-driven methods like DL and RL have gained prominence in 2014 and gaining pace with each passing year. During the period ranging from 2013 to 2017, the research in this domain reached its peak with multiple methods, algorithms, and techniques being proposed. These methods have attained popularity in 2019. The years from 2014 to 2018 have seen equal contributions in both fixed rule-based methods and learning-based methods. So, there is a transition from traditional AI-based methods to ML and data-driven techniques like almost any domain of AI. Table 3 can be referred to for the complete list of references (year-wise). The table also consists of the keywords (methods, focus areas, and technologies) for most of the references for a better perception of the readers.

This study systematically summarizes and analyzes state-of-the-art methods and challenges for group and interaction detection using f-formations. The methodology is based on a generic framework for analysis of the relevant literature and a synthesis of research findings in a systematic manner.

This section summarizes the input methods in the group/interaction detection framework (Figure 8). The main input methods are cameras and sensors. There are different types of cameras used in the literature such as omnidirectional camera, helmet camera, robot camera, fisheye camera, and webcams. The camera sensor may be equipped with depth perception or provide only red green blue (RGB) images. The other main sensors found in the surveyed literature are audio sensors, blind sensors, radio-frequency identification (RFID) sensors, and so on. These are chosen based on the application areas and the working environment.

There are many f-formation detection methods proposed in the literature. In this article, we broadly categorize these methods into two classes—(a) rule-based methods (fixed rules, assumptions, and geometric reasoning) like the conventional image processing and vision techniques and (b) learning-based method (or data driven approach), with learning-based methods coming to prominence in the recent past. Multimedia and visual analytics [120] from big data remain the lucrative tool for large-scale f-formation detection and group interaction analysis.

In group discussions, people stand in a position where the conversation can happen effectively. Kendon [84] proposed a formal structure of group proxemics among the interacting people in a formation (described in Section 2). [122] provides a dataset of top-view images for capturing the same and subsequently methods for analyzing the dataset and detecting interaction and conversational groups are proposed. In [145], the authors use the Hough-voting approach with a two-step algorithm—(1) fixed cardinality group detection and (2) groups merging. Using these two steps, they detect the type of f-formation. In [53], the experiment uses a heat map-based method for recognizing human activity and the best view camera selection method. In [122], the graph-cut f-formation (GCFF) is used for detecting f-formations in static images with the graph-cut algorithms via clustering graphs. Yasuharu Den [46] says that formations are also dependent on the social organization and environment. He explains formation with outsiders where people stand based on their position. In this article [117], there are three constraint-based formations namely triangle, rectangle, and semi-circular formation. They use a game-theoretic model for the position and orientational information of people to detect groups in the scene. For checking the formation, they use an algorithm proposed by Vascon et al. [166, 167] that generates the 2D frustum of the position and orientation of people in the group. In [115], the authors use the Haar cascade face detector algorithm to detect the faces and eyes of people. Based on the face and eye detection, the method decides how many frontal, right, and/or left faces are there and then decides the formations. In [88], the Haar cascade classifier is used with quadrant methodology. The paper differentiates the person's facing direction by looking where the eye is located and in which quadrant. In [53], the authors use a new method to find the dominant sets (DS) and then compare with modularity cut. But this method is applicable only when everyone is standing. Hedayati et al. in their work [67] state that a robot needs a finer characterization of f-formations and introduce tightness and symmetry as two such characterizations. In [127], Raman et al. describe that such a finer characterization might be the occurrence of multiple conversations within the f-formation making the argument of detecting “conversation floors.” This method uses speaking turns for indicating the existence of distinct conversation floors and gets the estimation of the presence of voice. But this method cannot detect the silent (inactive) participants. In [134], proximity and acceleration data are used and pairwise representations are used with the long short-term memory (LSTM) network. They are used for identifying the presence of interaction and the roles of participants in the interaction. However, using a fixed threshold for identifying speakers can create mislabel in some instances. In [10], structural support vector machine (SVM) is used for learning how to treat the distance and pose information, and correlation clustering algorithm is used to predict group compositions. Furthermore, tracking learning detection (TLD) tracker is used for blur detection for ego-vision images. But the trackers cannot perform detection when the target is moving out of the camera field of view. In [131], the method uses ego-centric pedestrian detection. The pedestrian detector generates bounding boxes (BB). It uses optical flow for estimating motion between consecutive image frames. For detecting groups, they used joint pedestrian proximity and motion estimation. In [186], the method detects the group with a group detector first then uses the trained classifier to differentiate the people involved in the group. Some researchers use pedestrians, vision-based algorithms, and pose-estimation algorithm to detect [165] groups. In the article [165], authors use BP for handling f-formation detection and finding the joint estimation of f-formation and target's head and body orientation. They also use multiple occlusion-adaptive classifiers. There are many more methods scientists use, but each of them has its own strengths and weaknesses. Figure 11 presents a taxonomy of different methods/techniques/approaches surveyed in this article that are used for group/interaction/formation detection. Moreover, some works also attend to the navigational and joining aspects of a already detected group/interaction for application in robotics and HRI such as [105, 132, 164, 171].

Table 6 comprehensively lists all the surveyed datasets in the literature. A total of 71 datasets have been mentioned out of which only 15 are publicly available, 51 are private to the authors/researchers and 5 are not known. Fifteen of the datasets have outdoor scenes (mostly from public area captures), 42 have indoor scenes, and only 5 of them have both types of scenes. Thirty-six out of the list have multiple group scenarios in the scenes and 20 have single group scenes and 15 are not known. Ego-vision scenes of groups and interactions are seen in 20 datasets, whereas 39 datasets have exo-vision or global view images of groups, 1 dataset has both of them, and camera-view is not known for 11 datasets. Figure 14 gives a comprehensive idea about the taxonomy of datasets (training/testing) generally used in group/interaction and formation detection tasks.

One of the most important considerations in handling vision-based datasets involving people and their personal information such as interaction/conversation/discussion with other people in any indoor/outdoor scenes s the ethical issues/concerns and implications [73]. Here, it is evident that privacy and protection of visual data of people is at stake. There are multiple facets of this problem. Data collection itself is a huge challenge. To create a good dataset for learning models to detect human groups and f-formations, we need visuals from a diverse situations. So, as researchers, the very basic question we need to answer is the permission to collect, utilize, and sharing/distributing such data for research purpose.

Collecting such vision data of humans itself requires consent from the involving people in the scene. There are legal procedures in some jurisdictions for this. However, where no such legal dependency exists, it is always a good practice to do so. Similar consent also needs to be attained while distributing or sharing such data with other users and researchers for further exploration and learning model designing. Here, the consent of the people visible in the dataset along with the consent of the original creator of the dataset is required. This may be the reason behind less publicly available datasets in this domain. Another facet is the annotations of these datasets. Generally researches incorporate weak annotations in such human-based vision datasets. There is no personal information involved in such annotations. The annotations are also anonymous in nature. This can be a good way to comply with the ethical regulations.

Another facet is the in-the-wild versus in-the-lab settings for f-formation and human group/interaction detection. In case of lab settings, it is often conclusive that we already have the consent of the participating people in the visual data collection for usage and sharing. So, ego vision camera captures are also easy in such cases. However, in wild settings, often data are collected without the participating people knowing that they have been captured visually. Here, serious ethical concerns arise, which needs to validated and tracked. This also leads to less datasets in the wild with ego-centric cameras. In case of exo-centric cameras, privacy preservation is easy. It is very hard to preserve privacy in case of ego vision in the wild setups.

Next is the question of potential bias and discrimination⁷ in the dataset, i.e., non-inclusion of data from underrepresented communities. The major ethical concern in today's world can be seen in the diversity and inclusion facet considering factors like age, ethnicity, and gender. We can see that most of the datasets are created from scenes of western countries. There should be a mix of data from diverse communities across the globe which should include Asian and African countries as well. Another important consideration is the inclusion of both black and white skinned people in the vision data. Due to such under representation the learned models might predict wrong classes. Age consideration is also necessary in these datasets. Although, human groups/interactions in social setups may not consist of humans in age groups like infants and children, but there is a strong possibility of mixtures of teenagers and adults. Finally, equal inclusion of both male and female candidates in such data is also a concern to ponder upon. The datasets, as we see today, have much higher share of male people in them. Fairness in such cases is important.

There are organizations which are in place for such concerns in different geographies, such as the US or Europe. To name, Institutional Review Board⁸ or Research Ethics Board is a council (specifically in the US) that implements ethics in research activities by reviewing the entire research pipeline. They approve or reject any research and collecting of data which involves humans which is not ethical. Monitoring and reviewing of such research progress are also being carried out by such a committee. Another one is General Data Protection Regulation⁹ in the European Union (EU) jurisdiction. This is a part of the human rights law for EU in data privacy and protection. According to this, it is unethical and illegal to transfer personal data out of EU unless permitted specially. The board monitors these concerns strictly allowing individuals to have full control and rights over their personal data. This is another reason that vision data involving humans is rarely distributed or shared in public domains and platforms.

This section puts forward a categorization of the surveyed literature on the basis of group/interaction detection capabilities and scale for a method. After surveying the literature and the methods, it is evident that detection of groups in scenes is a non-trivial task and many factors are to be considered in the process as well. In real-life scenes, there can be both static groups of people interacting without much movement and there can also be groups with constant movement. There can be cases like group members leaving a group or new members joining a group. Group dynamics also is an important factor to be considered. People may sway or move their bodies occasionally too. Apart from these, methods also need to consider a single group or multiple existing groups in a scene. Outliers to one group can be a part of another group or can be noise at the global level. Figure 15 depicts a taxonomy of group detection in interaction scenarios in real-life cases.

The most important part of the formation or interaction detection framework (Figure 8) is the evaluation methodologies. The conventional methods to compare methods and techniques in such vision tasks are accuracy and efficiency. The accuracy defines how accurately a method detects/predicts or recognizes an f-formation. The efficiency parameter relates to the real timeliness aspect of the method. Apart from these, the papers in the surveyed literature also speak about simulation-based evaluation and human experience study-based evaluations (for robotic applications specifically). Figure 18 shows the simple taxonomy of evaluation methods for various group/interaction and formation detection methods or algorithms.

Simulation-Based Evaluation. This type of evaluation is conducted using simulation tools. The simulators have different features and have the ability to simulate the real world in complex environments. A range of simulators are used in the surveyed literature—Gazebo [117], RoboDK [133], and Webot [45]. Nowadays researchers are also focused on using Virtual Reality or Augmented Reality technologies for evaluation purposes. The evaluations are performed mainly to access the perception of a virtual robot or an autonomous agent. The question to be answered is how well a simulated robot (in a simulated environment) can perceive a group of simulated people involved in an interaction. Second, after detection, is the simulated robot joining the group naturally without discomforting the simulated people? (see Section 2.3 and Figure 3). Parameters like stopping distance for the robot, orientation, and pose based on the perceived group pose/orientation and angle of approach depending on the group's angle and position should be considered. Extensive discussion on these factors post-group/interaction detection by a robot or autonomous agent is out of the scope of this survey.

Human Experience Study Based Evaluation. This type of evaluation is based on testing the detection methods using ego-vision robots or on a real scenario with human participants as evaluators. A questionnaire is provided to the human participators to rate the quality of the method being used by the robot in real scenarios. The questions and parameters similar to simulation-based evaluation can be considered in this case as well but with a real robot perceiving human groups (who are also the evaluators) and interaction. In real-life scenarios, the groups are not static and tend to move when a member joins and leaves the group. Accordingly, the robot or the autonomous agent must detect the changes in group formation, orientation, and pose to re-adjust itself in a natural and more human-like manner without causing any discomfort to other humans.

Accuracy/Efficiency Evaluation without Using Robot or Simulators. This kind of evaluation is based on the accuracy or efficiency aspects of the methods but not tested in the real environment or by using robots. Here, the focus is mainly to evaluate the computational aspects of the methods/algorithms without evaluating the usability in real-life applications like robotics. However, applications like human behavior/interaction analysis, scene monitoring, and surveillance depend entirely on such evaluation. Table 9 classifies the surveyed papers on the basis of the evaluation strategy adopted. It also shows the descriptions/names of simulators in the simulation-based category.

Group or interaction detection has seen vast applications in many areas of computer vision. Specifically speaking, with the emergence of robotics and AI, this domain has realized its true potential. In this article, we categorize the application landscape into two broad areas: robotic applications and other vision applications. Furthermore, these have been broken down into five groups as summarized in Table 10 (also see Figure 19). The robot vision implies the applications where the robot's camera is placed in an ego-centric view for finding the groups only, but there is no purpose of initiation of interaction with a human. In HRI, f-formation detection is used to detect the group in order to participate in the interaction with fellow human beings autonomously. In telepresence, a remote person uses the robot to interact with a group of people. In such a scenario, the semi-autonomous robot can detect the group and join them while the remote human operator can control the robot to adjust its positioning.

Scene monitoring is useful for analyzing indoor or outdoor scenes with people interacting and forming groups and f-formations for various activities. On the other hand, human behavior and interaction analysis refer to the behavior between humans and how they are interacting based on the situation. Furthermore, visual analytics in big data has empowered the domain beyond imagination. People are trying to use these technologies in various aspects of life. In the current scenario of the COVID-19 pandemic, we can utilize this technology in monitoring social distancing in human groups and interactions as well. As already mentioned, telepresence robotics can be utilized by doctors/nurses and other medical staff to attend to patients in remote locations without physically being present.

Some of the discussed application areas have serious ethical implications and potential negative societal impacts. HRI can be seen in many embodied robots and humanoids nowadays. Specially in cases like assistive and therapeutic robots, there are many ethical implications [130]. First is the data privacy and protection of the person who interacts with such robots. The robots ought to capture visual data of the humans with whom it interacts. The proper protection of such data is mandatory. These data are easily available to the owner or user of such robots. Any kind of misuse of such data raises ethical concerns. Second, the emotional and psychological attachments of a human with such a robot is at stake when the job of the robot is done and it is taken away (specially in case of assistive and therapeutic robots). This may lead to serious harmful effects on the humans and leave them in much worse conditions than before. These kind of situations need proper and planned strategies to be handled without adverse effects. Next, the robots may be involved physically with the humans, such as lifting patients, taking them to bathroom, and helping them bath. So, the robots must be designed in a way such that the privacy of the humans are preserved by deactivation of video monitors during intimate procedures. Similar ethical concerns prevail in case of HRI in telepresence and teleoperation robots [49]. Here, transparency and accountability also come into the play.

Another application area stated in our survey has direct ethical implications on human privacy. Nowadays remote surveillance, monitoring, and tracking of humans and their behaviours and activities have gained popularity. Usually drones, UAVs, UGVs, and even fixed camera systems are used for such application in many public places as well as private areas. This poses a serious breach in privacy of human life and activities. Many countries have their own set of laws governing this arena¹⁰. Some of the locations where a person expects privacy are hotel rooms, own residence, public restrooms, changing rooms, and so on. Having surveillance or monitoring systems in such locations may lead to the violation of the prescribed ethical law, and the concerned people may face legal consequences. So, it is recommended to review the set of laws for the country where such vision based applications are put to effect. Finally, most of the researchers release their methods and codes on human group interaction or activity detection for research purposes only. So, such limitations in real-life usage of these methods need serious consideration and workaround.

The survey is treated based on a generic framework of concern areas about group/interaction detection using the theory of f-formation (see Section 4.1 and Figure 8). It addresses various identified modules and concern areas such as camera view and availability of other sensor data, datasets, feature selection, methods/techniques, detection capabilities/scale, evaluation methodologies, and application areas.

With the emergence of computer vision, robotics, and multimedia analytics, the world is changing for good with the progress of AI. Computation systems and autonomous agents are expected to show more human-like behavior and capability. One of the most important problems in this domain persists to be group/interaction detection and prediction using f-formation. Although some research has been conducted in the last decade, much more progress is still envisioned. This survey aims at generalizing the problem of group/interaction detection via a framework, which is referred to as the theme of this survey as well. This article presents a comprehensive glance at all the concern areas of this defined framework. This includes definitions of various f-formations, input camera views and sensors, datasets, feature selection, algorithms, detection capability and scale, quality of detection, evaluation methodologies, and application areas. The article also discusses the limitations, challenges, and future scope of research in this domain. The researchers can try and solve some of the unattended problems in this domain with the help of more recent and efficient approaches in DL, reinforcement learning, adversarial, and meta-learning paradigms. Another direction can be the utilization of advanced models such as Diffusion and NeRF which holds the potential to work very effectively in dynamic and crowded scenes. A combination of different neural networks can also be explored to create efficient models in application-specific cases.

We are thankful to the Editors and Associate Editors who were involved in the review process of our manuscript. Their guidance in revision iterations has significantly enhanced the quality of the work. We also are deeply grateful to the reviewers of this manuscript whose constructive criticism has improved the technical quality and readability of this manuscript.