Sharing Ambient Objects Using Real-time Point Cloud Streaming in Web-based XR Remote Collaboration

With advances in hardware and vision recognition technology, virtual reality (VR) and augmented reality (AR) have permeated many aspects of our lives. Prospective buyers can tour houses using VR without physically visiting the house [TopVIEW 2021]. Students can effectively learn about the elements of nature through AR and solve quizzes augmented on a see-through screen instead of reading a textbook [Alakärppä et al. 2017]. VR and AR are collectively referred to as extended reality (XR), and XR systems can eventually allow users to choose and alternate between VR and AR according to their preferences. Moreover, XR technology has gained significant attention in the context of the web community, considering that an XR specification for the Web, called WebXR, is being established in the World Wide Web Consortium.

XR technology is also active in remote collaboration because it allows intuitive information delivery by displaying virtual objects instead of delivering information abstractly through speech or writing. Early XR collaboration studies focused only on conveying the information necessary for collaboration. Using their methods, collaboration is conducted by marking parts or attaching annotations of work information on a screen where the local worker needs to concentrate [Gauglitz et al. 2014; Nuernberger et al. 2016]. Recent research has focused on quickly and effectively communicating workers’ intentions by sharing various communication cues, such as their gazes, hand gestures, and heart rates [Bai et al. 2020; Dey et al. 2017; Piumsomboon et al. 2017].

Meanwhile, research on improving the spatial awareness of remote workers is being actively conducted [Gao et al. 2020, 2017; Irlitti et al. 2019]. Improved spatial awareness allows a remote worker to easily identify their location relative to another worker or an object of interest (OOI). Two-dimensional (2D) images (including videos) or three-dimensional (3D) models are used as a medium for delivering spatial awareness cues. Using a 2D image has the advantage of allowing the remote worker to view a realistic representation of the local worker's workspace. However, there is no depth in the image, and the remote worker is unable to change the viewpoint of the local worker's camera at will. When using a 3D model, the remote worker explores a virtual space consisting of 3D models that replicate objects existing at the local site. The remote worker can freely navigate the virtual space, changing their viewpoint independently of the local worker camera. However, a reconstructed 3D model is less realistic than a 2D image. Thus, recent studies have conducted photo-realistic 3D rendering using several methods, such as neural nets [Aliev et al. 2020; Dai et al. 2020].

To generate 3D models of objects existing at a local site, devices with embedded optical depth sensor systems, such as Microsoft HoloLens and Microsoft Kinect, are widely used. Using such hardware, the local worker scans the local site before starting the collaboration, to obtain depth maps at various angles. The acquired depth maps are then combined to reconstruct the geometric information of the local site into a single point cloud or polygon mesh [Piumsomboon et al. 2017]. The reconstructed 3D model (point cloud or polygon mesh) is delivered to remote workers and used as a cue to help them gain spatial awareness of the local site in VR. In addition, a method has been studied to reconstruct the 3D model in real time by installing multiple sensors on the local site without scanning around the site [Lindlbauer and Wilson 2018]. Both pre- and real-time 3D model reconstructions in preparation for collaboration (e.g., scanning the local site and installing sensors) are time consuming. Therefore, it is difficult to initiate a collaboration promptly when no preparation has yet been made. Moreover, it is difficult to collaborate outside of the replicated space, even if a 3D model of the local site has been prepared.

We previously studied a system that captures and broadcasts the geometry of a local site as a point cloud in real time [Seo et al. 2017], a methodology for webizing various interaction devices [Seo et al. 2018], an XR collaboration system with distributed web technologies [Huh et al. 2019], a web-based XR content representation method and its rendering algorithm [Lee et al. 2020], and a unified coordinate system for VR and AR [Lee and Yoo 2021]. This study extends the aforementioned concepts. Our method is a hybrid approach combining the conventional pre-reconstruction approach for OOIs and the real-time reconstruction approach for ambient objects (AOs). The reconstructed 3D model for OOIs is a polygon mesh and is used to represent the state of the OOI (e.g., pose information). The reconstructed 3D model for AOs is a point cloud and is utilized as a background for VR, helping a remote worker acquire spatial awareness of the local site. Through this hybrid approach, the preparation time for a collaboration is effectively reduced because only the OOI, as opposed to the entire local site, requires pre-reconstruction. Furthermore, workers can reuse the content in another location by simply scanning again for spatial awareness of that location. This reuse enables flexible content design and efficient XR collaboration.

Contributions of this paper can be summarized:

1. An XR collaboration system that uses a hybrid approach in sharing a local worker's overall situation
   
2. A method for removing points representing an OOI from a reconstructed point cloud
   
3. A packet structure that can be used immediately upon receipt in point cloud data streaming with WebRTC
   

Our primary idea is to separate the objects that form the workspace of the local site into OOI and AO. To represent the OOI, a 3D modeled polygon mesh was used. A remote worker can intuitively teach a local worker about task instructions by demonstrating them with the polygon model. The XR collaboration process using the polygon model of the OOI follows the method presented in our previous work [Lee et al. 2020; Lee and Yoo 2021]. For the AO, the local worker generates a point cloud replicating only AO, and then delivers it to the remote worker via peer-to-peer networking. The remote worker receives the point cloud and uses it as an immersive VR background.

3.2 Sharing ambient objects

The XR collaboration method proposed by Lee and Yoo [2021] does not consider AOs at the local site. However, sharing information about AOs can make collaboration more efficient by allowing a remote worker to grasp the context of the local site quickly. Sharing AO information requires three steps. The first step involves the 3D reconstruction of the geometry of the AOs at the local site into the point cloud (Figure 2(a)). The second step involves masking the part corresponding to the OOI in the generated point cloud (Figure 2(b)). The final step is to deliver the masked point cloud to a remote worker (Figure 2(c)). This procedure is repeated for each frame of depth data gathered from light detection and ranging (LiDAR).

3.2.1 Point cloud reconstruction.

A LiDAR measures the distance to an object by measuring the return time of a laser pulse. If the surface of the hit object is very smooth (e.g., polished metal or mirror) and total reflection occurs, the laser pulse may not return [Gao et al. 2021]. The measured distance information is obtained in the form of a 2D depth map, using which we can estimate the geometry of the local site. In the depth map, the intensity of each pixel refers to the distance z_c from the LiDAR to the object (subscript c implies that its value is defined in the camera coordinate system). Using the pinhole camera model, a point (x_c, y_c, z_c) in the real world can be obtained from a pixel point (u, v) and distance z_c in the depth map [Szeliski 2010].

3.2.2 Masking OOI. The point cloud represents the geometry of the local site precisely. In general, a point cloud is sufficient to represent the AOs of the local site because they have static or limited movement. Conversely, because the OOI is frequently moved during a collaboration, capturing an OOI as a point cloud disturbs the virtual space that a remote worker is experiencing, making it difficult for the remote worker to understand the situation of the local site.

Figure 3 chronologically shows the result of moving an OOI while reconstructing the point cloud. In the figure, the tablet model and valve flange model (the cyan colored virtual object in the Figure 3(a1)) in front of the male avatar are synchronized with the AR device (held by the local worker) and the real valve flange (above the workbench), respectively (Figure 4(a) depicts the local site configuration corresponding to Figure 3). The OOI for collaboration is the valve flange, and the AO to be reconstructed is the workbench. Because the valve flange is located on the workbench, a point cloud for the valve flange is also generated when the local worker scans the workbench with the LiDAR which is embedded in the AR device. At this point, as the local worker moves the valve flange to the left, afterimages of the flange remain along the path of movement (Figure 3(a)). To prevent these afterimages, the reconstructed point cloud is masked to remove the points in the OOI's area. Consequently, the masked area in the point cloud is left as a blank, and the blank area is substituted by the flange model (Figure 3(b2)). In Figure 3(b3) and (b4), the local worker moves the flange to the left, but no additional afterimages occur as the points in the area of the flange are masked. After moving the flange, the hole once occupied by the flange model is filled with newly generated points.

Because an OOI and its 3D model are known through XR content, which points should be masked from a point cloud can be identified. When comparing the rendered results of the point cloud (Figure 4(b)) and the 3D model (Figure 4(c)), the pixel area representing the OOI (valve flange) is the same for each rendered result. It means that when the points of the point cloud are projected onto the screen, the point is a point to be masked if the pixel position of the projected point is within the area of the rendered 3D model. Whether a pixel is within the area of the rendered 3D model can be determined using the z-buffer of the 3D model renderer. If the z-value of a pixel is not 1 (typically in the OpenGL [The Khronos Group Inc. 2019], the range of a z-value is [0.0, 1.0] and the default depth clearing value is 1.0), it is within the area of the rendered the 3D model. When the z-value equals 1, it indicates that no 3D object occludes that pixel. Finally, the group of remaining points after being masked is defined as follow:

\begin{equation} Point = \lbrace p \vert Z_p = 1\rbrace \end{equation}
(1)

where Point denotes the group of remaining points after being masked, p denotes a point in a point cloud, Z denotes the z-buffer of a 3D model renderer, and Z_p denotes the z-value corresponding to p in the z-buffer.

We provided an example code of this section in appendix A.

3.2.3 Data transmission.

During the transmission step, the masked point cloud is delivered to other workers participating in the collaboration through a WebRTC data channel. The maximum buffer size of the data channel is 16 KiB (16 384 B)¹. However, this is generally a tiny size to hold all points of a point cloud. Thus, a point cloud must be split to fit the buffer size.

The position data consist of three 32-bit float values, and the color data consist of three 8-bit unsigned int values. Thus, the size of a single point is 15 B. Consequently, a fragment of a point cloud that fits in the data channel buffer can contain up to ⌊16384/15⌋ = 1092 points. With this fact, we can derive the number of fragments generated from a point cloud. For example, a depth map with a resolution of 192 × 256 = 49152 requires ⌈49152/1092⌉ = 46 fragments to be delivered through the data channel.

In general, point cloud data are managed separately as position data and color data for efficient memory management. When the point cloud data are divided according to the buffer size of the data channel in the order stored in the memory, the position data and color data for one point are separated into different packets. Rendering a single point requires both position and color data. In other words, packets containing position data and packets containing color data are dependent on each other. Packets can be delivered out of order as they pass through the network (Figure 5(a)). Therefore, a remote worker may receive other packets between two dependent packets. Because both dependent packets must be available, the point cannot be rendered until the receipt of the second packet.

Figure 5(b) presents a structure dividing the point cloud data such that there is no dependence between packets. The position and color data that comprise one point are contained in the same packet. Consequently, the relevant portion of the point cloud can be rendered as soon as each packet arrives.

We implemented a prototype to demonstrate the proposed XR collaboration concept. Figure 6 shows the architecture of the prototype system. The WXR Library is a JavaScript library that controls and renders XR content. The WXR Library was improved based on A-Frame [Marcos et al. 2020] from a version introduced in our previous study [Lee and Yoo 2021]. The XR content loader downloads static resources, such as XR scenes or textures, from the WXR workspace server. The event handler applies interaction events (represented as Event Data in Figure 6) to the XR scene and transports them from and to the server for synchronization. The point cloud receiver adds the point cloud delivered through WebRTC to the XR scene. Because the data size of a point cloud is generally big, controlling them loads the server. Thus, we opted to use WebRTC for streaming point cloud data. On the contrary, we chose WebSocket for sending Event Data because the data size of an Event Data is generally tiny, and the logic for handling some Event Data (e.g., joining in or exiting from a collaboration session) is executed in the server.

We created a browser application (WXR Browser) to use the object tracking function and LiDAR functionality in the tablet. The AR engine used in the prototype is ARKit, which provides pose information of the OOI obtained from the object tracking function and a depth map collected from the LiDAR embedded in the tablet. The pose information of the OOI was applied to the 3D model of the OOI in the XR scene via the event handler. The point cloud generator reconstructed a point cloud using depth data from the AR engine and masked the points in the corresponding area of the OOI. Other than that the WXR library rendered the XR scene on the user's screen, the point cloud generator also performed off-screen rendering only on the OOI to obtain its z-buffer. In the point cloud streamer, the masked point cloud was divided by the buffer size of the data channel and passed to other users connected by WebRTC. The process of point-cloud reconstruction could be switched on and off via a separate user interface.

Figure 1 presents the configuration of the collaboration. The local worker with the AR device reconstructs the point cloud of the AO around him and recognizes the ball valve (Figure 1(c1)), which is the OOI. The remote expert on the VR side can understand the situation of the local site by looking at the 3D model of the OOI and the point cloud of the AO, and deliver task instructions to the local worker. The collaboration process is observed by a third person. The AR device used by the local worker is iPad Pro 12.9-inch (4^th generation), and the VR device used by the remote expert is Oculus Quest 2. The Oculus Quest 2 was tethered via Firefox browser on a PC (AMD Ryzen 9 3900X 12-Core 3.8 GHz CPU, 64 GB DDR4 RAM, NVIDIA Quadro RTX 5000 GPU) with a Windows 10 operating system using Oculus Air Link². The third person used Chrome Browser on a laptop (Intel Core i9 8950HK 2.9 GHz CPU, 32 GB DDR4 RAM, Intel UHD Graphics 630 GPU) configured with a Windows 10 operating system. The server program was written using Node.js with MariaDB and run on a PC (Intel Xeon E5-2699 v4 2.2 GHz CPU, 64 GB DDR4 RAM) with a Windows 10 operating system.

In the reconstruction process of the point cloud of the AO, the points collected around the ball valve were masked, and the space of the masked points was occupied by the 3D model of the ball valve. The remote expert accumulated the point cloud delivered at each frame, as shown in Figure 1(b). The accumulated point cloud formed the background of the virtual space that the remote expert was experiencing. Once the reconstruction was completed, the point cloud surrounded the 3D model of the ball valve, allowing the remote expert in VR to grasp the situation of the local site.

The remote expert demonstrated a task in three steps: locking the ball valve, disassembling the flange, and pulling the ball out of the valve. At each step, the remote expert directly showed the local worker how to perform the task using the 3D model of the ball valve part (Figure 7(a) (center)). The 3D model was augmented on the local worker's AR device screen (Figure 7(a) (right)). The local worker could intuitively understand the remote expert's instruction by viewing the 3D model's movement, and perform the task by manipulating the handle based on his understanding of the task instructions (Figure 7(b) (center)). The AR device carried by the local worker recognized the movement of the handle using an object tracking function and synchronized the pose of the handle to the corresponding 3D model (Figure 7(b) (right)). When the local worker rotated the handle, the area (Figure 7(a) (1)), which was initially not reconstructed because of the masking, was exposed to the LiDAR and filled with a newly reconstructed point cloud (Figure 7(b) (2)). In the process of disassembling the flange, a spanner was required to disassemble a set of bolt nuts that combined the ball valve flanges. The remote expert was able to find the spanner in the background consisting of the reconstructed point cloud and inform the local worker of its location (Figure 8). A complete demonstration video of the prototype is available at the following link: https://bit.ly/3yyLR1Q.

In this paper, we introduced a new XR collaboration concept that separates OOIs and AOs and conveys the situation of a local site in an appropriate manner for each object. This collaboration method uses a LiDAR to reconstruct the point cloud on the fly only for the AOs. Through this, a local worker can manipulate OOIs while sharing their AOs. This method does not require installing a complex sensor system to communicate the situation at the local site. Moreover, this method makes it possible for XR collaboration to proceed in any space. Through the demonstration, we observed that it was possible for the remote expert to recognize AOs (e.g., the spanner) in the local site and induce the local worker to use it. In addition, the proposed method is practical because it does not create virtual replicas for all objects in the workspace, thus saving preparation time for collaboration.

The limitation of this research is that the proposed system is not fully qualified against quantitative user surveys due to the lasting COVID-19 pandemic. Thus, our future work involves conducting a user evaluation of the proposed method to quantitatively verify the practicality and efficiency of our XR collaboration concept against traditional collaboration methods, such as video conferencing.