ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings

Prior work has studied remote object-focused collaboration [22, 73, 77, 83], with a focus on understanding users’ perception of different perspectives on physical objects. Various research systems have been proposed to offer new ways of visually accessing remote spaces [26, 30, 68], such as remote camera control [83] or multiple viewpoint capture [59, 103], which use stationary or head-mounted cameras to expand the remote access to a physical space and its objects. Media space research has studied approaches to connect spaces across remote office sites, where there are challenges of spatial inconsistency [100], multiple views [82] (e.g., desk view, birds-eye view) and affordances to control viewpoints [103]. They found the particular importance of accessing objects and remote environments and supporting collaboration in various work-related contexts [82, 100]. In contrast, other studies focused on families and friends who are geographically dispersed [53, 76, 106]. where interfaces were developed to enable always-on media space systems [50, 51, 81] to enhance family connection and awareness, and specialized shared table space for activities such as reading (e.g., [81]) and playing (e.g., [79, 96, 105]). In spite of these studies, many of the systems developed either do not focus on accessing physical objects (e.g., [50, 51]) or require additional hardware (e.g., additional tabletop or projector [52, 79, 96, 105]). As a result, only a limited number of these systems can be used in everyday life, especially when people are working from home or connecting with remote friends and family, due to the specialized hardware required. In light of recent efforts to improve video-mediated communication [34, 41, 43, 75, 89] and the current trend of remote work [5, 8] and online social gatherings [25, 43], studying object-sharing practices might suggest new and alternative approaches to enhance remote meeting technologies.

In this paper, we consider details of how video-conferencing systems can be designed to accomplish everyday work-related or leisure-related activities with a single display and camera. This motivates us to design and investigate a system that tackles the challenges of sharing physical objects in the conventional face-to-face, opposing view perspective of video chat systems. We outline the following subsection to see how our work connects and extends prior work by supporting the perspective of space (2.1.2) and remote referencing (2.1.3) on the physical objects.

The conventional face-to-face view can be challenging due to the opposing view perspective of video chat systems [22, 73], which flattens the view of physical objects [22]. Furthermore, the need for dynamic manipulation and re-orientation of handheld objects towards the front camera in a live video can be tedious, as users are responsible for presenting the handheld object to their remote partner. In particular, users may forget to present the object to their remote partners when their attention is on the physical material experience with the object, as found by Oehlberg et al. [73] in a scenario of critiquing design prototypes. To solve these challenges, ReMa [22] enabled a remote re-orientation of objects by supporting tele-manipulation with robot arms. CamBlend [77] utilized the wide FOV camera with multiple controllable, in-context views that enabled remote reference and captured snapshots. Licoppe et al. [61] explored how users orient objects with the support of a maneuverable device, the Kubi telepresence system. In contrast to these hardware-intensive solutions, our work focuses on developing a hardware-free approach to address object-sharing challenges when the user holds and orients objects towards the camera, with both dynamic live sharing and static non-live storage of physical objects in the virtual space.

The central theme in Computer-Supported Cooperative Work (CSCW) systems has been to facilitate remote collaboration by conveying the conduct of remote collaborators via remote referencing and gestures. Various methods have been explored in previous studies to facilitate remote gestures, including the use of cursor [77], laser pointers or tele-pointers [42, 56], as well as virtual overlays of bodies and arms [90] to provide supplementary non-verbal cues and subtleties. Furthermore, AR-based mobile collaboration supports a different perspective of space of what we aim to investigate. To solve ambiguous verbal referencing problems, this work focuses on supporting annotations and labeling to videos and helps users to move their mobile camera around the scene freely for physical tasks [10, 27, 29, 48, 74].

Moreover, prior work also supports remote gestures by enabling re-orientation and tele-manipulation of a physical proxy of the object, e.g., utilizing robot arms [22] or shaping-changing actuation [22] to create physical affordance. However, such systems often require physical proxies of the same objects from both sides. Extending this work, we want to further investigate how enabling remote reference, access, and control to the parallel and shared digital copies of a physical object can help or hinder the remote user’s understanding without physical proxies and hardware setups.

Current commercial tools, such as Microsoft Teams, use contour detection to capture the shared task space and support desk or whiteboard view. Apple has recently announced an approach that uses a single iPhone ultrawide angle lens camera as the webcam to quickly transition from face-to-face to desk view. FocalSpace [102] utilized the "focus+context" approach and captured both persons and objects as interactive elements using depth and shared image space in layer models, but its shared task space still primarily focuses on capturing specific surfaces, such as a whiteboard. CamBlend [77] supports task space using a different approach, as it integrates the person and task space yet blurs the human and unrelated task spaces, with a movable focus-in-context window deblurring certain objects for object-focused discussion.

Commercial video-conferencing platforms can support the quick showing of physical objects that form part of the task space via the person space video channel [7]. Media space and video-mediated collaboration research such as ClearBoard [46] primarily focuses on integrating the person and digital task space, limiting its ability to share physical objects through its person space channel like video-conferencing tools. ThingShare builds up on this metaphor of integrating the person and (physical) task space and focuses specifically on supporting the design of remote meetings for sharing physical objects with person-centric () and task-centric sharing ().

We devised a design space (Figure 2) to situate our techniques in relation to previous object-focused sharing system, suggest connections between techniques, and direct attention to relatively under-explored combinations.

On the left side of Figure 2, the rows indicate the communication configuration of either (2.3.2) or (2.3.1) of person and shared task space. We identify the task space here as the surrounding or wider physical space and environment rather than the digital shared task space.

The columns of the Design Space encompass our design of remote reference space into three levels of access to remote object referencing. We define this as a qualitative measure of how a given interaction technique limits the remote user’s access to the physical objects of shared task space, i.e., the degree to which a user provides the input (i.e., remote reference) into the remote space, from no/limited referencing to object referencing. refers to tools that do not support remote gestures and thus require a remote user to use ambiguous verbal referencing. Supporting tele-presence [59, 61, 88, 93] enables , allowing a local user to be aware of where the remote user is looking at, and the remote user to access different viewpoints. However, only the direction of the space is noticed, and access to specific physical objects is limited. The use of ephemeral or persistent spatial-anchored remote gestures, such as arm shadows [90], cursor pointing, simple annotation [50, 51] and focus-in-context windows [77] can provide more remote control over to a specific object. By contrast, , which our work supports, allows the remote user to manipulate parallel and shared digital copies of objects. While many techniques for shared task space, including the work that supports entirely digital sharing or other display configurations such as VR/AR (e.g., [91]) and hardware-intensive work that requires physical proxies (e.g., [22, 23]), have been proposed, we have intentionally emphasized examples of work that support sharing physical environment and object-focused work with 2D screen or projected display, since those are the most relevant to the ideas in this paper.

Most respondents “Hold the object up to the camera” in work-related (50.3%) and leisure-related meetings (45.1%)¹ (See Figure 3 for other behaviors).

We analyzed the variety of use cases for sharing physical objects in video meetings. Overall, the purposes in our sample include sharing objects for collaborative, instructional, or ad-hoc needs that vary across work-related and leisure-related meetings. We first categorized use cases found in work-related meetings (72 respondents) as follows.

For respondents who never (12.4%) or rarely (29.8%) showed objects, respondents generally indicated three reasons for not needing or wanting to share physical objects: 1) their work-related objects are digital, 2) the objects felt too personal and not professional for work-related reasons, or 3) it is challenging with current commercial tools.

During leisure-related meetings (71 respondents), respondents often share personal items in their conversations. Most respondents reported showing handheld and background items during social settings when the purpose could be ad-hoc and related to the conversation.

Based on the results of the formative study, it was discovered that people often share physical objects via their video feeds in an ad-hoc manner, such as causally sharing their personal belongings or prototypes without focusing on the fine details of the object. However, users may still encounter difficulties in sharing objects due to challenges in presenting the object from various perspectives (C2), keeping it within the video frame (C4), and avoiding occlusion by the virtual background (C5). To address this disconnect and enable lightweight and expressive sharing of physical objects, we designed the Person View. Unlike traditional video containers, Person View serves as a temporal container for digital copies. Local users can create digital copies of physical objects in their Person View, which can then be manipulated and placed in the Person View, or moved to another user’s Person View, Object Library, or the shared Task View. This allows users to easily create digital copies of a physical object from various views, combine them with their own video, and separate digital copies from the background.

Physical objects are brought to meetings not only for ad-hoc demonstrations but are also produced, modified, and reviewed afterwards, such as product reviews, hands-on training, and design sessions. Our formative study identified several challenges in showing objects in detail, especially during work-related meetings. Existing video meeting systems require users to hold objects up to the camera to show details (C1), which can be difficult to keep steady and within the camera view (C4). Additionally, referencing a specific section of an object can be challenging (C3) and effective work-related discussions may require sharing multiple objects or perspectives (C2). Our study also found that sketches and paper artifacts were frequently shared and held up to the camera during discussions involving complex concepts, e.g., showing quick drawing, scribbles of data visualization, or quick calculations. Furthermore, direct group attention to specific parts of artifacts can help. All of these use cases require thorough and close-up sharing beyond the Person View. To solve this, we designed a Task View (Figure 4b), a magnified object view, that covers the majority of the interface.

Utilizing these two video windows

support a balance between the focus and context, and enables both and , resulting in two video layouts. Context Mode (4.2) consists of the person views (Figure 4A-1) so that it shows physical objects via its person space channel, helping the user to drag-off a digital copy from the person view for a quick, ad-hoc, and expressive demonstration. Focus Mode (4.3) enabled two main UI changes: person view (Figure 4B-1) will be shrunk to the left side and a shared task view (Figure 4B-2) appears at the staging area, allowing the user to display different data formats of physical objects in more detail and collaborative manner.

Moreover, we support Object Library (Figure 4A-2) for both layouts as a temporal space to store captured perspectives of physical objects. In sum, the person view and task view video windows and Object Library served as containers for Digital Copies of Physical Objects. Any user can capture the items from any video window and store them in the Object Library. They can also reuse the digital copies by bringing them back into any video window.

Right-clicking on the detected object triggers a Context Menu (Figure 4A-4) with six options: show/hide object, freeze, snapshots, short video, focus share, and delete.

To enhance and augment the immediate and ad-hoc demonstration needs of physical objects and the efficient hands-free and temporal manipulation of objects (G4), we enable the user to capture the static viewpoint of physical objects as a digital copy in the user video feeds (Figure 6).

Within the user’s video streams, the replicated 2D digital copy can be dynamically re-positioned and re-scaled (Figure 6a). Another option is to record a short video of the chosen item and save it so that the user may play it back and forth with different viewpoints of the object and narrate it to display a dynamic activity with the object. Moreover, handheld items (Figure 6b), hybrid items such as contents on the mobile phone screen (Figure 5A-2), or background items (e.g., Figure 6c-d) can be captured. To enable both local and remote reference on the captured object, all users can reuse the any copies by grabbing the stored parallel objects or frozen things into their own video feed to manipulate them in parallel (Figure 5B-2). In addition, the local user’s cursor on a remote user’s video feed will be tracked from the remote user’s side, so that references to the remote user’s physical objects in the virtual space are visible. As everyone can interact with the digital copies of a physical object, this provides distributed shared virtual items for everyone to view and engage with, as well as a quasi-shared view of two people looking at the same physical object as if they were side-by-side (G3).

Supporting the detailed view of snapshots, short videos, or live videos does not include viewing different perspectives of an item or two items at the same time (G2). Collage View supports all users to capture their own objects from the physical environment or bring digital copies stored in the Object Library into the freeform collage in a What-You-See-Is-What-I-See (WYSIWIS) canvas. Hence, switching to the Collage View (Figure 4B-5) by toggling it in the Sidebar enables both local and remote users to drop stored items into the Task View for collaboration, thus supporting the discussion over multiple perspectives or different items on the Task View with the moving, rotating and rescaling interactions.

Annotation (Figure 4B-3) on the static and key frames of a short video will be saved in the displayed thumbnails in real-time. Moreover, as the live video of objects is stabilized at the center of the Task View, any annotation in the system during live video sharing is object-anchored rather than spatially anchored. This indicates that when annotating on a physical object in a live video, the annotation will follow the physical objects rather than stay in mid-air or world-stabilized [28, 29, 48].

The web client sends video stream from the user’s webcam to a Flask back-end server using socketio as a feature input into the model. The model in the server then predicts the mask and class list of object detection before sending the output prediction back to the front-end. The default training set for the instance segmentation model Yolact [4] is the COCO dataset [1], which contains over 200,000 images and 80 item categories. Based on the best model performance given in [4], we utilize the pre-trained ResNet-101 [40] model with FPS 33.5 as the backbone.

Body segmentation was employed to segment persons out of the surrounding home environment in order to ensure privacy, yet instance segmentation approaches were not used for object-sharing practices in video meetings. Most errors are caused by mistakes in object detection such as misclassification, or box misalignment. There are also some issues caused by the mask generation algorithm [4] and the nature of classification tasks. Hence, prior to obtaining a suitable mask for the end-user experience in video meetings, there are several critical pre-processing steps: 1) Eliminating Leakage Noises: The Yolact approach crops the masks after assembly, and does not suppress the noises outside of the cropped region [4]. For example, Figure 17 in the Appendix exhibits several leakage issues. This indicates that whenever the predicted bounding box is imprecise or misaligned, the leakage noise may creep into the instance mask, generating some tiny artifacts or even merging the segmented mask of some other classes (e.g., human) with the predicted item. Hence, we first used a MediaPipe body segmentation model [2] to get a separate mask of the person to segment the whole virtual background. We then performed Morphological Operations for opening (i.e., erosion followed by dilation) and closing (reverse of opening, dilation followed by erosion) to filter out the holes in the scene on the output masks (with binary color) (See Figure 7: Pre-Processing) and applied Gaussian Blur to smooth the boundary. 2) Maintaining the Reference Space: As the general instance segmentation approach is aimed for classification tasks that output a unique label to every instance, it directly detects and segments all the items apart out of the scene, including the hands (reference space) that are pointing towards an object (task space). However, to apply this in the remote meeting context and preserve the remote gestures in Task View for detailed sharing, we perform Compositing Operation (Section 4.5.3 and Figure 8) in the front-end to composite the masks of objects and human body or human hands with the video stream. Therefore, we segmented the hand and then utilized Convex Hull (See Figure 7: Pre-Processing) to expand the convex polygon of the hand mask so that the hand could be composited with the item in the foreground.

The output mask for an item and the remote gesture (i.e., hand or human body) are key requirements for implementing instance segmentation in a video meeting environment. In the Context Mode, for the scenarios with a virtual background, we sequentially compose the person segmentation (from MediaPipe) with the output mask of handheld objects (Figure 8b). To temporarily hide the object, the composition only includes human segmentation and the video stream. The composition involves human segmentation, the object, and the video feed in order to reveal the item. When there is no virtual background, we composite the object mask with the video stream (Fig 8 a).

Moreover, In the Focus Mode that aims to show a close-up view of the object, in order to get an ideal mask where only the remote gesture (i.e., hand) is included within the segmentation boundary (see Figure 8c), we compose the hand with the original mask of the physical object.

The first task was designed to emulate the ad-hoc and informal object-sharing scenarios that we observed in our formative study, both in work-related and leisure-related meetings. This task was relatively simple, serving as a warm-up with a brief round of introductions. Participants were asked to choose two objects, one from handheld objects (Figure 9A) and another from background objects (Figure 9B) prepared in the room, and show and explain the objects to each other using ThingShare and baseline. The participants completed the task in the following two contexts: 1) Work context: the participants were asked to turn on the virtual background as if they were having a work meeting from home; 2) Social context: the participants were instructed to assume that they were friends on a casual video call and not use the virtual background.

This task mimics the remote assistance meeting where a local user with a device needs the help of a remote user to operate a device, which was seen in previous studies (e.g., [58]) and the formative study responses. In this task, the participants took the roles of a local user needing assistance and a remote expert who helps the local user. The local user was given either a VR controller or a game controller (Figure 9C) and the remote expert was given an instruction document of the controller that explained how to perform two specific tasks (each for ThingShare and baseline) using the controller. The objective of the task was for the local user to learn the procedure for using the controller from the remote expert, and the task was completed when the local user successfully restated the procedure. After completing the task, the participants switched roles and completed the same task with a different controller that uses different instructions. The complete controller instructions are available in Appendix A.2.

This task was designed to mimic a scenario where two people need to collaborate remotely on a task that involves showing and referencing objects in their environments. In this task, we told the participants to assume that they would be sharing an apartment as roommates. The participants were asked to decide on the room allocation and find furniture to decorate their rooms. The same sets of floor plans and furniture brochures were provided to both participants (Figure 9D). The brochures had page numbers and item numbers that the participants could use to refer to a specific item if they wanted. The two sets of floor plans and furniture brochures were prepared to differentiate tasks between ThingShare and baseline. The task was completed when they finished allocating rooms and finding the furniture.

After the completion of all tasks, we conducted a semi-structured interview to ask the participants how they used ThingShare to interact with each other, how ThingShare influenced the way they share objects, their positive and negative experiences with ThingShare, how they would use ThingShare in future applications, and other thoughts on the system. We also asked them to explain key sharing behaviors that we identified with the live observation notes.

We observed that participants frequently used digital copies of physical objects to blend with their person view videos for casual and playful conversations. For example, P9 froze an apple in the person view and pretended to eat it in the video, saying “See, I am eating this” (Figure 10b). Another participant (P10) blended digital copies of her psyduck toy and her partner’s apple to create an interesting scene, saying, “my duck just stole your apple.” P7 created multiple copies of her partner’s object to decorate and tile her virtual background. Participants also used their body projections to perform embodied interactions with the captured objects, as if they were interacting with a physical object (e.g., see Figure 10c). Moreover, similar to findings in [73], participants occasionally forgot to present the item to the remote audience when looking at their objects in the baseline. However, after creating digital copies with ThingShare, participants rarely showed objects live to their remote partners via the person view. Instead, some of them would freeze an object in the person view and then shift between looking at the physical item and pointing to the digital copy in their video window using their embodied projections (see Figure 10d). This demonstrated the efficiency of the blended effect in facilitating the transition from examining the physical properties of an object to physically referencing it. Overall, ThingShare provided an effective embodied affordance for more efficient sharing.

Participants (16/16) appreciated that they were able to interact with the digital copies of their partner’s objects despite being remote from the actual object. From the remote user’s perspective, we also observed them dragging off digital copies of physical objects from the local user’s person view to their own view to magnify and see details during a discussion. For instance, when a local user was promoting an apple variety in Task 1, the remote user dragged a stored apple into his own person view and zoomed in on it, saying “but this apple has a hole on it.” In another example, a user was describing the details on the Starbucks cup sleeve (Figure 10e), while the remote user viewed it in his own person view and asked questions about specific details. This enabled parallel exploration and provided a more efficient way to reference specific details of an object for instant questions.

According to the questionnaire results, participants were able to show objects without comprising their privacy with ThingShare (ThingShare: Median = 4.5, IQR = 1; Baseline: Median = 3, IQR = 1). Furthermore, participants (14/16) appreciated the ability to maintain privacy while selectively expanding the shared space to include physical objects, while all participants found it difficult to show objects with baseline with the virtual background (e.g.,“the object gets blurred if the object is placed outside the human body.”, P4). This was particularly the case for background objects that could not be held up to the camera. Some participants appreciated the controllability of the baseline condition, as they found that the physical object could be shown when the handheld object was within the boundary of the human body, and hidden when moved out of the human boundary (i.e., P1 noted “I can control whether I want the audience to see the object or not using the background.”). However, all participants found the amount of shared area enabled by the body area in the baseline condition to be limited (See Figure 10a).

Participants (13/16) preferred non-live copies (i.e., snapshots and short videos) to live videos as non-live copies allow them to free their hands from the object and efficiently discuss the objects (e.g., “I do not need to hold the item all the time, and also I can zoom in to let the picture become bigger, P12). In particular, some participants (4/16) expressed concern about the difficulty of digesting multiple frames of information and remembering details in live videos, e.g.,“I need to digest multiple details of live video over time and remember the details I saw versus what I am seeing at the moment" (P11). Three participants also commented on the need to capture copies for local users and revisit past activities in the case of live sharing only.

While non-live copies were useful in many situations, we also observed instances that showed the potential value of combining live and non-live digital copies for increased awareness and collaboration. In Task 2, we observed that some local users would occasionally show live videos of themselves holding the controller through their person view window in order to ask the remote user if they were holding or using it correctly while both users were viewing the non-live copy of the controller in the task view. Some participants suggested combining live and non-live copies to provide more detailed information in certain situations with a trade-off between live and non-live views, e.g., P13 commented -“I hope in Focus Mode, there could be a way to expand my video [person views] when I wish to signpost the non-live copies but also show it lively through my video.”

Participants have divided preferences over capturing the snapshots and short videos. Some participants (11/16) liked that short videos can mitigate the tediousness of capturing snapshots multiple times, especially when the captured snapshot is not optimal. They preferred the short videos when showing objects with multiple facets (P11, P13, P16), such as a cube, or capturing the motion of an object as tutorials (P9, P12) such as demonstrating switching light bulbs. In addition, some participants (4/16) did not favor short videos as the short videos can be unnecessarily long with transitioning scenes, e.g., P7 commented - “it is hard to find the best framing of the short video so that I need to coordinate the framing again and again and make a longer video than expected.”

For Task 2 (Asymmetric Task), participants generally used two object-sharing behaviors in the baseline condition: 1) the remote user(s) instructed the local user(s) to show different perspectives of the object, and the local user(s) rotated the object in their hand to interact with it before rotating it back towards the webcam for communication; 2) the local user(s) turned themselves towards the object to share the same perspective with the webcam as if they were looking at the object from the same side (See Figure 11a-b). With ThingShare, however, local users rarely turned themselves or moved objects back-and-forth for interaction, instead utilizing the snapshots and Task View (See Figure 11d). Participants appreciated the Collage View, which allowed for the display of objects from multiple perspectives at once (See Figure 11c), reducing the need for inefficient verbal communications requesting for perspective changes. Participants (15/16) also reported that ThingShare helped construct a detailed shared view with the remote user reference, allowing for live annotation on the focus-shared object and the ability to interact with the object based on remote instructions without the need for switching views (Figure 11d).

As both users had the same set of objects in Task 3 (Symmetric Task), we observed that participants using the baseline condition struggled to synchronize their views. When discussing furniture, most participants (13/16) were looking at their physical artifacts instead of holding up the object to create a shared digital view. This made it difficult to confirm whether they were looking at the same page (“In baseline, I didn’t know whether the remote user was looking at the same page with me”, P9) and to gauge the other user’s attention (“when I was showing it [to the camera] to confirm, my peer is still looking at his book [brochure]”, P12). Figure 12c-d shows that a user did not pay attention to the screen when his partner(s) were holding up to show something. As a result, participants preferred to communicate using verbal descriptions, such as page numbers and locations on the page. In contrast, participants using ThingShare were able to use the Collage View to place and discuss multiple snapshots of the objects. Some participants also used the snapshots as a "cursor" by minimizing the snapshots to indicate the position they wanted to place furniture. However, we observed that two participants used finger gestures (Figure 12f) to point to shared items on the screen, which was not usable with ThingShare.

All participants mentioned that they liked that the shared object automatically filled the task view window without showing unrelated objects and details in ThingShare. In baseline, the participants had to physically move an object close to the webcam to show the detailed view, which was also difficult to maintain as the object could ward off their view (see Figure 12d). However, some participants (7/16) still brought objects closer to the webcam for better resolution, which led to challenges with object tracking. It often led to losing track of an object as the object tracking model could not detect the object boundaries, which may need to be addressed. Some participants suggested a post-cropping feature to further eliminate unnecessary information on snapshots.

Our findings showed how the physical belongings of a user could be blended into the remote user’s person space to enhance social experiences. Users reconfigured physical objects to create blending illusions with stored digital copies and remote users’ video feeds. This enabled playfulness in their social interactions, embodied referencing, and privacy protection. The blending of spaces showed similar benefits to those of ClearBoard [46] and MirrorBlender [34]. In terms of the enhanced social experience, previous work has explored the photorealistic aspect of digital copies, but it has primarily focused on the person space rather than objects, as evidenced by the work of Venolia et al. [94] to support the co-presence of distributed users in group photos, or Follmer et al. [24] to place human video streams in a storybook for playful interactions and emotional togetherness. Thus, ThingShare extends these ideas to the sharing of physical things.

Furthermore, prior work has explored how the choice of virtual background in video chats can affect the remote user’s perception of the local user’s personality traits and exhibit self-presentation [45], which leads to future investigation of the implications of incorporating personal belongings into remote shared environments. With ThingShare, we have shown how users can use digital copies in their person view to customize their virtual backgrounds, e.g., tiling shared personal belongings from a remote partner’s video into one’s own person space. Moreover, ThingShare also addresses the challenge of objects being hidden by the virtual background (C5). By selectively allowing objects to be visible regardless of the presence of a virtual background, users have flexibility to show objects in their local environment without compromising privacy. While ideas such as diminished reality are not new (e.g., [11]), most previous work has not discussed or designed remote collaborative settings that do not require complicated setups [62]. Prior work has explored privacy use cases such as diminishing human activities from the video (e.g., by hiding the temporary absence of activities such as looking at mobile phone [17] and freezing the user’s video feed in a private conversation [43]). To complement these, we have studied the compelling use case of supporting object-sharing while the virtual background is being used.

An interesting future implication is to investigate the potential of using additional modalities to control or automate the boundary of diminished reality with speech, gesture, or proximity cues [11, 17, 102]. Designers could consider some semi-autonomous or gestural detection to initiate the control of an object being diminished or shown that can immediately respond to their behavior, e.g., detecting movement of tangible objects to reveal/hide it to enable touchless and remote pointing like [14, 15].

ThingShare offers three options for sharing objects: live video, snapshots, and short videos. These options are available in both the Context and Focus Mode. We have discussed that interacting with these digital copies in a person view provides augmented context information that includes the surrounding home or workspace environment, the person, and the reference space.

In contrast, the use of digital copies for task-centric sharing enables the user to organize detailed information about the physical object with a shared perspective. This allows for more direct interaction with the object itself, and particularly raises two interesting dimensions to consider: 1) Live versus non-live formats of sharing and their combination, and 2) Images and videos with additional annotations as different types of captured data.

First, live videos were intended to support real-time interaction and manipulation of physical objects. Our study discovered that users appreciated that the Task View further diminished background details into a detailed object sharing. As expected, the non-live capturing of physical objects (i.e., snapshots and short videos) also mitigated the onerous coordination efforts (C4). Moreover, participants preferred the non-live options that freed their hands, as opposed to live video sharing which requires holding items. This aligns with the findings by Fakourfar et al. [21] that showed the effectiveness of freeze-frame in making annotations. Aside from temporally freezing digital copies in video window, a more frequent way to share object with the Task View is to use the combination of live and non-live format of copies to share physical objects. A local user may have the non-live copies (snapshots or short videos) displayed in the task view, while transitioning between looking at the object and showing, orienting, and physically referencing finer details of the item to the remote partner via his or her person view. An interesting implication here is that future designers may consider supporting a balance between the size of the Task View and the user’s person views.

Second, our study found that capturing non-live data (i.e., snapshots and short videos) in addition to live videos improved the ability to show multiple aspects of objects to remote users (C2). While short videos were designed to provide a more efficient way of taking multiple sequential snapshots at once, users were concerned about the potential inefficiency of making longer videos due to frequent coordination, similar to the live video.

In addition to the ad-hoc use of digital copies during conversations, participants requested the feature to download captured objects for later use after the conversation. Though this was not explicitly implemented, they still opened these copies in another window to privately check or attempted to right-click to "download" for later use. This points to an interesting new direction to support different kinds of recording formats, e.g., recording bodily movements [9], for later knowledge sharing [55] or note-taking [72], meeting summarization [64, 107], or for post-hoc learning [12, 20, 32]. Future designers may then need to consider additional curation for the large number of captured non-live copies.

ThingShare can be used to support the ad-hoc and improvised showing of physical objects, while also protecting privacy in both social and professional contexts. The user can also augment the part they are pointing to remote partners by hovering over the object so that the remote user can see exactly where the user is pointing to in their person space. Moreover, as found in the study, users can use ThingShare in remote social gathering for playfulness and background decoration [45]. ThingShare can also be useful for creating memorable experiences between child-parent pairs or between grandparents and grandchildren [25].

ThingShare could be also useful to support a quick transition from ad-hoc conversation to teamwork. A user can share a screen together with the shared document with an object. Additionally, ThingShare’s Task View feature allows users to capture digital copies of physical sketches or scribbles, making it easier to answer remote user’s questions and clarify complicated calculations and visualizations in unplanned manner during synchronous meetings. Distributed users can also use ThingShare as a virtual Miro Board, where they can quickly jot down ideas on physical sticky notes and gather them in the Collage View for brainstorming sessions. ThingShare can also be a valuable tool for designers who rely on everyday objects to inform their design process.

ThingShare can also be helpful for classroom presentation with prototypes or collaborative learning for hands-on activities. In such scenarios, both Person and Task View can be used. Prior work in video-conferencing [31] and live streaming [13] have investigated to support viewer/remote user input systems with chat interface [101], aggregated [13, 31] responses with different levels of streamer adaptability and viewer expressiveness. ThingShare can support a more expressive way of interacting with physical objects in reality realm that differentiate with prior work that supports only digital visual inputs on visuals of physical objects. Moreover, ThingShare features suggested a new modality for systems supporting breakout room like FluidMeet [43] to support the awareness of activity and provides permeable, flexible spaces for hands-on activities [58].

We currently use the Yolact instance segmentation model trained with COCO data set with additional image processing steps. Though the existing implementation was sufficient for evaluating the concept, the model’s performance depends highly on the training data set. The study found that the model may also fail to detect objects that are partially visible in the camera view. A robust instance segmentation method is essential because improper object detection can compromise user privacy. We believe that the image segmentation methods will only become better in the future. Additionally, incorporating 6DoF ad-hoc object tracking [18] may further improve the robustness of the instance segmentation. However, we should also consider methods to prevent potential privacy leaks in case of model failures. We also encourage future work to enrich data sets for physical objects in home/office environments.

All interactions with the interface are limited to the 2D screen and the user’s manipulation of the object. There were other lines of research that investigated the overlay of gestural menus or chironomia with tangible feedback when interacting with objects or data [35], speech-based interfaces [60], and visual captions [63]. These can be interesting future extensions of the current study but not under the scope of this study for communication.

The main scope of this work is to explore the transition between the person and task space using a single video channel, thus designed for the single-camera setup. However, the potential of the ThingShare system could be further extended to other perspectives and camera setups [71] in future research, including contexts such as fabrication workshops [86] that require the use of multiple cameras. In these circumstances, balancing simplicity and flexibility is important as earlier research has found that the variety of cameras and adjustable viewpoints could produce distractions and a lack of mutual orientation and coordination [86, 92]. Considering the semi-fixed cameras and mobile camera collaboration use cases, the mobility of camera setups and viewpoints can cause distractions and difficulties in mutual orientation and coordination, and the rapidly shifting environment in mobile camera collaborations [49] should be take into account for future design of digital copies.

The scope of our study is limited in terms of its scale and lab setup. Thus, further in-depth and long-term evaluations in real-world contexts would be necessary to gain a better understanding of its impact in such settings. For future work, deploying the ThingShare tool in work meetings, classrooms, and informal social gatherings could provide valuable insights. However, studying ThingShare in different contexts may require different design considerations. Currently, ThingShare only examines object-sharing experiences between pairs, so supporting one-to-many scenarios beyond pairs will necessitate additional considerations relating to how to aggregate multiple remote users’ inputs and references [13, 31] in an expressive and effective way [65], as well as how to effectively manage multiple snapshots. Additionally, the integration of digital copies with digital task spaces (e.g., shared screen) while maintaining the benefits of what-you-see-is-what-you-get (WYSIWIS) [44] will need to be taken into account when introducing ThingShare in real contexts.

We present the results for platforms used in video meetings for work-related and non-work-related online meetings or video calls in Table 2. For work-related meetings, respondents reported using other platforms such as Discord (1/2) and RingCentral (1/2). For non-work-related video calls, respondents reported using other platforms such as Discord (2/7), Duo (2/7), Telegram (1/7), Amazon Alexa (1/7), and Viber (1/7).

When asking the type of devices respondents used for online meetings or calls, the majority of participants (74%) reported using Desktop Computer (28.6%) or Laptop (45.4%) during work-related meetings, compared to non-work-related meetings, where only half of respondents (46.2%) using Desktop Computer (17.1%) or Laptop (29.2%). For non-work-related meetings, around half of respondents (52.3%) reported using mobile devices - mobile phone (43.2%) and tablets (9.1%) during non-work-related meetings, compared to only 24.5% respondents using mobile devices - mobile phones (19.9%) and tablets (4.6%) during work-related meetings.