AdHocProx: Sensing Mobile, Ad-Hoc Collaborative Device Formations using Dual Ultra-Wideband Radios

The field of cross-device computing explores how to design interfaces or applications spanning across multiple devices (e.g., tablets, phones, laptops) to best support individual tasks or group collaboration (themes across this field are synthesized in [12]). While part of cross-device computing is about individuals interacting with device ecologies [53, 56, 86], a considerable focus is on multi-device applications for mediating collaboration in small groups. For instance, co-located group collaboration has been facilitated through digital tabletops [20, 72], shared usage of electronic whiteboards and vertically projected screen spaces [9, 10, 40, 41, 65], or the use of diverse ecologies of devices with different form factors [21, 73, 80].

Studies further investigated the use and value of multiple devices in knowledge work activities. For example, studies highlighted how access to both mobiles and additional synchronized interactive tabletops [75], or the use of a supplemental mobile overview device [11], can facilitate the collaborative decision- and sense-making process. And broadening beyond primarily screen-based devices (tablets, phones), other studies found that mobile-AR across devices (multiple viewpoints) mediates collaboration but also observed increased cognitive demand due to context switching. [78]. Later work also identified how different device configurations (e.g., form factors, formations) impact “collaboration strategy, behaviour, and efficacy” in co-located AR applications across multiple devices [79]. More generally, studies investigated the mechanics of mobile co-located experiences [51], including a framework mapping the social, technological, temporal, and spatial characteristics driving design. And more recently, Yuan et al. mapped out the current use and patterns of multi-device use [86].

Towards facilitating groups’ interactions across devices, related research proposed interaction techniques for easier access to and manipulation of digital content, complimentary functions of devices, or migration across devices (Table 3 in [12]). To inform the design of such interaction techniques and to better match interfaces and applications to people’s expectations and current practices, work in cross-device computing has leveraged insights from seminal social theories. Two theories that closely relate to small-group collaboration are proxemics [28] and micro-mobility [50].

Hall’s proxemics theory [28] correlates physical to social distance, where people move closer to and orient towards others for increased engagement. This seminal theory served as a fruitful inspiration in interaction design, where Proxemic Interactions [26] operationalized this theory, building devices and applications that react to the proxemics of people and other devices around them. Five main characteristics often drive these interactions: distance, orientation, movement, identity, and location. Examples include systems that adjust interactions with digital whiteboards [42] or tabletops [4] based on a person’s proximity, mobile devices revealing sharing opportunities when moving closer [7], or environments where control of appliances is mediated and filtered through proxemics [48].

Related – and most relevant when in close proximity – micro-mobility [50] of physical artifacts is “the fine-grained orientation and repositioning of objects so that they may be fully viewed, partially viewed, or concealed from other persons” [57]. Examples of micro-mobility are subtle – and often fluid and ad-hoc – changes of position and orientation of documents to either suggest or inhibit shared access. Everitt et al. applied micro-mobility to the design of interactive tabletop workspaces for easy sharing, reorienting, and segmentation of documents [21]. GroupTogether later used micro-mobility to drive cross-device interaction [57], using subtle changes of position and tilting of devices as cues to provide fluid, less disruptive techniques for co-located collaboration. We incorporate notions of both proxemics and micro-mobility in our work, where we derive proxemic device arrangements to inform interaction techniques relevant to co-located collaboration – focusing on both proximity and relative orientation between devices, fused with micro-mobility cues of tilting and grip changes.

On a technical level, many multi-device collaborative systems need some form of sensing technology to recognize the location of devices, their distance from one another, or even fine-grained changes of proximity and orientation. We synthesize key technical approaches for the sensing of device arrangement, proximity, and orientation.

One common approach is to use camera-based tracking of device location [23, 37, 66], such as with RGB, infrared, or depth-sensing cameras. For instance, PolarTrack uses polarized filters to locate devices [64], EagleSense uses the segmentation of the depth-image to track people and devices [84], and ProximityToolkit uses a combination of infrared marker-based tracking and depth cameras [54]. These tracking approaches have in common that they are implemented as an outside-in sensing infrastructure, where a fixed setup needs to be deployed in the environment (in this case, one or multiple tracking cameras positioned at permanent locations in the environment). Increasingly, research moves towards inside-out tracking, where all tracking hardware is integrated into the device itself —- an example of this is Dearman et al.’s use of the back-facing cameras determining device formations [18].

Measuring acoustic ultrasound signals enables inferring approximate device positions. Systems used deployed networks of fixed infrastructure and mobile ultrasound transducers for tracking device location [1, 30]. Similarly, infrared outside-in setups can track devices, such as with the tracking of ActiveBadge tags [77], the ParcTab cell-based positioning approach [68], or in combination with visual and audio to sense people’s devices [2]. Towards inside-out mobile setups, RELATE Gateways used mobile-only ultrasound transducers for sensing relative device proximity [29], and Wang et al. used ultrasound signals with commodity earphones sensing head position and orientation [76].

Existing internal sensors in off-the-shelf devices, such as magnetometers, can detect pairing of devices when touching [39], sense stacks of devices with magnet and reed-switch combinations [46], or recognize nearby docked devices with several magnets + sensor combinations [35]. Other embedded sensors, such as accelerometers or gyroscopes, can measure the tilting angle of a device, allowing users to automatically flip pages when the device is rotated [14], effectively navigate through long lists [22], or move content across devices [57].

Radio-based signals allow absolute or relative sensing of device location by measuring signal-strength estimates or time-of-flight. For example, Bluetooth signals in combination with audio data can support spatial location of devices [13, 38]. Ultra-Wideband (UWB), another wireless radio technology, has been used to localize objects in space [36], or to perform activity recognition [71]. Traditional UWB applications range from radar imaging to precise locating [88] and tracking [47, 62]. UWB radios incorporate a time-of-flight ranging method that estimates the distance between two radios. The UWB ranging method is used in many outside-in systems with fixed ‘anchor radios’ in the environment, allowing signal trilateration [17, 43], or combined in sensor fusion with camera-tracking and IMU sensors to increase tracking fidelity [67]. Because radio-based signals do not face some of the limitations of other tracking approaches – such as the visible line of sight needed with camera tracking, issues of attenuation or interference when using acoustic signals, or the limited ability of internal sensors to detect other devices – we use UWB radios for our inside-out tracking of device proximity. We further extend earlier UWB sensing approaches with a dual UWB radio setup and by fusing with signals from grip sensors and IMUs, allowing the angle of relative device orientations to be measured.

Human grip and grasp represent a rich source of information to better understand user activities [52, 60]. It has been widely used to reveal users’ intentions [8, 44, 81]. The way people shape their hands to interact with a physical artifact to handle objects or switch between them [8, 58] makes the interaction more explicit to understand [81]. Other research investigates how users grip digital devices such as tablets [74, 85, 87] or small devices [44]. Device bezel [45] and back-of-device [59, 83] remain the most common areas where the recognition happens. Grasp sensing allows detecting handedness [22, 82], avoiding unintended screen rotations [15], or calling up a graphical keyboard [16]. Other work explores front and back touch gestures [83] and bi-manual tablet interactions [74]. Capacitive sensing is a common approach to detecting grip [27]. Motion sensing has also been investigated in combination with touch to enable multi-modal gestures [24, 32, 34, 69]. Our work goes beyond these efforts by using grip sensing on the entire surface of the device, and its position relative to other devices in combination with inertial motion sensing. In our context, we use sets of grip sensors around the device to increase the reliability of our UWB-based inside-out tracking.

UWB radios transmit data in short pulses ( \(2-3~ns\) ) across a wide ( \(> 500~Hz\) ) bandwidth, enabling proximity ranging as well as communication. We build on UWB’s proximity ranging for relative orientation detection using by employing pairs of UWB radios per tracked device. Furthermore, we leverage UWB’s side-channel communication capabilities to transmit data for driving cross-device user interface updates in a peer-to-peer fashion.

We observed that human bodies can greatly affect the UWB ranging signal; a static hand between two radios can introduce up to 1 meter of error (Figure 5). In order to mitigate this interference, we included capacitive electrodes for coarse sensing of how the device is gripped. Four segments of copper tape were added to the AdHocProx enclosure to form the capacitive sensing electrodes that enable detection of grasp along any of the four edges. These electrodes were connected to the ESP32’s touch sensing GPIO (general purpose input/output) pins. This information is used in the sensor processing pipeline to compensate for grip-related deviations in the ranging signal.

The UWB antennas in our prototype exhibit a ‘butterfly’ radiation pattern. We found that a hand moving along one edge of a device did not affect the signal as significantly as the hand being there or not. It might be possible to train a machine learning model to compensate for this based on the UWB signals alone, (i.e., without capacitive sensors) but this process would either require explicitly designed and labeled training data, or a more sophisticated model that learns time-series features. Thus, we deemed the capacitive sensing approach to be simple yet effective, keeping the training and processing steps lightweight.

In addition to the sensors mentioned above, the system uses the inertial measurement unit (IMU) onboard each host device. We fuse the raw accelerometer and gyroscope signals to determine the orientation of the device and detect tilt gestures. These tilt gestures, in combination with device arrangement information, are used to drive cross-device interaction techniques.

AdHocProx’s sensor processing pipeline consists of two components, as shown in Figure 6: the device arrangement tracker, which outputs the orientation of a remote device; and the tilt gesture recognizer, which detects tilt events. The outputs from these two components are combined and provided to the user interface to render feedback.

To determine the arrangement of a network of devices, each device tracks the relative orientation of other remote devices using the UWB proximity ranging signals obtained by the ad-hoc, peer-to-peer networking protocol described previously. For each device, this protocol produces four signals to every remote device in the vicinity: distances are obtained for all permutations between the source device’s two UWB radios and to the remote device’s two UWB radios. The sensor processing pipeline, namely the top half of Figure 6, is then run on each set of signals, outputting an orientation for each remote device.

The UWB signal is extremely noisy. We found that one significant source of noise – inherent in designing a mobile, interactive device – is the presence of human hands. Figure 5 illustrates a device held with the right hand first, then swapped to the left hand in the highlighted region, producing an offset in the UWB signal of around 0.5 meters. We implement an algorithm that suppresses large changes in the UWB signal based on the capacitive grip signal: in a touch event, i.e., when a hand touches or releases the device, the corresponding change in the UWB signal is recorded and compensated for.

After such large jumps in the UWB signals are suppressed, they are then passed into a sliding window of 0.2 seconds, with a 50% overlap (0.1 seconds), from which a set of features is extracted. These features are the differences in means, mins, and maxes for each pairwise combination of channels. On four channels, this step produces a feature vector of \(3 * {4\choose 2} = 18\)  elements.

This feature vector is then fed into scikit-learn’s default implementation of a random forest classifier with 100 estimators. The model outputs one of four cardinal directions; the remote device is determined to be to the left of, right of, above, or below the source device. Although it might be possible to increase the resolution of this output using more sophisticated modeling techniques, we found that the nearest 90 degrees was sufficient to effectively drive cross-device user interfaces. In practice, we observe that device arrangements on the same plane tend to be orthogonal.

In addition to tracking of neighboring devices’ orientation, the sensor processing pipeline also uses the IMU to recognize tilt gestures of the local device. We fuse the accelerometer and gyroscope signals to produce an absolute orientation in space. The tilt gesture recognizer then simply applies thresholds to this orientation signal to determine if a tilt occurs.

We combine the outputs of these two processes to produce a contextually-aware tilt gesture recognizer. We use this output to drive a series of designed cross-device interaction techniques.

Devices. We recruited three groups of four participants at a time. Each participant received a tablet device augmented with the prototype AdHocProx hardware. We used multiple device form-factors to observe a wider range of multi-device physical arrangements and activities. Thus, each group received two Microsoft 12” Surface Pro 7’s (292 mm x 201 mm x 8.5 mm), a 10” Microsoft Surface Go (245 mm x 175 mm x 8.3 mm), and a Microsoft Surface Duo (folded, to present one 5.6” screen, 145.2 mm x 93.3 mm x 9.9 mm). Each device recorded UWB ranging between every tag to every other tag (7 channels), capacitive sensed grip points (4 channels), accelerometer values (3 axes), and gyroscope readings (3 axes). The AdHocProx attachment was connected to its host device over USB, and a JavaScript application enabling quick cross-platform deployment (Windows and Android) using the WebUSB API ran on the host device logging AdHocProx’s sensor data. In addition to recording the inside-out sensor data, a video of each session was also recorded for ground truth, using an Azure Kinect.

Material. We used lightweight digital devices overlaid with paper “displays” for this activity in order to focus on the human behaviors of interest to us, such as device formations, the dynamics of these ad-hoc spatial layouts as focus of a collaborative activity shifts and evolves, as well as the interactions afforded by distinct arrangements. Further, devices overlaid with analog, paper-based information graphics avoid bias towards entrenched digital habits with status-quo touchscreen user interfaces. Our goal was to observe natural physical dynamics of interest, and eliminate factors unrelated to the co-located collaboration task, such as reorienting devices because of screen glare, or shifting grip to reach onto (and tap, touch, or scroll) the screen.

We printed a stack of charts related to COVID-19 trends generated on Our World in Data¹ and clipped them to each device. We printed each chart three times on a single piece of paper, in full, half, and quarter size, matching the 3 device form-factors of the group (Figure 7). It thus enabled participants to exchange a chart between their different devices simply by folding the paper to display the appropriately sized chart. Note that while we printed the exact same charts in different sizes, smaller form-factors were naturally harder to read, especially at a distance for example, with smaller printed text and labels for example. We also allowed participants with larger devices to place two half-size charts side by side.

To collect accurate sensor data from the devices (and each of their motions), we instructed participants to keep each sheet of paper clipped to a device, in any and all of their interactions. We only allowed participants to unclip charts to exchange them with another participant. They did so by unclipping the chart from their device, passing the sheet to another participant, this participant then (un)folding it to the correct form-factor and clipping it to their own device right away.

Finally, we provided analog pens to each participant and informed them of the possibility to annotate directly on paper charts or to clip a white sheet of paper (folded to the correct size) for notes or annotation if desired.

Task. We opted for a task requiring participants to deeply engage with the material provided, leading to natural interaction with their devices (to read and write on the charts clipped to these) and fostering discussion and information/data exchanges with other participants. The goal of each group was to produce a poster depicting the difference in spread and measures taken during COVID between countries. To achieve this, they had to identify the most interesting countries to feature in the poster along with the most interesting set of charts.

To limit the session duration while maximizing the different types of co-located collaboration behaviors, we opted to structure the session into 4 phases. We selected such structure following insights from Olson and Olson [61], characterizing aspects of team work such as working in sub-groups and noting that most productive teams often transition from whole group meetings to sub-groups and back (Figure 8). We selected phases that encompass multiple activities and roles (reading data, writing notes, sharing insights, etc.) as well as transitions from group to sub-group structures to enable us to observe a large range of device arrangements and co-located collaboration behaviors.

Procedure. Participants reviewed and signed a consent form for their participation. The experimenter then explained the rules to manipulate devices and paper props enumerated in the previous section, followed by the presentation of the four different phases of the study. The experimenter reminded them of each milestone before moving on to the next phase, also giving time reminder when a phase lasted longer than estimated from our pilot session. At the end of the session, the experimenter provided a $40 USD gift card to compensate participants for their participation. This study protocol was approved by our Institutional Review Board.

We recruited three groups of four participants each for a total of twelve participants, with different backgrounds such as support engineers and administrative staff. Their self-reported familiarity with the other participants in a group ranged from being complete strangers to very familiar (i.e., they work together on a regular basis). Each session lasted around 75 minutes.

After the session, two researchers independently made observations from the video recordings of the sessions. After discussing with the broader research team, we report the most salient and interesting behaviors observed (Figure 9):

(F1) Device Line-up. Participants frequently re-arranged their devices’ position and orientation when interacting with others in the group. When discussing content on two devices, participants would frequently move their own device closer to the other person’s device. Both would then re-arrange the devices to line them up to align content. Figure 9 also shows that participants changed the orientation of their devices (portrait or landscape) depending on the type of content to align. We observed occurrences of this behavior for all three groups in different phases and activities. For example, two participants aligned their devices next to each other to compare two different charts to triage them, or later line up visual features on each device (i.e., to continue a line chart that started on one device and ended on another) during the analysis phase. Such device arrangement suggests the need to create one larger workspace with multiple devices. (F2) Device Roles. We also observed participants adopt task-based division of labor at times, attributing different roles to their respective devices. We observed occurrences of the behavior in two of the groups. For example, one participant would tilt their device towards another participant to enable both of them to read the content, while the other participant would point at data on his own device to focus the discussion on a specific data point. In another instance, one participant held up a device vertically for an extended duration, while their collaborator noted relevant information on (the paper clipped to) another device using a pen. Figure 9 also illustrates that the roles devices play may be consistent across multiple group arrangements, for example the smaller device was used to show reference material both in pair and whole group work in Group 1. Such device arrangements suggest that people need ways to interact with and flexibly propagate content to other nearby devices. (F3) User Proxy. We observed that devices often served as a proxy for the orientation and relative arrangement (formation) of the user’s body relative to their collaborators during the sessions. This echoes findings from Marquardt et al. [57], with the distinction that participants were seated at a table, instead of standing (as studied by Marquardt et al.). Thus, even when devices remained resting on the table, participants still rearranged them (including sliding devices around) to afford the evolving collaborative activity. Figure 9 illustrates proximity and orientation of the device in response to the shifting focus of collaborative activity. These observations suggest that capturing device arrangements – and their dynamics – offers an insightful proxy for many aspects of individual and group activity around shared physical artifacts.

(F4) Multi-device Territoriality. During individual work as well as dyadic collaboration, we observed instances of participants orienting their device toward themselves to signal unavailability to varying degrees – at times, even tilting the content away to partially conceal it from others in the group. By contrast, during the sharing phase, participants often placed their device in a more “public” orientation to afford viewing by others, including tilting it towards collaborators. During sub-group work, participants often reoriented their devices to form distinct clusters on the table (i.e. with one pair of devices proximal to one another, yet at a distance from the clustered pair of the other sub-group). Figure 9 shows an example of such “Pair territory” as well as other common device arrangements. This demonstrates that the principle of human territoriality, as previously observed in digital tabletop collaboration [70], also applies to multi-device co-located collaboration.

Our formative study doubled as a sensor data collection process. In particular, the sensor data reflected naturalistic behaviors of people interacting with the devices, albeit in a circumstance with a prescribed task for the group activity. From the sensor data collected, we compiled a dataset for offline evaluation. We annotated the orientations of the devices with respect to each other based on the videos recorded of the sessions. The granularity of our annotations was to the closest 90 degrees, in other words, one of the cardinal directions (i.e., left, right, top, bottom) of a remote device with respect to the proximal one. These annotations consisted of labels and corresponding timestamp ranges, and they were paired with the time series sensor data for conducting our offline evaluation.

We sought to build a single generalized model independent of user or device form-factor. Thus our evaluation protocol, as shown in Figure 12, trained on data from two groups and tested on the third. Training and testing on the entire group’s activity, across multiple persons using different sized devices, avoids overfitting our model to any individual user’s idiosyncratic behaviors, or a particular device’s characteristic properties. Therefore our evaluation protocol treats all devices equally, mixing data from all four devices per group for training, and evaluating on all four devices with the resulting generalized model.

To assess the impact of our capacitive grip sensing correction technique, we further conducted a separate ablation study. We ran the same evaluation protocol twice, once without any correction (Figure 10), and once with the previously described grip sensing correction (Figure 11). In the without-correction condition, we fed the raw UWB signals into our sensor signal processing pipeline. This baseline confirms that, without awareness of hand presence and placement, the system can confuse whether devices are to the left or to the right. But devices at the front produced more symmetric signals (because users most commonly tend to grip devices from the sides, rather than at the top or bottom).

In the capacitive-grip-corrected condition, we ran a jump suppression algorithm based on capacitive grip sensing, offsetting the UWB signals accordingly before feeding them into our signal processing pipeline. With the contextual awareness of what hand is holding the device, and when the grip starts and ends, the system was able to disambiguate devices to the left and right, resulting in an accuracy of \(95\%\) (Figure 11). We also noticed that most of the errors produced were temporally fragmented; by simply merging every 5 predictions using a majority vote, we obtained an accuracy of \(99\%\) . Since grip and device placements change on a scale of a few seconds (as opposed to multiple times per second at the level of individual sensor samples) this signal conditioning strategy offers a stable foundation for reliable interaction techniques.

An AdHocProx portal serves as a semi-public, limited shared space between two devices, much in the way that the porch of a home acts as a constrained social space to greet a possibly unfamiliar guest [33].

The sending user can only place a certain amount of content into a portal; the receiving user pulls the content out of the portal to complete the transaction. This design avoids a collaborator potentially interrupting and foisting content onto the main work-area of one’s device at an untimely moment, as well as preventing (possibly unwanted) content from appearing on a receiving device, unless accepted by the receiver (via dragging the content from the local portal to complete the gesture). Note that the latter half of this handshake, i.e. explicitly dragging content out of the portal via touch, serves as a salient gesture that supports social awareness through its visibility to both sender and receiver (while also observable to other collaborators nearby, to support small-group awareness as well).

A portal automatically reveals itself along a screen edge when a remote device approaches from the corresponding direction. This design affords cross-device collaborative activity in accordance with observation (F4) of our formative study, where participants formed different territories with devices placed closer together to reflect collaborator’s intent to work together. Moving an AdHocProx device within a threshold of approximately 0.8 meters triggers the portal. We selected this threshold based on Hall’s notion of proxemic zones, where an arm’s-length of interpersonal distance defines an important perceptual and social boundary for one’s personal space [28].

Our sensor processing pipeline runs locally on each device, recognizing which side of the remote device lies closest – and in what cardinal direction. The AdHocProx user interface layer then opens a portal along the corresponding edge of the local screen; the portal dynamically updates its on-screen placement if the remote device moves closer to a different edge of the screen, or disappears when the remote device moves away.

Due to the completely peer-to-peer nature of our round-robin UWB protocol, these portals open based solely on information available to each device locally. No external server or cloud service is necessary for spatial awareness, nor for portals to appear (or disappear) in response to devices coming and going, respectively.

The sharing of content is likewise peer-to-peer. Namely, AdHocProx shares references to files (rather than the content itself) by sending a URL over the spare bytes of the UWB ranging frames. For example, this can be a permissioned (read/write or read-only) link to an online document. Further, during the interactive drag-and-drop phase of a handshake gesture, AdHocProx keeps the coordinates of file icons synced between devices using these bytes as well. Due to UWB’s limited bandwidth, we do not attempt to directly send media such as images and videos; rather, by sending a reference only, the other device can retrieve it when WiFi (or server access, if hosted in-cloud) becomes available at a later time.

Based on the four phases and collaborative behaviors observed in our formative study (section 4.3), we designed four interaction techniques on top of the portals user interface layer, as detailed below (and in Figure 13.)

Note that we use the scenarios explored and behaviors revealed in our formative study as points of departure, rather than limiting the resulting interaction designs to a literal interpretation of passing pieces of paper around (for example). We also co-opt “digitally-authentic” behaviors such as copy or pan/zoom interactions that tablet and mobile device users have come to take for granted. Nonetheless, by having cross-device versions of such digital super-powers rooted in natural user behaviors, our hope is that the interface can embody these more abstract actions, making them more salient and memorable once discovered.

Move. Once a portal opens, a user can drag a file into it. This experience draws direct inspiration from phase 1 of the formative study, where the physical act of handing a sheet of paper to collaborators lets groups quickly triage and pass related data to one another.

Unlike email or using a cloud service, in which the sending of a file is an entirely disembodied digital experience, the physical metaphor behind this interaction remains consistent with the natural expectations of in-person collaboration. The interaction requires physical co-location and (unlike a digital sync progress bar) leverages the act of dragging the file as a salient physical indicator of the offer, exchange, and completion; a receiver can see the sender’s hand approaching the portal, and likewise, the sender can observe if the receiver accepts their offering. The portal appears automatically, but the exchange of content is designed as a dyadic gesture with one portion completed by each user. (However, since touchscreens cannot detect which user makes contact, it remains possible for a single person to complete the gesture by reaching into the other user’s personal device-space, but such a gesture is observable to both users, and would only be socially acceptable with tacit consent).

Copy. In addition to dragging a file into the portal, a tilt gesture (implemented as a tilt exceeding 20 degrees) triggers the duplication of a file that falls into the portal on its own. The tilt must correspond to the direction of an open portal; tilting in other directions has no effect. Hence, the gesture requires proximity as well as corresponding actions, while fluidly integrating three distinct elements – the verb (Copy), the selection source device, and the selection destination device – into a single, unified command phrase.

Since there is no physical metaphor for the duplication of a sheet of paper, we draw on a micro-mobility gesture [50] to initiate this action – namely the embodied, mutually visible act of tilting one’s device towards a collaborator to afford sharing of content. This consideration also arose during the formative study, particularly during phase 2, where many participants asked the researcher conducting the study if some kind of copy operation was allowed in order for multiple people to look at the same item closely. Our observations with regards to lining-up devices (F1) and tilting the device as an indication of sharing information with others (F3) also informed the design of this technique.

Pan. Unlike the asynchronous Move and Copy operations, Pan enables users to zoom and pan around an image displayed across multiple devices. Respecting the limitations of the portal, to activate a Pan operation, the initiating user enlarges (i.e. zooms into) an image until it expands to reach the portal. In order to confirm the transaction, the receiving user can drag or zoom the image out of the portal on their own device. At this point, both users are able to perform zoom and pan operations on the image. The two screens functionally operate as one large screen, such that one user can touch their finger to one screen, and the other user drags their finger across the other screen, if desired, achieving a distributed zoom operation. Pan was inspired by multiple examples of participants arranging sheets of paper next to each other (mostly in phase 3 of the formative study, while sharing findings) in order to get a larger window into the data visualizations (F1). For example, in many cases, two participants would have two different line graphs that were sequential along the X-axis, so they would arrange the two line graphs to be next to each other to give the feeling that it was one large graph; Pan attempts to create a digital version of this experience. Our observation of the different roles devices played (F2), indicating that actions on one device can impact user expectations for another in a context-dependent way, also informed this interaction technique.

Note. A remote device held at a large tilt (greater than 70 degrees, as used by GroupTogether [57]) – much steeper than the 20-degree tilt that distinguishes Copy – creates a sticky-note icon in the portal that the user of the local device can conveniently drag out and write upon immediately. This interaction was inspired by instances of participants holding a sheet of paper up for another person to inspect or take notes on (for example, see Figure 8).

Since this facilitated one person more easily consuming content while writing on their own device, we assigned the role of capturing notes (writing) to the device flat on the table – with the devices thus serving different roles per (F2). As a result, for the Note interaction the vertical device serves as the “remote” device, while the horizontal device flat on the table acts as the “local” device.

In phases 2 and 4 of the formative study, participants took notes or produced poster content (i.e., headers for the posters) while in this pose, respectively.

The goal of this follow-up study was to gather qualitative feedback on the usefulness of each techniques, in light of the task of the first study. For this reason, we invited participants from the first study to return, with six eventually able to do so. In individual sessions, each participant tried each the interaction techniques described above, implemented on top of the JavaScript application previously used for data collection, but with live sensor signal processing and machine learning models trained on the dataset collected in the formative study.

To elicit feedback on each technique, we used a semi-structure interview.

First, participants responded to each of the following statements on a 7-point Likert scale:

For each question, the experimenter encouraged them to describe the reasons behind their rating.

Then, the experimenter asked them about the strengths and weaknesses of each technique. At the end of all four techniques, they were asked to rank them for overall preference as well as brainstorm any potential interaction techniques of their own.

Each participant received a $20 USD gift card for their participation.

Six participants from the formative study returned for the follow-up study. In general, participants responded favorably to the designed interaction techniques, especially when keeping in mind how they would have applied to the formative study.

Figure 14 depicts the results from the Likert scale for these 6 participants, and Figure 15 depicts the final preference ranking of the four techniques. Note that, while these quantitative results are an indication of the potential usefulness of the techniques, our main goal with these questions was to elicit qualitative feedback to gauge whether our system could offer compelling solutions for ad-hoc, co-located collaboration. We report below the three major themes that emerged from this feedback.

We explored the use of UWB radios in this work due to their increasing availability in commercial devices, including smartphones [3, 6], and location trackers [5, 19]. However, current uses of UWB largely involve a number of static anchors (i.e., UWB radios with a fixed, known location) that triangulate and track the location of a single moving UWB radio. In our work, we developed an ad-hoc round robin algorithm that not only allows devices to take turns ranging to each other without cross-talk, but also without any fixed, centrally coordinating device. This permits new devices to dynamically come and go, with their arrival or departure accounted for in the group round-robin.

AdHocProx required the use of two radios to estimate the angle of a remote device. While in principle UWB radios can measure phase-difference-on-arrival to infer angle [31], in practice this feature remains unavailable on current commercially-available devices or developer kits. We hope that this work and the AdHocProx prototype present compelling demonstrations of valuable functionality that make a strong case to include multiple radios in future mobile devices and tablets.

In this work, we used computing devices such as tablets and smartphones as proxies for the proxemics of their users. In our formative study, we observed users might hold up their device for another person to inspect for an extended amount of time (F3), creating an opportunity for a digital intervention powered by AdHocProx. However, we did observe instances in which computing devices were less robust as proxies to their users. In particular, for larger device form factors laid out on the table. Participants moved their devices along (sliding them on the table) when engaged in deep discussions focused on the material. However, we also observed multiple instances of users leaving their device static on the table while orienting themselves differently in order to speak to someone else at the table.

In addition to the relationship between a person and their own device, it is interesting to look at the relationship between this person and other persons’ devices. In our formative study, we found that participants, no matter how familiar they were with each other, felt no sense of invading private zones. While they did not grab other people’s devices, participants frequently touched other people’s devices to point out to specific content, sometimes reorienting it towards them. Figure 8 (right) and Figure 9 (top-right) depict such behaviors including a participant writing on another participant’s device. Future work is needed to better understand the implication of this observation and if it would hold for multi-device interactions. The paper props used in the study, the somewhat “public” data (not owned by any of the participants), or the lack of access to other personal data on each device all may have contributed to such looser interpretation of private devices.

The AdHocProx prototype described in this paper illustrates the potential of our approach. However, a number of additional considerations come into play when considering the suitability of AdHocProx for more widespread adoption.

Power consumption is a particularly important consideration for mobile devices. Decawave’s DW3000 datasheet reports a typical current consumption of between 40 mA and 55 mA depending on the specific operation and configuration of the radio [63], and we empirically confirmed these values with our prototype. We used two DW3000 radios per device, but we would like to investigate the possibility of using a single radio with an RF switch to alternate between two UWB antennas as a possible cost and power-saving option. Our prototypes also include two ESP microcontrollers; we measured the combined current consumption of each ESP+UWB unit as 8 mA with both ESP and UWB radio idle increasing to 110 mA with both ESP and UWB fully active. While this power draw is not insignificant, we would expect a real-world implementation to use a single, low-power MCU instead of two ESP MCUs—and possibly even leverage an existing system MCU in the mobile device. This would of course reduce the power consumption, as would duty-cycling the AdHocProx hardware whenever possible. The full-active mode is only needed when a device is nearby several other devices, and until/unless this situation is detected it should be possible to run in a ‘discovery’ mode where a single UWB antenna is used at a much lower rate, such at 1 Hz. This could also offer a strategy to scale AdHocProx beyond its current limitation of four simultaneous devices. Although our prototype consumes extra power for signal processing and machine learning algorithms to be run on the host device, all of these processes are reasonably lightweight such that they too could be integrated into a single-MCU solution. The commercial success of UWB-based mobile device services (e.g., the interaction between Apple iPhones and Airtags [5]) indicates that it is indeed feasible to incorporate the necessary hardware components in commercial mobile devices.

Security and privacy are also important considerations for wider adoption. At present, our UWB-based communication protocol is unencrypted. Security would thus require the development of new device attestation and encryption protocols to use in fully public, untrusted settings. With regards to privacy, since our system leverages proxemics, it inherently maintains a degree of privacy as physical proximity between devices is required for any information exchange to occur. Coupled with proximity, the interaction techniques we developed also offer users agency — a user that does not want a portal open to someone else can simply ignore it, or pull their device away.

Our formative and follow-up studies also have a number of limitations. As with all qualitative studies, the generalizability of the results is limited. Insights from the formative study may have been impacted by the familiarity of members of the group and their personal styles of collaboration. The observations made during the formative study inspired a number of techniques, and the feedback gathered during the follow-up study shed some light at to the potential of these techniques (and the potential value of a dual UWB hardware setup) for co-located collaboration. However, additional rounds of studies are needed to formally evaluate the usability of the techniques and more systematically understand their usefulness in a co-located collaboration task.

In addition, both studies were conducted in the same space. It is important to note that the reception of UWB may be affected by reflection and absorbance in the environment, thus, testing this technology in different environments is important to better understand the robustness of the system.

The primary goal of this work was to lay the groundwork for a system that reduces the effort necessary to work across dynamic formations of mobile devices in the context of ad hoc multi-user, multi-device collaboration scenarios. AdHocProx achieves a step in this direction by facilitating a peer-to-peer network connection between devices as well as automatically determining the formation of proximal devices. It is important to note that we only implemented a few interaction techniques inspired by our formative study on top of this platform. Our examples highlight how automatic spatial configuration of devices can make even simple interaction techniques more useful when applied across multiple devices. Furthermore, by combining AdHocProx’s ad hoc networking capabilities with feed-forward graphical affordances in the form of AdHocProx portals, we show a more natural “pairing” experience driven by physical proximity. As the basis for the system is laid out and demonstrated by the ad hoc dynamic realization of the interaction techniques described in this paper, we can envision future work focused on uncovering a suite of novel interaction techniques made possible by AdHocProx, for a variety of co-located collaboration tasks and contexts.