EtherPose: Continuous Hand Pose Tracking with Wrist-Worn Antenna Impedance Characteristic Sensing

The most popular approach for hand sensing near or on the hand are optical methods, as it generality offers high resolution data. While worn systems have employed depth (e.g., WatchSense [52]) and thermal cameras (e.g., Fingertrak [23], Pyro [18], Yamato et al. [67]), by far the most common camera variety used are those operating in the visible or near infrared light range. The latter systems include Digits [29], CyclopsRing [7], Hand with Sensing Sphere [2], Back-Hand-Pose [63], and Opisthenar [68]. Range-finding sensors (optical or acoustic) are also fairly common, and utilized in systems such as ThumbTrak [55], RotoWrist [46] and WristWhirl [17]. A commonality in the above systems is the need for a sensor line of sight, which EtherPose does not require.

Non-optical approaches have also been explored, but generally offer more limited hand tracking functionality. Chief among these methods are electromyography (EMG) and electrical impedance (or capacitance) tomography methods, which we discuss briefly in the next section. Acoustic approaches include in-air ultrasound beamforming [24] and in-body interferometry [25]. Researchers have also looked at passive acoustic approaches, such as listening for touch events on the hand and vibrations induced from object use [32]. Finally, pressure or contour sensors (optical, capacitive or mechanical) have been used to sense deformations in wrist geometry (see e.g., WristFlex [13], Jung et al. [27], Fukui et al. [15], GestureWrist [43], Rudolph et al. [45]), from which certain hand poses can be recognized.

We note that among these systems, most demonstrate static hand ”gesture” classification, with sets of around ten poses. More rare are systems demonstrating continuous hand pose tracking, which is more challenging. Systems that perform continuous hand pose tracking include Digits [29], Fingertrak [23], Back-Hand-Pose [63], WR-Hand [37], and NeuroPose [38]. Beyond hand pose, there are two other input modalities worth noting (which EtherPose implements and evaluates). First is wrist angle input, explored in systems such as WristWhirl [13] and RotoWrist [46]. Second is ”micro-gesture” input, utilizing small and subtle movements of the fingers for interactive control, previously demonstrated in systems such as Soli [36], Pyro [18], AtaTouch [30], ElectroRing [28], and Serendipity [60].

There are several categories of electrical and radio frequency (RF) based hand sensing systems, which we review in order of increasing similarity to EtherPose.

Farthest from our work are electrical bio-sensing systems, such as electromyography (EMG) [37, 38], electrical impedance tomography [71, 73], and bio-capacitive sensing [57]. One step closer to EtherPose are radar-based hand tracking systems, such as Yu et al. [69], Paradiso et al. [40, 41], and Sluÿters et al. [48] (see [1] for an excellent survey of radar-based hand gesture methods). Of these radar systems, only Soli [36] has achieved sufficient miniaturization to be worn (the technology has shipped inside smartphones) and demonstrated fine-grained, micro-gestural input. Highly related to radar approaches are systems utilizing RF reflections from a user's body. This technique has been used with backscattered ”wifi” signals for full body pose estimation (e.g., RF-Pose [75]), as well as with RFID tags worn on the body (e.g., RF-Wear [26]).

Next most related are worn hand input systems that inject RF signals inside the body. For instance, Touché [47] measured swept frequency impedance between two wrist bands for coarse two-hand gestures. SkinTrack [74] used an powered ring that injected 80MHz RF into the body, the phase of which could be captured using four electrodes operating on the skin-side of a smartwatch, enabling on-skin 2D touch tracking. ActiTouch [72] used a 10.5MHz RF signal injected into the wearers arm, and measures the inter-body and in-air radiated energy for on-skin touch detection (with finger position tracking done with computer vision). Essentially the same approach is employed in ElectroRing [28], but with a finger-worn apparatus. Finally, there are systems that may be broadly categorized as in-air electric field sensing approaches, first explored for interactive use in seminal work by Smith et al. [49, 50]. More recently, eRing [62], and PeriSense [61] are ring-like devices that use several electrodes to capacitively sense around the finger for discrete hand pose classification. Cohn et al. [10] demonstrated a wrist worn electric field sensing implementation able to detect five user locomotion modes. AuraSense [76] used four receiver electrodes on the sides of a smartwatch to detect changes in the electric field caused by input from the user's other hand.

Finally, and most related to our work, are other electric field sensing systems, but which use a vector network analyser (VNA) to measure S-parameters of the holistic system (device, user body and environment). We provide a brief primer on the sensing principle in the next section. VNAs are generally large and expensive pieces of bench equipment. Nonetheless, they have been used in a tethered fashion (i.e., antennas worn on body, connected to a benchtop VNA) for sensing coarse human motion, such as arms swinging, rowing, sitting and hopping motions [35]. Xu et al. [64] also used a benchtop VNA, and demonstrated classification of four finger motions using single-frequency impedance over time. We note that we were inspired by this work's use of EM simulations, and included this method as part of our iterative design process. The latter work also proposed a compact folded cylindrical helix antenna design, which we included in our experiments. Finally, in the HCI literature, AtaTouch [30] used a worn VNA (but required wall power) and one dipole antenna to detect discrete pinching gestures. In contrast to the latter systems, EtherPose is comparatively small, self-contained, battery-powered, and demonstrates continuous, real-time 3D hand pose estimation.

EtherPose leverages ”loading mode” electric field sensing, in which a radiating element is sufficiently proximate to a human-body that they capacitively couple (see Smith for an excellent primer on this subject [50]). In our system, the proximity of the radiating element to human tissue (e.g., wrist and hand) means the wearer becomes part of the radiated element ground plane. A ground plane with dimensions around 2 - 3 wavelengths can be approximated to a virtual electric infinite ground plane. In other words, adopting a frequency of operation which has a single wavelength approximately the same dimension as an average human hand, creates a condition where the radiated structure is coupled to a finite ground plane [39, 56]. Therefore, any change in hand pose (e.g., fingers moving closer or father from the antenna) manifests as a change in the antenna structure that, in turn, changes the self-resonance frequency and performance. Depending on the antenna topology, this coupling effect can be varied and enhanced. As illustrated in Figures 2 and 4, the hand and forearm are capacitively loaded to the antenna. As the hand pose changes, the distance between antenna and hand affects the capacitance between them (C₂). Additionally, the relative distances between the fingers varies the capacitance between each digit (C₁). This ground plane effect also modifies the mutual inductance (L₁) between the antenna and hand as the amount of carrying current in the opposite direction is varying according to the ground plane's size, distance, and shape (e.g., loops when fingers pinch). Also, different gestures limit the extent of the hand covered by the electric field, changing the resistance (R₁). The S11 parameter (i.e., scattering parameter, RF transmitted from port 1, reflected RF measured at port 1) describes the ratio between returned signal and incident signal reflected by an impedance discontinuity in the medium. The impedance changes are characterized in the S11 parameter in a certain way [19] and can be measured by a VNA. The discrimination of hand poses is defined by the antenna complex impedance change at a predetermined frequency due the self-resonance shift and performance change caused by alterations in the coupled virtual ground (i.e., hand pose). These principles are well understood, allowing us to run simulations using commercial software, which we describe in Section 3.3.

Left - Illustration of hand, forearm, and antenna. The dotted line is shown as an electric field in the air. The hand pose is an index-pinching gesture. The antenna is located on the wrist. Right - There are two boxes with electric circuit models on the top and bottom. The top one is indicating the antenna, which has an AC power supply, capacitors, resister, and inductor. The inductor is mutually coupled with the bottom box’s inductor (L2). The bottom box is indicating the hand and forearm, which has a parallel inductor (L2), resistor (R2), and capacitor (C2). The bottom box is capacitively coupled with an antenna (C1) and the ground.

As a test platform, we used a NanoVNA V2 [58] connected to a MacBook Pro ”13 (2020) over USB. We could programmatically control the VNA with serial commands from a Python application we wrote for experimentation. Different antennas can be attached to port 1 using a standard SubMiniature A (SMA) connector with a 50 Ω-impedance. The antenna feed line is connected to the signal pin of the SMA connector, while the ground-side line is soldered to the FR-4 copper clad laminate and the SMA's ground pin. An elastic velcro strap was used to affix the apparatus to users’ arms. All antennas were mounted to a 6mm acrylic sheet with cutouts for the wrist band to loop through. For all experiments in this section, the VNA was configured to record S11 parameters, specifically the return loss magnitude and phase shift of a 50MHz sweep (51 data points each), centered at each antenna's resonant frequency.

We selected three exemplary hand poses for all of our experiments: neutral, thumb-to-index pinch, and fist (seen in Figure 9). With our human participants, we had them perform the three hand poses five times each, during which three frames of data were recorded. We selected two evaluation metrics to gauge progress: (1) what is the magnitude and difference between different hand pose signals, and (2) how well does a machine learning classifier distinguish between the three hand poses using such signal. In our Figures, we visualize the difference of signals from one hand pose stimuli to the other and then calculate the average of the difference. To capture ground truth 3D hand keypoints for evaluation, we use MediaPipe Hands [70] and a webcam operating 30 cm below a user's hands (which provides 21 hand 3D keypoints). As a first machine learning evaluation metric, we use SciPy's ExtraTreesRegressor (default parameters) to predict the relative 3D position of 21 hand keypoints given data from our EtherPose band (with results reported as mean per-joint 3D position error). We use the Mano library [44] to generate an animated, 3D hand mesh (Figures 9 and 10; see also Video Figure). As a second machine learning evaluation metric, we train and test a three-class ExtraTreesClassifier (default parameters) on the three discrete hand gestures noted above (with results reported as accuracy percentage or confusion matrices).

We also ran a simulation campaign designed to complement our real-world experiments. This process was done with 3D full electromagnetic (EM) models in CST Microwave Studio [53], a commercial electromagnetic analysis suite. CST uses finite element method, finite integration technique and transmission line matrix method, and is suitable for electrically large or small, low or high Q radiated structures. The simulation models were designed based on the antenna topology, dimensions, hand pose, and material properties adopted in the background experiments. We use a commercially available EM phantom for the hand and arm with material properties provided by SPEAG [51]. In short, EM phantoms are models of the human body that accurately reproduce the effect of the body on electromagnetic radiation.

While we endeavored to create simulations as faithful to real life as possible, there were several limitations. For instance, the antenna manufacturing process and construction varied slightly. There are also inevitable differences between the dielectric constant of complex human tissues vs. a simplified EM phantom. That being said, the largest discrepancy between simulation and measurements was with the fist hand pose. This dissimilarity was chiefly due to the fact our EM phantom had limited articulation and could not be equivalently posed. However, despite the relative compromise between measurements with human tissue and simulations with EM phantoms, we found there was strong correlation in antenna measurements, boosting confidence in our prototype designs and the theoretical principles that underpinned their operation.

Five rows and four columns of a table. The first row indicates antenna topology, which has the actual photo of the antenna. The second row is indicating the real-world measurement of changes in magnitude and phase shift from the vector network analyzer. The third column (Cloverleaf antenna) shows the most distinctive signal changes. The third row notes the Simulation result. The second column (Folded cylindrical helix antenna) is empty with “Not Simulated” text. The third column (Cloverleaf antenna) shows the most distinctive signal changes. The fourth and fifth rows show the mean error of hand pose tracking and classification confusion matrix, respectively.

Three images are showing simulated electric field strength at the resonant frequency for each hand gesture - open, pinch, fist from the left.

A table of five rows and eight columns. The first row indicates the antenna position on the wrist. The second and third rows are showing real-world measurements and simulated S11 signals. The fourth and fifth rows are noting classification results and hand pose tracking errors on the centimeters scale, respectively.

The first and most fundamental design parameter we explored was antenna topology. This impacts not only antenna resonant frequency, but its radiation pattern, both of which have significant implications for coupling with a user's hand. While frequencies between 300-2450Mhz strongly couple to the human body [3, 20, 35], antennas that operate in this range are generally too large to be integrated into a worn device, adding another design constraint.

We identified four antennas topologies of interest (Figure 3): basic monopole, cloverleaf, pagoda [5] and folded cylindrical helix (FCH) [66]. All four types produce a toroidal radiation field in-plane with the arm, which envelops the volume where the hand operates. We purchased off-the-shelf 5GHz monopole, 5.8GHz cloverleaf, and 5.8GHz pagoda antennas, to which we added ground planes (made from FR-4 copper clad laminate) that shifted their resonant frequencies to 2.25, 1.38, and 1.80 GHz, respectively. We fabricated our own folded cylindrical helix antenna with a resonant frequency of 800MHz based on instructions in Xue et al. [66]

With these four antenna designs, we proceeded to run real world experiments to see how our three exemplary hand poses altered the antennas self-resonance, and therefore its characteristic impedance. We also ran matching software simulations for all but the folded cylindrical helix antenna. For all of these experiments – real and simulated – we held constant the antenna position: centered on the arm, just below the wrist crease, which we define as the ”front” position (Figure 5).

Figure 3 provides an overview of these results. The second row shows all 15 trials (5 repeats × 3 frames) for each of the hand-pose pairs (subtracted from one another to highlight the difference) as performed by a real user. The darker colored single lines indicate the average of the lighter colored 15 lines rendered in the background. The third row in the Figure 3 is the difference of the simulated antenna S11 data across each pose (one curve for each pair). While there are differences between the simulated and real-world data, the main result is apparent. The average changes in S11 magnitude and phase shift were 0.52 dB and 3.60° for monopole, 0.25 dB and 1.85° for FCH, 1.93 dB and 12.11° for cloverleaf, 0.36 dB and 2.43° for pagoda. The max changes in S11 of the cloverleaf antenna were 11.55 dB and 99.55°, indicating a 3.8x S11 magnitude difference caused by hand pose stimuli. All antennas change their impedance characteristic in response to the different hand poses, though perhaps most dramatically with the cloverleaf antenna, with almost 50db radiated at its peak frequency.

To test how the measured signal impacted machine learning accuracy, we used our five rounds of real hand pose data to train and test (leave-one-round-out cross validation) a continuous hand model (see Test Procedure section above). We can compare our model's pose predictions against the MediaPipe-captured ground truth and compute mean per-joint position error (MPJPE), which we report in the ”Mean Error” row of Figure 3. We also trained a classification model that simply predicts the three discrete poses. However, all four antennas had 100% classification accuracy, and so we do not provide confusion matrices in this particular figure (”Class. Acc.” row in Figure 6).

All antenna designs showed promise, and were able to accurately predict pose, especially in the discrete pose classification task. The monopole antenna performed best in our machine learning evaluation, but its inherently tall profile was a significant detractor. Balancing accuracy and feasibility, we decided to move forward with our cloverleaf antenna, which performed second best in our machine learning results, demonstrated the most salient differences in its S11 data, and offered a compact geometry.

To better understand how the cloverleaf antenna's electric field was being altered by the three hand poses, we simulated and rendered the electric field distribution alongside the phantom. These sagittal cross-sectional electric field distributions at the antenna resonant frequency can be seen in Figure 4. We can see that in the open hand pose, the electric field distribution at the fingers is reduced, with an small concentration at the middle finger tip. In the pinch pose, the electric field is more evident along the length of the fingers, with an even higher concentration at the finger tips. Finally, in the fist pose, a different electric field distribution is observed with high concentration at the thumb knuckle and index finger tip. These electric field simulation results support the hypotheses that changes to antenna impedance characteristics are resultant from the coupled ground plane's (i.e., forearm and hand) morphology changes. If the extended coupled ground plane created by the forearm and hand would be electrically infinite (i.e., larger than several wavelengths), changes in the electric field distribution would not be noticed or appear in the antenna characteristic impedance.

With our antenna topology selected, we next moved to study the impact of body location on antenna signal. As before, we used a combination of software simulation and real-world measurements (using the same apparatus and procedure as above). Holding other parameters constant, we tested eight body placements: front, front-right, right, back-right, back, back-left, left, and front-left. Figure 5 shows the real-world and simulated S11 plots across the eight positions, along with classification and MPJPE. Coincidentally, the front position (which we utilized in our first experiment) performed best, with 100% classification accuracy for the three poses and the lowest joint error (4.5 mm). The S11 plots also showed the most expressivity in response to the three hand poses, which had the largest signal change of 1.93 dB and 12.11° on average. Both simulation results and measurements show a trend of decreasing signal difference as the antenna moves from the front to the rear for all hand-pose pairs. For these reasons, we decided to proceed with the front position in our design.

After selecting the front position to host a cloverleaf antenna, it was apparent there was sufficient space on the backside of the wrist to host a second antenna. However, as the two antenna will interact with one another, it was not as straightforward as selecting the next-best-performing antenna from the previous study. To give more confidence and clarity in our selection, we ran a final set of simulations and real world measurements. This time, all conditions included a frontal cloverleaf antenna, and we tested a second cloverleaf antenna in five possible positions (left, back-left, back, back-right, and right). Front-left and front-right positions were not possible, as the antennas could not physically fit side-by-side on our wristband.

Figure 6 shows these results, which were created using the same data capture procedure and simulation method as above. Among the five positions, the combination of front & back-right performed best (followed closely behind by front & back). For the secondary antenna position, right and back-right induced distinctive phase shifts (right: 2.31 dB, back-right: 10.39°) and magnitude (right: 1.14 dB, back-right: 13.07°) on average compared to other positions (magnitude:  0.66 dB, phase:  4.25°). For the front antenna position, the signal with the back-right antenna (magnitude: 0.94 dB, phase: 5.33°) was slightly larger than the one with the right antenna (magnitude: 0.72 dB, phase: 3.99°). The machine learning accuracy was essentially unchanged from the prior experiment using only a single front antenna. However, we decided to move ahead with a two-antenna design, as we saw in pilot testing that it could prove useful in capturing some hand configurations in an expanded pose set.

A table of four rows and six columns. The first row is showing secondary antenna locations, which all have a frontal place and then the additional antenna location is varied. The second row notes real-world S11 measurement of two antennas (frontal and the other location) while the only third column is showing a simulated signal. The third and fourth rows are showing hand pose tracking errors and classification accuracies.

A table of two columns and three rows. The first column is showing the simulated electric field heatmap with the front antenna on and the back-right antenna off. The second column is showing the simulated electric field heatmap with the front antenna off and the back-right antenna on. Each row indicates hand poses, open, pinch, and fist.

Our wristband (Figures 1 and 8) features two cloverleaf antennas with ground planes, located in the front and back-right positions, as identified in our background studies. The design and dimensions of these antennas can be seen in Figure 3, cloverleaf. Similar to our testing apparatus, the antennas are mounted to 6mm thick acrylic with cutouts that allow the elastic strap to loop through, permitting flexible antenna placement for a variety of wrist diameters. While we tried to make the two antennas identical, there were nonetheless small construction differences. The resonant frequencies of the front and back-right antennas on the wrist are 1.33 and 1.39 GHz, respectively, while the magnitude of S11 at resonant frequencies are -30 and -42 dB, respectively.

To increase physical robustness for our later user studies, we laser cut horseshoe-shaped wood shims to support the cloverleaf antennas. Each antenna is attached to its own dedicated NanoVNA v2 [58] with a rigid SMA connector. Both VNAs connect to a single Raspberry Pi Zero 2 W over USB, which also provides power. The Raspberry Pi runs our software (described in the next section), with machine learning results streamed over WiFi.

At maximum sensing rate and streaming data over wifi, our band consumes 4.5W. In other words, our bands’ 16Wh LiPo battery provides around 3.5 hours of runtime. Our device cost around $250 to make, with the two VNAs ($98 each), Raspberry Pi Zero 2 W ($15), and battery ($6) dominating the bill of materials. We stress that our design is a proof-of-concept, not optimized for size, aesthetics or manufacturing cost. There are several avenues towards miniaturization. For instance, a RF multiplexer would allow for a single VNA to utilize two or more antennas, rather than having duplicate VNAs. Further, a more advanced VNA could utilize the two antennas to measure S12 and S21 parameters, which could provide new and useful data for pose estimation. If research on single-chip VNAs [8, 9, 14] comes to fruition, it would allow for dramatic miniaturization in the future.

The EtherPose prototype with two antennas, two VNAs, battery, raspberry Pi Zero 2, and USB hub.

On the Raspberry Pi, our software communicates with the two VNAs over USB serial. To initialize itself, each VNA is programmed to measure the return loss magnitude ±20Mhz centered at 1.38 GHz in 21 steps. The peaks (antenna resonant frequencies) are detected and each VNA re-centers itself on this value, most often a small shift, but one that is useful to maximize sensitivity. The VNAs are then configured to sense this frequency range continuously in a alternating fashion (to avoid interfering with one another), such that only one VNA is transmitting and measuring at a time. Each VNA measures return loss magnitude (21 data points) and phase shift (21 data points). As we have two VNAs, a single complete frame of data contains 84 total data points. The data capturing takes approximately 410ms, resulting in a frame rate of 2.4Hz.

Then, for each of the 4 sets of values (two return loss magnitude arrays and two phase shift arrays), we take the first derivative (20 features × 4), find the index of the peak (1 feature × 4), as well as the mean, min, max, and standard deviation (4 values × 4). This produces an additional 100 features.

Lastly, using each VNA's 21 magnitude values and 21 phase shift values, we compute the impedance at each frequency, resulting in 84 additional features (21 real and 21 imaginary components × 2 VNAs). On this, we similarly compute the first derivative (20 features × 4), mean and standard deviation (2 features × 4). This produces another 172 features, for a grand total of 356 (84+100+172) features.

A table with eleven columns and three rows. The first and the last rows are showing the actual hand pose and the generated hand mesh from the EtherPose software, respectively. The second row is visualizing the S11 signals for each gesture.

The table of five columns and three rows. The first and the last rows are showing the actual hand pose and the generated hand mesh from the EtherPose software, respectively. This hand pose set is varying the wrist angles. The second row is visualizing the S11 signals for each gesture.

Drawing inspiration from the literature, we selected three hand input modalities of interest. First and most general is 3D hand pose, previously demonstrated in systems such as Digits [29], Back-Hand-Pose [63], FingerTrak [23], but never with an RF approach. Systems such as WristWhirl [17] and RotoWrist [46] demonstrated 2DOF wrist angle input, which we selected as a second input modality. And finally, we were intrigued by fine-grained finger input (sometimes called ”micro-gestures”) seen in systems such Soli [36], Yu et al. [69], Paradiso et al. [41], and Sluÿters et al. [48]. Each of these input modalities required a different model and training pipeline.

4.3.3 Micro-Gestures. As one example of micro-gesture input, we track the thumb's position relative to the other four fingers, held together and acting like an trackpad. We trained our model (ExtraTreesRegessor, default parameters, 300 estimators) on discrete hand locations presented visually on a computer monitor. Once trained on this grid of data, our model can interpolate to provide continuous tracking. For instance, one can use their index finger as a slider control (Figure 11), clutched if desired. In our user study, described next, we selected a subset of horizontal and vertical positions.

Three figures for hover, touch, and slide interactions. A laptop is beside the hand with the EtherPose wristband prototype. On the laptop screen, there are row signals on the left and the slide bar on the right. In the hovering stage (the most left image), the thumb tip is hovering over the index tip, and the slider head is in the color white. In the touch state (middle), the thumb tip is touching the index fingertip and the slider head is in the color red. In the slide stage (the most right image), the thumb tip is touching the index finger’s proximal interphalangeal joint, and the slider head moves to the right by filling the path to the blue from the white.

After a brief introduction to the study, participants were fitted with our prototype wristband on their right wrist, continuing when they felt it was comfortable and secure. Our study was completed in two phases with two sessions in each. At the start of each session, the band completed its initialization process, centering on each antennas’ resonant frequencies. During data collection, the user was seated in front of a computer monitor, which provided visual instructions.

For hand pose, we use eleven common poses drawn from prior work (Figure 9). To this set, we added four wrist flexions (i.e., angles; Figure 10), with the neutral pose acting as 0° left/right flexion and 0° ulnar/radial flexion. Rather than looking at discrete gesture classification, we follow the continuous hand pose evaluation procedure outlined in FingerTrak [23]. More specifically, users were prompted to slowly match their hand to the pose requested on the computer display. We continuously capture ground truth hand pose using MediaPipe and a webcam located to the side of the hand. When a new frame of data arrives from our prototype (at 2.4 FPS), the most recent MediaPipe hand keypoints are recorded along side the data. Deviating from FingerTrak's procedure, we do not request the user return to the neutral position between requested poses, as this allows us to capture more interesting and diverse intermediate pose states. A single round of data collection consisted of a random ordering of the fifteen hand poses. We collected eight rounds of hand pose data in this fashion, which formed one session.

After a small break, we then repeated the whole process, collecting another session of eight rounds of data, but with the armband covered with two layers of 100% cotton t-shirt fabric, simulating a sleeve of a medium weight garment. Completing all sixteen rounds (over two sessions) of hand pose data collection took approximately 60 minutes and yielded approximately 2000 hand pose instances per participant (19,170 instances across all 9 participants).

After a small break, participants proceeded to a microgesture study. In this procedure, the participants held their non-thumb fingers together in a flat manner, acting like a trackpad for the thumb. We found that neither a Leap Motion Controller or MediaPipe Hands was sufficiently accurate in this task to provide a ground truth. Instead, we collected discrete touch positions visually requested on a computer monitor, with the experimenter manually pressing a key to capture a trial at that instant in time. To account for variation in participant hand size, we use a normalized unit grid as labels for the requested touch positions.

One round of data collection consisted of touching the thumb to one of six positions on the grouped fingers arranged in a T-shape (one axis along the index finger, and the other axis running down the center of all fingers; see Figure 14 for an illustration). Five data frames were captured in each touch position, the order of which was randomly requested. One round consisted of all six positions, and one session consisted of eight rounds. As with our hand pose procedure, an identical second session was completed by participants, but with EtherPose covered in two layers of cotton fabric. Completing two sessions of eight rounds of microgesture data collection took approximately 30 minutes. All combined, this produced 8 rounds × 2 sessions × 6 positions × 5 data frames × 9 participants = 4320 trials.

As noted in the above procedure, half of our study data was collected with the EtherPose band uncovered, while the other half was collected when the band was covered with two layers of 100% cotton t-shirt fabric, simulating a medium weight sleeve.

We first ran our hand pose, wrist angle and microgesture results separating these two conditions, expecting at least some performance degradation when the fabric covered the antennas. However, and encouragingly, we found no performance difference: Hand pose MPJPE when uncovered: 11.32 mm (SD=7.60) vs. covered: 11.66 mm (SD=7.80); Mean wrist angular error when uncovered: 5.60° (SD=1.30) vs. covered: 6.04° (SD=1.37); Micro-gesture mean 2D position error uncovered 12.5 mm (SD=5.1) vs. covered: 11.7 mm (SD=4.3). Thus we simply combine session data for the following results sections, as this is more indicative of real-world use.

As a user's hand is unique to them, we first trained per-participant hand pose models using 12 of a user's 16 rounds of data (drawn from the two sessions of eight rounds). For testing, we use all combinations of four sequential rounds (i.e., rounds 1/2/3/4, rounds 2/3/4/5, etc.), yielding 7 train/test combinations, the results of which are averaged together. Across our nine participants, we found a mean per-joint position error (MPJPE) of 11.57 mm (SD=7.57). These results are broken out by hand joint in Figure 12. This is comparable in performance to the 12.0 mm positional error achieved by FingerTrack [23], which employs four wrist-borne thermal cameras.

Twenty-one bars are shown on the graph. Each bar is indicating the mean per-joint position error for each hand keypoint.

Following the same train/test procedure as our hand pose analysis, we found a mean wrist angular error of 5.87° (SE=0.06) across our nine participants. Broken out, we see a mean left/right flexion error of 5.36° (SE=0.056) and a mean ulnar/radial flexion error of 6.37° (SE=0.065). Figure 13 provides a breakdown of error by participant, though there is no significant effect.

Using the same train/test procedure, we also evaluated a combined hand pose & wrist angle prediction model (i.e., a single model outputs both hand pose and wrist angle given EtherPose data). We found a hand pose MPJPE of 11.58 mm (SD=7.57) and a wrist angle error of 6.16° (SE=0.063). This performance is almost identical to our prior results.

We also trained per-user models on 14 out of 16 rounds of a user's micro-gesture data, testing on the participants’ 15 and 16th holdout rounds (i.e., 8-fold cross validation; results averaged). To account for different participant hand sizes, study data was collected on a unit grid, as noted above. To ”reproject” this into real world units, which are more interpretable, we used an average human hand size with a unit scalar of 23 mm. Across our nine participants, we found an average 2D positional error (i.e., in the plane of the four fingers) of 12.1 mm (SD=9.9). The distribution of touch trials can be seen in Figure 14, along with 2σ error ellipses.

The graph of angular error. The y-axis indicates an angular error in degrees and the x-axis indicates the participant indices. The blue bar (left one among two adjacent bars) is indicating ulnar/radial flexion error and the green bar (right one among two adjacent bars) notes left/right flexion error.