SparseIMU: Computational Design of Sparse IMU Layouts for Sensing Fine-grained Finger Microgestures

Microgestures (or Micro-interactions) refer to the subtle finger movements that are fast, easy to perform, and may not interrupt the other ongoing tasks [5]. They enable myriad applications in different scenarios [31, 34, 35]. Such microgestures are further interesting because they can be performed while holding an object (e.g., a steering wheel [2]) in hand. In such conditions, the physical constraints of each finger vary based on grasp and object type.

Prior works have taken several paths for designing gestures that are possible with the same hand while holding an object: from interviewing experts [102] to using prototypes for understanding holding behavior [90]. Additionally, there is a rich body of prior work on the design of hand gestures (see [94] for a survey). These works adopted different design methods to develop gesture sets and focused on either empty hands or holding an object. A common technique to design gestures in HCI using Guessability-style elicitation studies was proposed by Wobbrock et al. [99]. We build on prior conceptual work that used this technique for deriving single-hand gestures in an empty hand [12], as well as for busy hands holding objects of different grasp types [83]. By consolidating a uni-manual gesture set from these two works, our goal is to enable a generic and scalable solution. In this article, we advance these conceptual foundations through a sensing approach, which makes their application in real-world deployments possible.

Various sensing techniques have been proposed to detect finger gestures. While each has its advantages and disadvantages, it is worth noting that the selected sensing type has a crucial role in the hardware’s placement location and the enabled gesture set. A large body of pioneering work relies on optical sensing for detecting microgestures. CyclopsRing [13] proposed a finger-worn fisheye camera device to detect on-finger and in-air pinch and slide gestures, as well as palm-writing, FingerInput [85] demonstrated the detection of thumb-to-finger gestures using a head-mounted or shoulder-mounted depth sensor. Sugiura et al. [87] have shown recognition of discrete finger-based gestures using an array of photo-reflective sensors placed on the back of hand. A variety of other sensing approaches include ultrasonic [41, 63, 64], infrared [27, 44, 63, 108], pressure [16, 21, 98], magnetic [3, 37, 69], and capacitive techniques [7, 91]. Due to the advances in deep learning, researchers have also demonstrated the detection of fine finger movements using radar sensing [96]. These systems show some remarkable success in enabling gesture recognition in freehand conditions. However, due to the inherent property of such sensing technologies, these approaches can fail under occlusion caused by holding an object.

Although occlusion can be compensated by augmenting an object, the scalability issue can be a bottleneck to practical deployment. Another approach is based on data gloves that are instrumented with sensors [26, 33, 88]. Despite being able to capture high-fidelity information, they are often bulky and hence impede the dexterity of fingers. For a more detailed overview of the different vision-based and glove-based approaches, we refer to [14]. The most closely related approach to our goal of supporting gesture detection in both conditions, freehand and while grasping an object, is proposed using an electromyography band by Saponas et al. [77]. However, the selected grasp variations and the number of gestures are limited due to the lower resolution of the technique. Laput et al. used a smartwatch accelerometer to detect coarse freehand gestures and also demonstrated activity detection [52, 53]. Furthermore, placing an IMU on finger segments has been shown to be effective in capturing subtle finger movements and does not get affected if there is an object in hand [67, 92, 103]. Recently, DualRing [55] presented the usage of two IMUs placed on the thumb and index finger’s proximal segment to detect four grippings postures but did not consider any gestures while holding objects. Bardot et al. [6] suggested the usefulness of a smart-ring (embedded with an IMU and a touchpad) for gestures in hands-busy situations. We take inspiration from these systems and selected IMUs as our sensing technique to simultaneously support gestures with freehand and while holding an object conditions.

While the aforementioned works presented a viable technological solution to capture finger information while holding objects, these do not investigate the optimal sensor placement to fully harness the capability of IMU sensing. Yet, the placement of sensors is as crucial for gesture detection as selecting the appropriate sensing type. This is prominently shown by the findings from Gu et al. [28] and Shi et al. [84] who used a single IMU and determined that touch-contact recognition performance can be strongly increased by investigating the optimal position on different finger segments. Lin et al. [56] used an array of strain gauge sensors to detect finger gestures based on American Sign Language and reported the minimum accuracy of 70.8% can be increased to 95.8% for an identified optimal location. Kubo et al. [51] applied pizo-electric elements to detect thumb, thumb-to-finger, gestures, and palm touches and reported the change in accuracy from 90.6 to 96.6% for an optimal location. All these works employed the trial-and-error approach of moving the sensor at different locations, requiring considerable time and effort. We leverage our dense setup of 17 IMUs to avoid the process of repeating manual trials involving the movement of the single sensor at different locations. Using the principle of compressed sensing and other sophisticated techniques, a large body of work has demonstrated that the human body pose can be reconstructed by a significantly reduced number of sensors [1, 20, 39, 68, 81]. However, as mentioned by Brunton et al. [10], reconstruction and classification are two different problems. While some work exists that uses sparse representation for gesture classification, it mainly uses visual data [62, 75]. To the best of our knowledge, our work is the first that presents a computational method for identifying a sparse layout for gesture classification using IMUs.

Gesture design and recognition have received a lot of attention in HCI. Wobbrock et al. [100] proposed $1 for rapid prototyping of gesture-based interfaces. Long’s Quill [58], a pen gesture system, enables users to create pen gestures by example. Similarly, several design tools have been presented in the HCI literature for the design of various gestures. These include work from Ashbrook et al. [4] and Kohlsdorf et al. [49] that allows the designer to compare a gesture with a corpus of everyday activity data for false positive testing. EventHurdle [47], M.Gesture [46] and Mogeste [70] enable users to compose custom gestures on mobile devices. Gesture Coder [60] is a tool to help developers add multi-touch gestures by demonstrating them on a tablet’s touchscreen. While there are existing machine learning-based frameworks and platforms for quickly prototyping and debugging various classifiers and implementing custom machine learning pipelines [36, 71, 72], they are targeted for programmers and do not consider aspects of interaction design. On the other hand, recent advances in technology have enabled novice users to train and classify custom ML models without the need for programming expertise [61]. However, these majorly address image or audio classification problems. Our main goal behind this work is to use machine learning as a design material [18] and enable designers to prototype custom microgestures without the need for having expertise in ML and programming. Motivated by the challenges of designing a sparse sensor layout, we strive to provide designers with a computational tool that abstracts from the complexity of multiple factors (choice of gesture, object, and location constraint), which are conventionally tuned by manual efforts and require technical skills.

Instead of utilizing commercially available gloves or marker-based solutions [23, 33], we performed the data collection with a customized hand sensor system that preserves the cutaneous properties of the hands, the sense of touch, and does not suffer from occlusion. The sensor system is shown in Figure 2. It offers an unobtrusive setup of 17 synchronized IMUs [76, 93] that provide detailed information about the full articulation of a human hand. It includes 9DOF inertial sensors with 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer (MPU9259, InvenSense Inc., CA, USA) with a footprint of \(3\times 3\) mm, deployed on all three segments of all five fingers using a medical-grade skin-friendly adhesive tape (Helvi Mogritz). The finger IMUs are mounted on flexible sensor strips and connected to a base unit attached at the hand’s back, which includes an additional IMU. A customized fixture with a thin velcro belt is used to fasten the base unit on the hand, and the data is sent to the computer through a USB connection. We also attached a wireless IMU (RehaGait, Hasomed GmbH, Germany) on the distal forearm, to include data comparison from existing consumer devices like smartwatches or fitness trackers, resulting in a total of 17 IMUs. All IMUs are precisely time-synchronized, and the data is captured at a framerate of 100 Hz. We refer to Salchow-Hömmen et al. [76] for full details on formal hardware validation, which found that sensor readings are accurate enough to infer fingertip positions with errors \(\lt\) 2 cm. For the use of the raw IMU data, the hardware does not require any calibration, making it particularly practical and feasible for studies. However, we integrated an initial pose with the hand flat on the table and the straight thumb abducted at a known angle for a few seconds at the beginning of each subject’s recording, in order to boost the dataset’s versatility in light of potential future uses where a baseline or calibration pose might be desired. We also note that the framerate of our dense setup of 17 IMUs is in line with that of Xu et al.’s [104] recent work, which suggests that 100 Hz is sufficient for hand gestures’ classification. Furthermore, prior studies have found that even the quick movements of the fingers are slower than 10 Hz [40, 42].

We collected data in Freehand and while Grasping an object conditions. For the latter, we selected a set of objects that are representative of real-world tasks. Specifically, we chose objects labeled in the VLOG Dataset [24] which is based on internet video logs of everyday activities. To ensure we have representatives for each type of grasp, we categorized the objects based on Schlesinger’s Grasp Taxonomy [79]; this has been widely employed by prior works [19, 22, 77, 83]. For each grasp type, we focused on non-deformable objects with two size variations Small (S) and Large (L). The VLOG Dataset does not contain objects that correspond to Small Tip and Spherical grasps, which is presumably a result of not all grasp types being equally well-represented in everyday life [11]. Therefore, we added two additional objects, a Needle and Pestle, to obtain an exhaustive list of objects covering all grasp types [83]. The complete set of 12 objects and their corresponding grasp type is shown in Figure 3.

For Freehand and Grasping conditions, we collected finger movements while performing microgestures and non-gesturing states. For the microgestures, we focused on conscious subtle finger movements that do not require altering the grasp. We selected six primitive finger movements based on bio-mechanical characteristics [43, 95], shown in Figure 4: Tap, Flexion, Extension, Abduction, Adduction, and Circumduction. For consistency of gestures across different fingers, we use the Ring finger as the reference to define Abduction (away from the Ring finger) and vice-versa for Adduction gestures. Furthermore, the swipe gesture was recorded with the participant’s finger motion from one extreme until it reached the opposite extreme. Following Ashbrook’s definition of micro-interactions [5], we further limited our set to gestures with a short duration (4 seconds or less). Moreover, we centered our data collection on single-finger gestures because they promise to increase robustness [82]. In terms of gestural input, these movements translate to both - continuous and discrete gestures through directional sliding and tapping.

The collected non-gesture states include a variety of finger movements that users perform consciously or unconsciously during conventional hand/object interaction. For instance, free hand movements while talking, adjusting the grip, turning the object for visual inspection, manipulating the object, or fiddling. For capturing non-gesture conditions, we recorded Static hold, Primary action (e.g., writing with a pen, drinking with a glass), and Unscripted actions (e.g., adjusting grip, fiddling). The participants were given no explicit instructions while the data for Unscripted action was recorded.

Since moving a finger while holding an object risks dropping the object, we empirically verified which fingers can be moved while holding objects. To consolidate our choice of finger movements, we conducted a pilot study. Two interaction design experts independently recorded their response on a 7-point Likert scale (1: impossible to perform and leads to dropping the object; 7: very intuitive and easy to perform). This resulted in a total of 360 gestures: 6 (gestures) \(\times\) 5 (fingers) \(\times\) 12 (objects) inspected by each expert. Of 720 Likert scale readings, 42 gestures received a rating of 1 by both the experts and these were marked as impossible. Consequently, we focus on the Thumb, Index, and Middle fingers as our main gesture fingers; a choice which is in-line with prior works [12, 82].

We recruited 12 participants (6 M, 6 F, mean age: 26.1; SD: 3.4) with different professional backgrounds, including a computer graphics researcher, firefighter, and kindergarten teacher. Ten were right-handed, and two reported themselves as ambi-dexterous. We measured their hand size from the Wrist to each finger’s tip and found an average length to Thumb’s tip: 137 mm (SD: 8 mm), Index: 181 mm (SD: 12 mm), Middle: 192 mm (SD: 12 mm), Ring: 181 mm (SD:10 mm), and Pinky: 157 mm (SD: 9 mm). For context, the average hand length (middle finger’s tip to the wrist crease) is 193 mm and 180 mm for males and females, respectively [74]. Participation to our data collection was voluntary while adhering to the institution’s COVID-19 rules and regulations, and each participant received a compensation of 30 Euros.

Before starting the data collection, we demonstrated the gestures on an abstract cylindrical object that was not used any further. Once the participants got familiarized with the gestures, we attached the hardware to their dominant hand, and they performed the initial pose by placing the hand on the table. For the Grasping condition, we asked the participants in perform gestures on the object (while maintaining the grasp), and use the palm as the surface for the Freehand condition. Of note, the same hand was used for holding the object and for gesturing. Furthermore, the directional orientation was kept constant across each participant. They performed all the gestures while sitting on a chair, except for Box and Bag, wherein we systematically added variation in posture and orientation for each participant by asking them to perform the gestures while standing and facing perpendicularly. We counterbalanced the two conditions (Freehand and Grasping) and further counterbalanced the order of objects (grasp variations). Once the Freehand or the Grasp variation was selected, we presented the gestures with the specific finger name and non-gesture states in a randomized order. We recorded five trials for each gesture. To collect data from non-gesture states without interruption, we recorded one long sequence of around 30 seconds and split it into five trials. The dataset collection took approximately 3 hours per participant with breaks in-between to avoid fatigue. The sessions were also video recorded. Using a custom MATLAB application, the experimenter manually annotated the trials during data collection with the participants orally communicating the start and stop of the gesture. The labels include information about the freehand or specific grasp variation, gestures along with the instructed finger, and the three non-gesture states. Overall, our dataset contains a total of 13,860 trials (1,155 trials \(\times\) 12 participants) with 18 different gesture and three non-gesture states performed on 12 Grasp variations and with Freehand.

Aiming to understand the underlying factors affecting performance rate due to IMUs’ location, we started off by creating a classification pipeline. Given the size of our search space has the large number of 393 K layouts, we created a gesture detection pipeline with two essential requirements: scalable and rapid train-test time.

Feature Extraction. From a given trial and for each of the 9 axes of an IMU, we extract six statistical features: maximum, mean, median, minimum, standard deviation, and variance. In total, the number of features from all 17 IMUs \(\times\) 9 axes \(\times\) 6 features amounts to 918. To compile this list of features, we drew inspiration from the automatic feature extraction library, TsFresh [15], which has shown promising results in prior work on gesture and activity recognition [27, 45, 57]. Due to multiple sensors and reduced computational load, we used the minimum configuration of the library’s functionalities. To further minimize the effect of different trial lengths, we removed the sum and length features. Due to the lower sampling rate of our 17-IMUs setup as compared to single-sensor approaches [53], we did not extract features from the frequency domain. However, we note that our released dataset will allow the research community to feed more features of TsFresh into the neural network [45], take advantage of a single feature, such as derivatives as input into the neural network [84], or further perform feature engineering for input in non-neural-network or neural-network classifiers to improvise the recognition rate based on the optimal location. In Section 4.1, we show the correlation of our selected features and a different set of features from related work to show the correlation in the ranking of layouts.

Method. We selected 10 random participants as training set and the remaining two as test set (80:20 split) and created grasp-independent models, i.e., the class labels do not include any grasp information. We also performed a leave-one-person-out analysis in Section 4.5. For our multi-class classification, we used 19 classes: (3 fingers \(\times\) 6 gestures) + 1 Static hold. Different IMU layouts may contain different amounts of IMUs (from 1–17); therefore, to compare different state-of-the-art classifiers and estimate the classification time required for the full combinatorial classification, we evaluated randomly selected 100 layouts for a given IMU count of 1–17, totaling 1,435 layouts. Note, for count = 1, 16, and 17, the total possible layouts are slower than 100.

Classifier Selection. We fed our extracted features into multiple commonly used classifiers to evaluate their recognition rate and training time. Specifically, we used scikit-learn’s implementation of Support Vector Classification (SVC), Logistic Regression (LR), k-nearest neighbors (KNNs), Random Forest (RF) with max_depth = 30; and PyTorch implementation for Neural Network (NN) with 4 fully connected layers of decreasing hidden layer size (n = 1,024, 512, 256, ReLU activation) and a final softmax activated classification layer. Only NN models were trained on a GPU machine and others on a 40-core CPU. We used the default parameters for all the classifiers to perform trial-by-trial basis classification. As a performance metric, we used the macro average of the F1 score because it considers both precision and recall.

Results. As shown in Figure 5, the F1 score and training time largely depend on the choice of classifiers. Since we wanted to use the same classifier for multiple settings in the following analyses, as well as the later-described computational design tool (see Section 6)—we opted for Random Forest. This classifier achieves an average F1 score close to the highest one obtained by Neural Network while having a lower training time than Neural Network. Furthermore, RF models can be easily computed on a consumer-grade CPU machine. In-line with findings from prior work [101], our results show that Random Forest Classifier has superior performance than KNN.

As shown above, our released dataset allows for generating results with various classification models techniques. Through our analysis, we found that, while different models may yield different accuracy levels, the order of performance of individual layouts is very similar. Specifically, to understand our results’ dependence on a particular classifier, we used F1 scores of all layouts with sensor count = 1 from the top-performing classifiers, namely KNN, Ridge, RF, and NN. Following that, we sorted the results alphabetically by IMU labels. Then, using a pairwise Spearman correlation (as used by Guzdial et al. [32] for comparing ranked lists), we obtained a correlation of 0.919, 0.975, and 0.919 with p<0.001 for RF vs. KNN, NN, and Ridge, respectively.

In addition, we conducted a similar analysis to understand the change in the ranking of IMUs for different sets of features. We selected five features (maximum, minimum, mean, skewness, and kurtosis) used in the existing literature on IMU sensing [28] and trained 17 models with RF. Subsequently, similar to the analysis comparing different classifiers, we calculated the Spearman correlation on the F1 score of alphabetically-sorted IMU’s list from both feature sets. Our results show a high correlation of 0.995 with p<0.001 between the layout ranking produced by 2 different set of features, indicating that while selecting other features may result in a different F1 score, the order of IMUs remains very similar.

The large count of IMUs offers the possibility of creating vast layout combinations. However, not every count and layout may produce a similar recognition performance. Therefore, an important aspect that we examined was identifying the best-performing sparse layout for a given number of IMUs. This analysis provides three major insights: Firstly, it allows us to understand how the recognition performance varies with the number of IMUs. Secondly, it gives insights into the interval in which F1 scores fall for any given number of IMUs. Lastly, the results inform the optimal IMU placement location with a fixed budget of sensors [10]. Of note, we use the term IMU Count to refer to any given amount of IMUs from 1–17.

Method. To explore the full combinatorial space, we trained models with all possible layouts from 1 to 17 IMUs on our initial train-test split as described in Section 4.1. Moreover, to systematically understand the variation in performance for both types of microgestures, we performed this analysis for three conditions: Freehand, Grasping, and Both Combined. This totals to 3 \(\times\) ( \(2^{17} - 1\) ) = 393,213 models. For each model, we performed multi-class classification with 19 classes: (3 fingers \(\times\) 6 gestures) + 1 Static hold. Note, Grasping and Both Combined conditions utilized grasp-independent models; therefore, we did not encode grasp information in the class labels. In Section 4.6, we compare our results with grasp-dependent models.

Results. Figure 6 plots the F1 score on the test set from each 393 K models trained in all three conditions (Freehand, Grasping, Both Combined), organized by the count of IMUs present in the model. We now discuss each condition in turn:

Having identified that the choice of finger segments for IMU placement can be crucial for obtaining high recognition performance, we now aim at investigating the influence of finger segments on recognition performance more systematically. This also informs the design of minimal form-factor devices that place IMUs only at the optimal segment.

Method. We used our initial 80:20 train-test split of the participants’ data and evaluated using a single IMU under multiple settings. To reduce any effects caused by different grasp variations, we created grasp-dependent models. Moreover, for a clear understanding of individual fingers and their respective gestures, we performed finger-wise classification, i.e., atmost six gestures and one static hold class per finger. Overall, we trained 17 single-IMU layouts \(\times\) [(1 Freehand \(\times\) 3 gesturing fingers) + (9 Grasp variations \(\times\) 3 gesturing fingers) + (3 Grasp variations \(\times\) 1 gesturing finger)] = 561 models. For the analysis in this section, we focus on the IMU on gesturing fingers and on three representative grasp variations that have been identified in prior work to each represent a cluster of Grasping microgestures [83]. The detailed results, including IMUs on non-gesturing fingers and all 12 grasp variations will be released with our dataset.

Results. As illustrated by Figures 8 and 9, the F1 score varies greatly across different segments for Freehand as well as Grasping microgestures. In particular, it indicates that for some cases, the F1 score for a gesture may even rise from 0.0 to 1.0 depending on what segment the IMU is placed on the same finger. In the following, we highlight this effect for Freehand as well as Grasping microgestures.

Implications. Depending on the grasp, finger, and type of movement during the gesture, the single-IMU performance across segments greatly varies. This formally validates our initial findings from the full combinatorial classification results: The choice of finger segment for the IMU sensor placement can have a very strong influence on classification performance. However, since these classification results differ based on the subset of grasps and chosen gesture classes, a one-fits-all design solution will likely not lead to best results. Hence, we propose a computational design tool in Section 6, which provides layout recommendations based on the user-defined parameters.

Finger co-activation is a widely known phenomenon in bio-mechanics [78]. Our goal is to leverage finger co-activation and investigate if micro-movements caused in neighboring fingers are sufficient for gesture detection from a non-gesturing finger. This would be beneficial in situations where placement of an IMU on the gesturing finger would hinder the primary activity–e.g., having an IMU on the Index finger may hinder situations like using a knife. In such scenarios, placing the IMU on an alternative location capable of detecting gestures from a neighboring finger would be more desirable.

Method:. To investigate the possibility of detecting gestures with any single finger, we used our initial 80:20 train-test split and trained five models for each of the three gesturing fingers; each model comprised a total of three IMUs placed on every segment of the respective finger. For a detailed analysis, we performed grasp-dependent and finger-wise classification. This gives a total of 5 fingers w/ IMUs \(\times\) 3 gesturing fingers = 15 models for Freehand. We trained another 150 models [(5 fingers w/ IMUs \(\times\) 9 grasp variations \(\times\) 3 gesturing fingers) + (5 fingers fingers w/ IMUs \(\times\) 3 grasp variations \(\times\) 1 gesturing finger]. In each multi-class model, we included all six gestures for an individual finger and the static class - totaling up to seven classes.

Results. Figures 10 and 11 show the F1 score on the test set for Freehand and Grasping when models are trained with IMUs on different fingers. These results indicate the feasibility of detecting gestures from IMUs on the non-gesturing finger:

Implications When the hands are busy, instrumenting gesturing fingers might not be possible in all cases. For example, while writing, instrumenting fingers involved in gripping the pen might hinder the primary activity. In such scenarios, placing an IMU on neighboring fingers can be efficient. Our findings show that placing IMUs on a non-gesturing finger may enable gesture detection at a comparable or even higher performance rate.

Next, we aim at understanding the extent of inter-personal differences in recognition performance. This is a crucial question because there can be inter-personal variations in the way the microgestures are performed. If there is a large difference in classification results across participants, the design tool that we describe in later Section 6 would need to account for it while suggesting a sparse layout.

Method. A comprehensive Leave-one-person-out (LOPO) evaluation with 12 participants \(\times\) 393,213 layouts = 4,718,556 models will approximately take 25 days of computation time on our 40-core machine. To circumvent this problem, we first identified the best layout according to the F1 score for a given count of IMUs on our 80:20 participants split from the combinatorial results obtained with the combined condition (Freehand+Grasping). Subsequently, we used these best layouts and trained 204 models (12 participants \(\times\) 17 best layouts for the IMU Counts) for a LOPO evaluation.

Results. Figure 12 depicts the results of the LOPO evaluation. We observe that the difference in F1 score from our randomly selected 80:20 train-test split and any LOPO model is about \(\pm\) 6%. It is worth noting that most participants achieved higher performance than our randomly chosen participants.

Implications. Despite the inter-personal variations in how the gestures are performed, our recognition pipeline still scales well and achieves high recognition performance with user-independent models. We observed only little variation in F1 scores across participants, which demonstrates that model predictions generalize to data from new users.

In our combinatorial analysis, we trained grasp-independent classifiers by combining all grasp variations. Here, we aim at investigating if these initial results can be further improved if a subset of grasps is selected. This would be relevant for application cases that comprise selected activities with a known set of grasps, or for systems that can identify the current grasp, e.g., by using activity recognition.

Method. We classified all 12 grasp variations separately (grasp-dependent models) by using our initial 80:20 split of participants’ data with 19 classes [(3 fingers \(\times\) 6 gestures) + 1 static hold]. To save on the computation time, we performed the full combinatorial evaluation of grasp-dependent models until IMU count = 5. There were 12 grasp variations \(\times\) \(\displaystyle \sum _{r=1}^{5}\) \({}^{17}C_{r}\) layouts = 112, 812 models.

Results. For 9 out of 12 grasp variations, the F1 score increased when the model is trained on a specific activity (see Figure 13). Grasps like Lateral-S (Spoon), Tip-S (Needle), Lateral-L (Paper) showed an improvement in recognition of 20–30% compared to the grasp-independent model. In contrast, grasps like Cylindrical-S (Knife) and Tip-L (Pen) did not show any increment, which can be due to the object’s geometry. Specifically, on such grasp variations, the fingers are tightly packed, hindering the finger movement while performing gestures.

Implications. The performance tends to improve if the model is trained for a specific grasp variation. Therefore, when a subset of grasp-variations are chosen that map to a specific context, our results from the combinatorial analysis can further improve. This feature of selecting grasps is also integrated in our later presented design tool for finding a sparse layout.

The key takeaways from the above in-depth analyses are:

Given this multi-factorial design space that influences the classification performance, providing an automated system to a designer will enable rapid design iterations and decision making for optimal IMU placement. Inspired by these findings, we present a rapid technique to identify sparse layouts and a GUI-based computational design tool in the following sections.

To benchmark the selections generated from the two proxy metrics (Feature and Permutation Importance), we use the IMU layouts from our combinatorial results that achieved the maximum F1 score in Section 4.2. To quantify the differences, we obtain a Spearman’s correlation ( \(\rho\) ) between the F1 score from the max. combinatorial layout and the layouts from the two metrics. Permutation Importance received \(\rho\) = 0.7785 for the Freehand, 0.6617 for the Grasping, and 0.8864 for the combined condition (all p<0.005). In contrast, Feature Importance received considerably higher correlations, with \(\rho\) = 0.8630, 0.9380, and 0.9419 for the respective conditions (p<0.005). The high correlation using Feature Importance is also visible in Figure 14, where the layouts consistently obtained an F1 score closer to the best performance in the combinatorial results. Therefore, we use this metric further to calculate the computation time.

Runtime. We now quantify the significant reduction of computation time required to select sparse layouts with the proposed SparseIMU method using Feature Importance. Given the 323K models needed to evaluate the entire combinatorial space, we used our institution’s cluster system with a 40-core setup. Of note, this high-end configuration machine used in our combinatorial results is not widely accessible. In contrast, we evaluate our rapid method’s performance on a commodity laptop (8-core MacBook Air). As shown in Figure 15, the time required to find the sparse layout by our method is significantly shorter, despite the use of a commodity laptop. This reduction is possible due to the considerably smaller number of model training required across each IMU count. For instance, if we were looking for a layout with \(k=5\) IMUs out of \(n=17\) possible IMUs in the Freehand condition, the time reduces from 3 minutes on the compute cluster to 1 minute on a consumer-grade laptop. Moreover, for Grasping Microgestures and Both Combined conditions, it reduces from about 50 minutes to 5 minutes and from 1 hour to about 6 minutes, respectively. While it takes longer to find solutions for IMU counts 7–11, we note that the method still performs significantly faster than the baseline. Moreover, we expect that layouts with this large number of IMUs need to be rarely considered, since going beyond 3–4 IMUs will only lead to a maximum increase of 4% in the F1 score, as we have shown above (see Figure 6). Overall, the reduction in time achieved by our method on a commodity laptop offers strong benefits for rapid iteration. In the next section, we use our method in a computational design tool.

In addition to the validation of the SparseIMU method in Section 5.1, we performed another benchmarking to compare the tool’s output with the combinatorial results when the designer applies constraints and opts for choosing a subset of grasp variation and gestures. Therefore we created six example cases covering all three conditions. We randomly selected grasp variations, gestures and added finger-wise placement constraints. Informed by results from the first validation study, we chose two variations of IMU counts that we consider particularly promising for applications: 3 IMUs for a good recognition performance with very good wearability due to the low number of IMUs; and 5 IMUs for further increased recognition performance with a level of wearability that is still acceptable in many applications. We compared our tool’s estimation by creating new combinatorial results for each case.

Results. Table 1 lists the example cases along with the results. In five out of six cases, the tool selected layouts that achieved an F1 score that was as high as the best-performing combinatorial result or a maximum of 2% lower. The largest difference of 8% occurred in case 4, wherein the tool selected a layout with an F1 score of 0.92, while the best-performing combinatorial layout achieved a full 1.00. Noteworthy, the tool also performed well in case 3, in which most of the randomly selected gestures involve the Middle finger whereas the constraint was to exclude the Middle finger from placing IMUs. Despite this demanding constraint, the tool successfully selected a layout that achieves performance close to the layout found by exploring the entire combinatorial space.

Smart kitchens, providing in-situ instructions while cooking, has been a popular research area over the last decade [50]. We envision our computational design tool to support a designer, Alice, in the development of an in-situ recipe manager that supports information access using microgestures while cooking. For her first prototype, Alice wants to enable microgestures on four objects commonly found in the kitchen: knife, bottle, cup, and pestle (cf., Figure 17(a)). For browsing a recipe, her application requires a small, concise set of gestures: back (abduction), forward (adduction), and select (tap). Due to frequent hand washing, the layout should be minimal (1 IMU) and restricted to the back of the hand or wrist (cf., Figure 17(b)).

Tool Output: With the selection of objects and gestures (and no further constraints imposed), the computational design tool suggests the thumb as a common finger capable of performing all desired gestures, and the thumb’s middle segment for IMU placement. Being “most ideal”, this sensor location achieves an F1 score of 99.4% (cf., Figure 17(c)). However, Alice, excluded the fingers as sensor locations for sanitary reasons. This restrains sensor placement to the back of the hand and wrist, which achieve an F1 score of 76.8% and 56.6% respectively. For both, the confusion matrices reveal that the adduction gesture has a lower score, likely due to the large distance between the IMU and the gesturing finger. As a result, Alice settles on a tradeoff between IMU location and available gestures. To keep the IMU position on the back-of-the-hand, she updates her design to include only tap and abduction gestures, increasing F1 score to 82.6%.

Exploring diverse gestural inputs for VR [48] has been a popular area for experimentation in HCI and media arts. Dan plans a VR media arts installation which uses microgestures on a hand-held VR controller to contrast private and public interactions by subtly expanding the controller’s range of functions. Thus, as demonstrated by [38], he aims for a miniaturized device equipped with 3–4 IMUs in combination of a commodity VR controller. He wants to avoid placing IMUs on the index finger which operates the VR contoller’s push button and also not use it as a gesturing finger. To facilitate playful public or private interactions, he hopes to support as many different gestures as possible.

Tool Output: Dan explores the solution space for all possible IMU locations excluding the index finger (14 IMUs total). The tool yields an F1 score of 80.2% if 12 gestures are supported. Dan iteratively decreases the IMU count (while keeping the amount of gestures to 12) inspecting performance after each decrement. He identifies a saturation in F1 score at 3 IMUs (80.5%), which illustrates that a higher number of IMUs does not necessarily imply better performance (cf., Figure 20(c)). After further tweaking their configuration, Dan settles on a 3-IMU configuration and a set of 10 gestures. This choice is a tradeoff allowing for a relatively high amount of gestures while still achieving an F1 score of 84.3%. As Dan aims for a rather playful, explorative VR installation, he considers this level of score acceptable. This highlights how the choice of a final layout depends on the weight the designer assigns to the different parameters (e.g., amount of gestures vs. performance) which in turn strongly related to the specific application (e.g., playful vs. safety-critical purposes).

Carla seeks to explore how users can make use of microgestures to access additional instructions during high-precision tasks such as soldering. She envisions tools such as a soldering iron, soldering lead, or a screwdriver (cf., Figure 21(a)). As these tools are not available in our dataset, she uses our computational design tool to make an informed best guess by determining a set of initial layouts to elaborate on. Here, our Tool draws strength from the similarity in grasp types: the soldering iron (not present in the dataset) is typically held in a fashion similar to the pen (present in the dataset); holding fine soldering lead or wire in place resembles holding a needle, and holding a screwdriver demonstrates a similar (cylindrical) grasp like holding a knife. Carla envisions four gestures: forward, backward, select, and circle which she intends to use to browse an instruction manual. She furthermore excludes the thumb and index finger–both as gesturing fingers and for IMU placement–to not interfere with the high-precision soldering task, and constrains the number of IMUs to 2 or 3 (cf., Figure 21(b)).

Tool Output: The computational design tool suggests placing the IMUs on the middle finger which achieves a competitive F1 score of 88.7% when 3 IMUs are used. Yet, at closer inspection, the tool also reveals that accuracy varies depending on the finger segment on which the IMU is placed, ranging from 80% to 88%. Hence, the choice of finger segment is crucial. Moreover, the tool shows that there is only 2% gain in score from placing 3 IMUs on the middle and pinky finger (88.7%), compared to only one IMU on its middle segment. Thus, a single IMU is sufficient to cover all gestures Carla had planned for her scenario. Further exploration shows that an increase in accuracy can be obtained for the 1-IMU layout to 93.4% by removing the adduction gesture (cf. Figure 21(c)). As follow-up, Carla conducts a small-scale data collection using the 1-IMU layout recommended by the tool. Here, the tool provided a best guess in terms of IMU placement and gesture choice which served as a strong foundation for further iterations.

When constructing our dataset, we leveraged prior work on grasp types [79] to build six categories and selected representative small and large objects as well as corresponding realistic actions (cf., Figure 3). While exhaustively covering all conceivable objects for each grasp type is impossible, we anticipate generalizability for objects not present in the dataset. A few characteristics of finger movements directly depend on the grasp type and hence generalize for objects beyond the ones present in the dataset, such as the feasibility of gestures with a specific finger, and the co-activation of the non-gesturing finger. There are a few other characteristics of object manipulation which might not be generalize and which future work needs to address. For example, two objects may afford the same grasp type but fulfill different purposes (e.g., pen vs. soldering iron) and require different movements (fluent writing vs. a steady hold for soldering). Our computational design tool incorporates this limitation by assuming the user would briefly pause the primary activity while keeping the object in hand. The additionally collected activity data allows future work to use it for Transfer Learning [109] as both gesture and non-gesture conditions are present.

Moreover, future work may choose to augment our dataset with additional objects and activities or gestures. A promising area to expand to are rhythmic gestures incorporating a larger temporal duration, or repetitive gestures (e.g., double taps), which indicate benefits such as robust wake-gestures or hot words [54]. It will also be relevant to study objects with advanced material properties, such as pronounced surface texture, friction, or deformability. For reasons of feasibility, our dataset contains gestures performed by Thumb, Index, and Middle fingers. Future work should investigate gestures performed by other fingers. Our dataset is collected using right-handed and young participants. Future work may study how this data generalizes to other populations such as the elderly (potentially limited range of motion, tremor) or children (smaller hands). We have carefully selected different object geometries that afford different orientations of hand and fingers to reduce potential dataset bias. For instance, the thumb faces upwards while holding the book, but it is sideways while holding the bottle. As a next step, future work may use data augmentation techniques to arbitrary facing (or even orientation) of the head by adding randomized orientation offsets to the raw data [97].

Data collection and labeling is a well-known problem in HCI and Machine Learning; the manually-labeled frames in our dataset can provide a quality source for auto-labeling of new data, reducing the tedious manual efforts of data labeling. Finally, it is worthwhile mentioning that our dataset offers a starting point to enable always-available input using IMUs. However, it would be fruitful if future works investigate effortless methods for data collection and labeling in the wild.

In this work, we contributed a tool that assists in rapidly iterating through layout suggestions for IMU placement. We understand our computational design tool’s output not as a final choice, but as a “best guess” for further refinement. For instance, if a layout with multiple IMUs is selected, an inverse kinematics (IK) model could be applied post-hoc to the set of suggested layouts to further leverage the inherent co-activation between the fingers and refine the final layout. Analogously, the current version of our tool comprises F1 score as evaluation criteria, but does not cover other metrics. In cases where robustness against false activation is a key design concern, individually showing precision/recall scores might be beneficial. Likewise, while our tool’s design is relatively easy to use, visually depicting the gestures to instruct new users and strategies for an alternative representation of the confusion matrix and the F1 score can help understand the classification results.

We anticipate that the tool’s layout suggestions can serve as a valuable starting point to quickly reduce the design space and for further improvement of performance in an end-to-end working system. Additional techniques such as collecting more training data to include additional variations, adding more features, performing hyperparameter tuning to tailor the classifier’s behavior to the specific dataset, creating an ensemble of classifiers, and optimizing the hardware’s sample rate to improve the recognition rate can be applied, if desired. Our findings show that grasp-dependent models may further improve the classification performance. This also suggests that the combination of target Freehand and/or Grasp variations affects the model’s performance, where our computational design tool can be useful in rapid testing and iterations to find the balance between users’ choice and classification performance. Currently, our tool suggests sensor placement based on gestures and finger choices. However, future work would include multiple alternatives at the first go or even further, it may work conversely as well, i.e., given the placement choice of sensors and the count, the tool will recommend the best gestures that can be detected. Inspired by Kohlsdorf et al. [49], future versions of the tool may also incorporate techniques to estimate the chances of false positives for each gesture by comparing the selected gestures to a large corpus of everyday activity data. This would facilitate the end-to-end framework for gesture recognition and the practical implications of real-world deployments.

While we performed user-independent evaluations in our analysis, in our initial tests, we found the performance of user-dependent models is higher with the same model architecture. With the advances in deep learning models and their interpretability methods, we believe a more sophisticated model pipeline can be constructed based on our analysis results. This would also help researchers in benchmarking different techniques to select sparse layouts.

Our current layout selections are measured by classification performance, but other factors like the required amount of training data, battery performance, hardware cost, or dimensions of the sensing device could be integrated into future versions of the tool. We also see some possibility that suggestions prove useful beyond their application with IMU data. While there is some uncertainty, other approaches making use of high-dimensional data from different sensors (e.g., EMG/FSR [16, 65, 77]) can potentially expand upon the suggested layouts.