HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands

2.1. Set-up

The set-up for data collection is shown in Fig. 1. A subject was equipped with a headband that held a camera to capture the human eye-view images, and a glove that held a camera to capture the (right-)hand-view images. An object is placed on the table with some arbitrary orientation. With the help of a ruler placed on the table, the subject positioned the hand approximately 12 inches (305 mm) away from the object, with the palm facing it. At this initial position, simultaneous pictures from the eye-view and hand-view cameras were captured. Following an audio cue, the subject started moving the hand towards the object and grasped it as appropriate for the particular object and its orientation, lifted the object from the table briefly, put the object down, and retracted the hand back to the initial pose and position. During this motion, (approximately) synchronous videos from both eye-view and hand-view were captured, along with EMG and IMU. Paired eye-view and hand-view images of four sample objects are shown in Fig. 3.

The initial positions and spatial separation between the object and the hand were selected such that (1) they both were visible in the eye-view image througout the process, and (2) the trial lasted long enough to capture approximately 3 seconds of EMG/IMU data during motion. The procedure was repeated a number of times, each time rotating the object vertically and horizontally, as can be seen in Fig. 1, to capture several possible relative orientations of the object with respect to the hand. The number of orientations varies depending on the shape of each object.

A black background and a white glove were used to simplify the problem of identifying the hand in images and videos. The hand-view images and videos were captured with a GoPro Hero Session camera (3648×2736 pixels). The eye-view images and videos were captured with a Logitech Webcam C600 camera (1600×1200 pixels). The ruler placed on the table for reference had ticks at every half inch. The EMG and IMU measurements from the arm were acquired with a MYO arm-band (Thalmic Labs Inc.) that was placed on the right upper arm of the subject.

The dataset consists of 413 such instances of multimodal data acquired using 102 different ordinary objects, such as office supplies, utensils, stuffed animals and other toys.

2.2. Data Annotation

Unlike object classification problems in computer vision, appropriate grasp type in a given setting may depend on personal preferences of humans as well as other factors. Consequently, for a given eye-view image, there may not be a single correct grasp label. In order to capture this interperson variability, 11 individuals were asked to sort five hand grasp types in decreasing order of preference to grab the object as seen in each eye-view image. The label acquisition process was carried out using an interface developed with Visual Studio 2013, which randomly ordered the images and allowed them to indicate their order of preference for grasp types.

Feix et al. [5] proposed that human grasp taxonomy consists of 33 classes if only the static and stable grasp types are taken into account. The human hand has at least 27 degrees of freedom (DoF) to achieve such a wide range of grasp types; however, most existing prosthetic hands do not have this many DoF. For instance, thumb abduction versus adduction, which is a crucial capability of the human hand, is a challenging robotic design problem, and some robotic hands resort to unpractical approaches like using a manually controlled switch to alternate between these movements [1]. Even contemporary state-of-the-art prosthetic hand designs [22] have much fewer DoF than the human hand, which limits their number of feasible grasp types to 14.

Gunay et al. [9] showed that with appropriate multichannel EMG signals, these grasp types that are feasible for state-of-the-art prosthetic hands can be classified accurately. While EEG alone may not be capable of discriminating a large number of intended grasp types, supplementing EMG evidence with EEG may lead to improved intent inference in the case of some amputations. Ozdenizci et al [19] demonstrated that 5 particular hand gestures (four of them grasp types) can be reliably inferred from EEG, and they provide a reasonable initial coverage of the grasp taxonomy variety established by Feix et al. [5]. Specifically, these 5 grasp types, shown in Fig. 4, are: open palm, medium wrap, power sphere, parallel extension, and palmar pinch. This list includes 3 thumb-abducted, 1 thumb-adducted grasp types, as well as a typical resting state of the hand.

2.3. Data Augmentation and Pre-processing

Data augmentation methods have been successfully used in machine learning, especially computer vision, to promote invariance in models to for certain anticipated variations in input data, such as illumination, sharpness, background clutter, and position of object of interest in the image. In addition to the original 413 hand-view images, the dataset also provides synthetically manipulated versions of these. A total of 4466 synthetic hand-view images were generated by first segmenting the object, then applying a random blurring filter (a square-shaped moving average filter with a randomly selected impulse response length) to the object segment, then randomly translating the object segment in the image, and finally adding to the background an approximately Gaussian distributed noise with zero mean and a random variance for each of the red, green, and blue color coordinates. Some sample synthetic images are shown in Fig. 5. Each synthetic image was assigned one of the human labelers’ top choice for that instance, without replacement (each label was used at most once for augmented images of an instance). For each synthetic image, the bounding box of the object location was also included in the dataset.

2.4. Gesture Classification

In order to illustrate the utility of having paired eye-view and hand-view images, where the appropriate grasp types for a given object/hand initial state were assessed by multiple humans, a CNN classifier was trained to predict the top choice of a human labeler from the hand-view image of the object. CNNs have been very successful in image classification problems [14], and pretrained CNNs can be used to initialize the grasp type classifier, which will be fine-tuned with the available labeled data. CNNs are interpreted as extracting visual features from the input image through their early convolution, pooling, and normalization layers [25]. Specifically, VGG-16 that consists of 16 convolutional layers and about 138 million parameters [26], will form the basis of our CNN grasp type classifier, because it has favorable performance when compared to some other deep neural network models [21,17]. A version of VGG-16 that has been trained on ImageNet [4] object classification dataset, with 1000 object categories, can be found online [8].

The last layer of VGG-16 was replaced by a fully connected layer with sigmoid nonlinearities and 5 outputs; and of the resulting CNN, the last three layers were trained using hand-view images of randomly selecting 80% of the unique objects to predict their associated grasp type labels. The images of the 20% of the unique objects were used for validation. The training procedure used stochastic gradient descent for binary cross-entropy [23] as the loss function. Furthermore, to cope with the overfitting, we added zero padding layers before each convolutional block, and L2 regularization terms were included in the objective function for the weights of every convolutional layer. The process is illustrated in Fig. 6.

2.5. Evaluation

The performance of the CNN was evaluated in terms of both loss function value and grasp type classification accuracy. The loss and accuracy values on both training and validation data sets are shown in Fig. 7. It can be observed that both of these measures show rapid improvement in early epochs and progress slows down significantly before epoch 30. Both loss and accuracy values on training and validation sets are similar, which may indicate the prevention of overfitting. The accuracy of the trained CNN was approximately 89% and its loss value was approximately 0.04. These results demonstrate that the hand-view images are informative about human grasp type intent that is based on eye-view images of the object.

The accuracy achieved for power sphere and medium wrap, 91.2% and 90.5%, respectively, were higher compared to the other grasp types. One possible reason for this could be that more objects are labelled as power sphere or medium wrap (large class priors). Another reason could be the single 80% versus 20% training and validation dataset split was favorable to these class labels. In future work, the dataset could be extended to include objects that lead to a more balanced grasp type class distribution.

3.1. The Usage of the Visual Information

An alternative use of eye-view and hand-view paired images is the use of the CNN to provide a probability distribution for all possible grasp types, based on the hand-view image [11]. Such a CNN could be further extended to process the videos during the approach to the object, in order to generate a sequence of grasp type probability distributions over time. This probability distribution sequence can be fused with EMG and other evidence to obtain a grasp type posterior probability distribution that evolves over time along the hand trajectory. Depth evidence can be extracted from the video using computer vision techniques [13,24], in order to help inform the prosthetic hand motion planner to facilitate appropriate timing of grasp execution, to simulate the potential benefit of having a depth camera built in to the hand.

3.2. The Usage of the Bounding Box

The bounding box of the object in each image can help identify the object for different augmentation procedures. Different CNNs, such as YOLO [21] and SSD [17], can be trained using these images for lighter solutions that facilitate embedded real-time implementations. Additional object coordinate estimators could be trained to help with motion planning and controller actuation in prosthetic hands.

3.3. The Usage of the EMG Data

Even though visual evidence from hand or eye views can provide strong priori expectations about what grasp type the user of a prosthetic hand might execute, ultimately, the intended grasp type of the human cannot be extracted solely based on the shape and orientation of the object in the view. The EMG data from the forearm will provide strong evidence regarding human grasp intent, especially as the hand gets closer to the object, and the muscles contract in anticipation of grasp execution. For instance, power sphere could be the most likely grasp type for a ball according to visual evidence, but the user might have the intent to push the ball rather than grasp it. Access to EMG is essential to distinguish between what grasp is likely a priori based on shape, and what the human intends to do in a specific instance. This dataset enables EMG-video fusion such as what Gigli et al. presented [8].

3.4. The Usage of the IMU Data

Prosthetic hands do not have control over the approach trajectory to the object; therefore, having access to independent kinematic measurements may be useful for motion planning and control when solving the shared control problem from a robotics perspective. The dataset contains 3-dimensional orientation, acceleration, and angular velocity extracted from IMU measurements. Human reach-to-grasp trajectories exhibit tightly coupled behavior between hand velocity and aperture opening, and access to IMU data can help use such dependencies in human reach-to-grasp trajectories [18,12].