Tactile Perception in Upper Limb Prostheses:
Mechanical Characterization, Human Experiments,
and Computational Findings

Alessia Silvia Ivani5

{}^{1,2,3}

, Manuel G. Catalano

{}^{1}

, Giorgio Grioli

{}^{1,2}

, Matteo Bianchi

{}^{2,3}

,
Yon Visell

{}^{4}

, and Antonio Bicchi

{}^{1,2}

*This work was supported by the European Research Council Synergy Grant Natural BionicS (NBS) project (Grant Agreement No. 810346) and by the Italian Ministry of Education and Research (MIUR) in the framework of the FoReLab project and Crosslab project (Departments of Excellence); by PNRR, M4 C2 I1.5 Ecosistema dell’Innovazione ”Tuscany Health Ecosystem (THE)” - Ecosistema dell’innovazione sulle scienze e le tecnologie della vita in Toscana (CUP I53C22000780001) - Spoke 9.

{}^{1}

with Soft Robotics for Human Cooperation and Rehabilitation, Istituto Italiano di Tecnologia, Genova 16163, Italy.

{}^{2}

with Centro di ricerca E. Piaggio, University of Pisa, Pisa 56122, Italy.

{}^{3}

with Department of Information Engineering, University of Pisa, Pisa 56122, Italy.

{}^{4}

with Department of Mechanical Engineering and Bioengineering Program, University of California, Santa Barbara, USA. 5 Corresponding author alessia.ivani@iit.it

Abstract

Our research investigates vibrotactile perception in four prosthetic hands with distinct kinematics and mechanical characteristics. We found that rigid and simple socket-based prosthetic devices can transmit tactile information and surprisingly enable users to identify the stimulated finger with high reliability. This ability decreases with more advanced prosthetic hands with additional articulations and softer mechanics. We conducted experiments to understand the underlying mechanisms. We assessed a prosthetic user’s ability to discriminate finger contacts based on vibrations transmitted through the four prosthetic hands. We also performed numerical and mechanical vibration tests on the prostheses and used a machine learning classifier to identify the contacted finger. Our results show that simpler and rigid prosthetic hands facilitate contact discrimination (for instance, a user of a purely cosmetic hand can distinguish a contact on the index finger from other fingers with 83% accuracy), but all tested hands, including soft advanced ones, performed above chance level. Despite advanced hands reducing vibration transmission, a machine learning algorithm still exceeded human performance in discriminating finger contacts. These findings suggest the potential for enhancing vibrotactile feedback in advanced prosthetic hands and lay the groundwork for future integration of such feedback in prosthetic devices.

I INTRODUCTION

Refer to caption — Figure 1: Impact response analysis of a prosthetic-shaped system modelled as a continuous body to an impulse on each fingertip. Acceleration propagation is shown. Grey circles highlight the differences in acceleration propagation depending on the finger contacted. The prosthetic system has a constraint on the end surface of the socket. Simulations are performed on Creo Parametric.

The loss or absence of a hand deprives a person of multiple functions that drive interaction with the outside world. Those obliviously include motor functions, communications, socialization and, prominently, sensory functions. Most commercial prostheses do not provide a substitute for sensing despite recent research efforts and technological improvements in haptics. Bensmaia et al. [1] present a comprehensive review of the current invasive and non-invasive methods and challenges encountered in developing sensory feedback systems for bionic hands. There is experimental evidence that non-invasive upper limb sensory feedback prostheses would improve embodiment [2, 3] and reduce phantom limb pain [2, 4]. Nevertheless, the discrepancy between the several feedback solutions proposed in the literature [1, 5] and the few commercially available sensorized bionic hands [6] is evident because very few provide a substitution for cutaneous feedback [1, 6]. One of the main technological difficulties behind that limit is the integration of sensors and actuators, which can compromise wearability and usability [7]. Another fundamental obstacle is understanding how to provide relevant feedback that does not require all of a user’s attention. Indeed, results about the effect of cutaneous feedback on prosthesis control performance are contradictory. Some highlight improvements, whereas others find no differences [8, 2]. The challenge consists of balancing the quantity of information to be communicated. Indeed, the execution of everyday tasks should provide meaningful information without confusing or increasing the conscious attention effort required to interpret signals [9]. It should also be noted that the solutions proposed in the literature are evaluated in research laboratories rather than in daily living tasks, where the user has to manage significantly more exteroceptive and proprioceptive information [7].

The propagation of cutaneous feedback in the human hand is naturally related to high-frequency vibrations which propagate in the skin away from the original point of contact [10]. During tool-extended sensing, Miller et al. [11] proved that a rod mechanically transduces impact location into vibratory patterns decoded by the somatosensory system. The mechanoreceptors in the human hand transduce the vibratory cues into neural response patterns that preserve the location-specifying information [11]. Thus, the authors provide evidence of how handheld tools function as sensory extensions of the human’s body. Likewise, as pointed out by Farina et al. [9], a prosthetic user has natural sensory feedback in addition to vision and, depending on the type of prosthetic device, can perceive proprioceptive information from residual muscles, hear and feel motor actuation, and perceive vibrations conveyed through the prosthetic socket. Thus, for prosthetic users, vibrations patterns are important tactile information usually felt on the stump and partake in users’ strategies to compensate for the lack of sensory information [7, 12].

Despite the possibility of providing insight into prosthetic users’ baseline perceptions and the implications for prosthetic design, limited research has been conducted to investigate and characterize the natural transmission of tactile signals through prosthetic devices. An evaluation of the threshold level for vibrotactile cues detection of socket-prosthetic limbs with respect to bone-anchored prostheses is performed in [13]. The authors demonstrate that bone-anchored and socket-prosthetic limbs can activate stump-level receptors upon vibratory stimulation of the prosthetic thumb. As expected, bone-anchored prostheses promote vibration perception more than socket-based prosthetic devices. Furthermore, according to the recent study by Amoruso et al. [14], the existence of intrinsic somatosensory feedback by artificial body parts has been demonstrated to be crucial in enabling precise motor commands.

In our experience, meaningful texture information is transmitted through socket-based prosthetic devices [15, 16]. During empirical observations, users exhibited the capability to discern minor tactile interactions on the digits of their prosthetic limb and, notably, to reconstruct the specific finger that had been contacted with a degree of reliability significantly exceeding random chance. We speculate vibrations to play an important role in this phenomenon. Indeed, studying the impact response analysis of a generic prosthetic-shaped system, different acceleration patterns arise when a contact cue stimulates different fingers (please see Fig.1 for impact response analysis results). These accelerations reach the other side of the prosthetic socket and may constitute a crucial component of prosthetic user baseline perception. However, considering the complexity of bionic hands, it is evident that vibration transmission of tactile cues differs depending on the point of contact and the shape of the prosthesis. Moreover, the mechanical characteristic of a given prosthesis is also expected to affect the propagation of vibrations and, therefore, the intensity and quantity of cues. The factors that we expect to affect such a phenomenon are the presence of articulation and the stiffness of the prosthetic hand. Thus, if the type of prosthesis demotes acceleration propagation, a possible enhancement of the natural vibration transmission would be beneficial for tactile feedback perception and for enabling prosthetic devices to function as a sensory extension. However, to the best of the authors’ knowledge, no study ever investigated in detail how and to what extent these baseline perceptions occur in socket-based prosthetic devices nor provided a quantitative assessment of the roles of the influencing factors.

This study examines the perception and transmission of vibratory cues through four prosthetic hands with different mechanical characteristics (Fig. 2). Specifically, we compare hands in a range from very simple, with lower degrees of freedom (DoF) and rigid mechanisms, to more sophisticated devices with an increasing number of articulations and soft mechanical designs. We first report on a study with a prosthetic user, assessing her capability to discriminate which finger is touched based only on vibration patterns passively transmitted by different hands through the same socket to a user’s stump. Then, we quantitatively characterize the phenomenon of mechanical vibration transmission of the four prosthetic hands in terms of impact forces and socket-level vibration generated by impacting them with a test machine on the five fingers. To do so, we design and place a set of vibration sensors in the socket. Our aim is to correlate those data with user perceptual performance and the different hand designs. Finally, we examine the information content of the acceleration signals recorded at the stump level to determine whether acceleration data can be used to improve tactile perception and partially compensate for the sensory loss in the softer, more articulated hands. In other terms, we ask ourselves whether the signals retrieved by the vibration sensors on the socket contain enough information to enable softer, more articulated hands to the same level of discrimination that a user exhibits with simpler, rigid hands. The paper is structured as follows. The prosthetic hands are presented in Section II. Sections III-V includes the experimental methods. Results are presented and discussed in Section VI and Section VII. Section VIII concludes the paper.

II Materials: Prosthetic Hands

We tested four bionic hands, representative of the most common mechanical and kinematic characteristics of the currently available prosthetic hands. The four prosthetic hands selected for the experiments are (Fig. 2):

•

a Cosmetic hand (CH) by Ottobock¹¹1Cosmetic hand by Ottobock, [Online], Available: https://www.ottobockus.com/prosthetics/upper-limb-prosthetics/solution-overview/custom-silicone-prosthetics/;
•

a MyoHand VariPlus Speed (VP) by Ottobock²²2MyoHand VariPlus Speed by Ottobock, [Online], Available: https://www.ottobock.com/en-us/product/8E385~59 (1 DoF);
•

a I-Limb Access (IL) by Össur³³3 I-Limb Access by Össur, [Online], Available: https://www.ossur.com/en-au/prosthetics/arms/i-limb-access (5 DoF);
•

a SoftHand Pro (SH) [17] (19 DoF).

The four hands exhibit an increasing complexity in terms of articulations and DoF. The Cosmetic hand is a passive hand mainly made of plastic with a silicone glove. The VariPlus is a tridigit myo-electrically controlled hand, with a single actuated degree of freedom opposing a thumb to two fingers, with two passive fingers and a five-finger glove. The three active fingers (index, middle, thumb) are rigid, made with metallic components, and mechanically coupled (i.e. close simultaneously), and the remaining two follow passively. The I-Limb is a myo-electrically controlled hand with metallic components and a soft silicone glove with thicker tips. The five fingers are individually powered. Each finger, except for the thumb, has a small mechanical play on the hand sagittal plane (see Fig. 2), while they are rigid in the other directions. The thumb has a transverse mechanical play. The SoftHand Pro is mainly made of plastic components with a silicone glove. A system of elastic ligaments connects the different phalanxes and makes the hand adaptable to all objects, especially in the sagittal hand plane. At the same time, each finger has a small mechanical play in the coronal plane. The mechanical structure of the fingers of the four test hands represents four of the main types of finger joints [18]: the rigid joint (found in the index, thumb and middle fingers of VP), the continuous joint (in CH, and ring and little fingers of VP), the flexible joint (in IL) and the dislocatable joint (in SH).

III Tactile Feedback Perception Experiment

We investigate the tactile feedback perception of a prosthetic user. The ability to discriminate prosthetic fingers during impacts on each fingertip is assessed. First, a preliminary experiment is conducted with the participant’s own cosmetic prosthesis and integrated socket. Then, four experiments are performed with the four specimen hands mounted on a reference socket.

III-A Participant

One prosthetic user takes part in the experiments and gives her written, informed consent to participate. The participant (43 years old, female) is affected by limb agenesis and is used to wearing a cosmetic prosthesis even if having experience with myoelectric hands. The subject has no cognitive impairment that could affect the ability to follow the instructions of the study. Approval of all ethical and experimental procedures and protocols was granted by the Local Ethics Committee of Area Vasta Nord-Ovest (CEAVNO), Tuscany, Italy, under Protocol No. 7803.

III-B Experimental Setup

The subject is comfortably seated on a chair in a quiet room. When the experiment starts, white acoustic noise is provided via headphones to mask any sound from the contacts. The participant is also blindfolded using a pair of goggles with obscured lenses. Different positions and supports are tested to select the most comfortable position for the participant and avoid contact transmission being affected by the supporting structure. When the arm is freely suspended, the participant experiences discomfort and is unable to maintain a stable arm position throughout the experiments. Using a supporting structure for the residual limb prior to the prosthesis device results in compression of the residual limb, which may cause discomfort or pain. Furthermore, completely supporting the prosthetic arm, such as laying it on a table, compromises impact transmission. Thus, the final design adopts a supporting structure of loose Velcro loops on which the prosthetic arm is positioned (see Fig. 3). By placing the support on the prosthesis instead of a user’s skin, the arm position can be kept for all the experiments, and impact absorption is reduced due to the presence of only two Velcro loops.

The same reference socket is used for the four hands selected to ensure that the sole differences arise from the different hands used. In the preliminary experiment, the subject’s own cosmetic hand and the integrated socket are tested. The subject’s cosmetic hand is a CH hand without a quick disconnect wrist. Tactile cues are delivered with a small hammer ( $\sim$130\text{\,}\mathrm{g}$$ ) on the prosthesis fingertips, perpendicular to the coronal hand plane (see Fig.2). The experimenter underwent training to ensure consistent contacts delivery.

III-C Experimental Methods

A preliminary experiment with the participant’s cosmetic prosthesis is carried out to choose the proper number of contacts to deliver. The daily use of the prosthesis is expected to promote the recognition of the different fingers contacts. The goal is to reach a significant number of impacts before the loss of motivation and concentration. Based on the results obtained with the subject’s own prosthesis, the number of impacts to be delivered to each finger is set to 20.

Each experiment (preliminary experiment included) consists of a familiarization phase, a test phase, and rest phase. In the familiarization phase, the subject experiences five contacts on each finger without sound and visual insulation. During the test phase, the subject is isolated. Twenty contacts for each finger are performed with the small hammer. The order of the finger to be contacted is randomized to prevent the subject from guessing the next one. After each impact, the participant is asked to answer the question: “Which finger was contacted?”. The actual finger contacted and the corresponding answer from a user are saved for analysis. A rest phase of about fifteen minutes is done between one experiment and the other. The experiment is repeated four times, one for each of the prosthetic hands. The order of the specimen is randomized. The participant is informed that the experiment can be interrupted at any time. At the end of the experiment, the participant undergoes through a subjective evaluation procedure based on a brief questionnaire about prosthetic finger contact perception depending on the prosthetic hand used (Table I).

\MFUnocap

for\MFUnocapthe\MFUnocapof\MFUnocapand\MFUnocapin\MFUnocapfrom Questions Q1 Please briefly describe what you feel when each of the four prosthetic hands contacts a rigid surface. Q2 When the prosthetic hands are contacted, how aware are you of which part of the hands, such as a finger, has been contacted? Please comment.

TABLE I: \capitalisewords questionnaire.

III-D Data Analysis

Participant’s prediction performance for each hand determines the four scores: $TP$ (True Positives counts), $TN$ (True Negatives counts), $FP$ (False Positive counts) and $FN$ (False Negative counts). Those scores are further analyzed using a confusion matrix, and the three metrics accuracy, recall, and precision [19]. Accuracy, defined as:

Accuracy=\frac{TP+TN}{TP+TN+FP+FN},

(1)

provides an estimation of the correct predictions. Recall, defined as:

{Recall=\frac{TP}{TP+FN}},

(2)

tells how frequently the subject can detect a specific category. Precision measures what percentage of all the positive predictions is truly positive, and it is defined as:

Precision=\frac{TP}{TP+FP}.

(3)

All the metrics are compared to the chance level

CL=\frac{100\%}{K}=\frac{100\%}{5}=20,

(4)

where $K$ is the number of classes (thumb, index, middle, little, and ring). The accuracy [19] considering only the ability to discriminate the thumb finger with respect to the others and the index finger compared to the other fingers are also computed.

IV Tactile Feedback Transmission Experiment

The four hands are investigated to characterize and quantify how the transmission of high-frequency stimuli is affected by the type of prosthesis used. Each hand’s impact responses to quantified contacts are assessed. During impacts, the accelerations inside the socket and impact forces of each fingertip are recorded with Inertial Measurement Units (IMUs) and a sensorized pendulum, respectively (see Fig.4, 5).

IV-A Experimental Setup

The four hands are mounted on the reference socket. The same supporting structure as in Section III is used. We attach five IMUs (MPU-9250) to the inner surface of the socket in a radial distribution to measure the differences in vibration patterns after each contact (Fig. 4, 5). This choice is grounded upon the outcomes of the impact analysis performed on a generic prosthetic-shaped system modelled as a continuous body (as shown in Fig. 1). Although the analysis simplifies the system compared to a real prosthetic system, it demonstrates that the same contact on each finger generates distinct acceleration patterns that reach the other side of the prosthesis. Furthermore, IMUs’ position and distribution are also justified by the physical constraints of the residual limb of the participant and its interfacing with the socket.

A sensorized pendulum (see Fig. 4) is used by the experimenter to contact the prostheses fingertips with repeatable impacts. As in the first experiment, each contact is perpendicular to the coronal hand plane (Fig. 2). The pendulum can be moved and adjusted to contact a particular finger in a given position. Before each impact, the pendulum arm is aligned to an adjustable end-stop to hold it at a desired angle. Then, the pendulum is released to impact the prosthesis. During each impact, a load cell (ATI Nano17 Transducer) at the lower extremity of the pendulum measures the impact forces while an encoder (AS5045) at the upper extremity the angles.

IV-B Experimental Method

In preliminary tests, the pendulum angle that can create an impact without reaching IMUs full scale is determined and kept for all experiments ( $3 °$ ). During impacts, acceleration signals inside the socket and impact forces at each fingertip are recorded through electronic boards [20]. A custom C++ software is developed to synchronize, start, and stop the recordings. The experiment starts when the pendulum is set at the predetermined angle through the encoder and the end-stop. The pendulum is left free to swing, and the load cell surface contacts the artificial fingertip. After each contact, the prosthetic limb oscillates. From the load cell, it is possible to record the three components of the impact force $F=(F_{x},F_{y},F_{z})$ , while from each IMU inside the socket, the three components of the acceleration $a=(a_{x},a_{y},a_{z})$ . The recording session is done at $\sim$1\text{\,}\mathrm{k}\mathrm{H}\mathrm{z}$$ to ensure that each impact is properly captured and within the limits of the hardware used to register. The experiment ends when the pendulum is manually stopped after adequately completing the impact. Since the recorded signals have proven to be repeatable, three strikes for each finger are performed, and the best one is saved for the subsequent analysis. The experiment is repeated for each robotic hand.

IV-C Data Analysis

After the experimental session, all signals are analyzed and processed. The impact force component $F_{z}$ is selected as the main contribution of the contacts because the pendulum swings in the same direction. As regards acceleration signals, various processes are carried out to analyze high-frequency signals only. Outliers are removed from the three components $a_{x},a_{y},a_{z}$ . Then, a high-pass filter is used to remove the low-frequency oscillation of the prosthesis caused by the contacts. Forces and high-frequency accelerations are cut to isolate the contact. Each acceleration signal is cut with respect to the impact force peak. The contact is accurately represented by three hundred samples ( $\sim$0.28\text{\,}\mathrm{s}$$ ). The next step is assigning an energy value to each signal to compare each bionic hand’s results quantitatively. With the algorithm DFT321 [21], the three components $a_{x},a_{y},a_{z}$ are then reduced into a one-dimensional signal $A$ with the same energy as their sum (Fig. 6). The DTF321, as part of the algorithms that seek the spectral difference, is considered by Lee at al. [22] one of the best 321 approaches for offline processing when perceptual similarity is prioritized.

Finally, the acceleration signal energy of each finger is computed for each IMU to compare the results of the prosthetic hands. The overall mean of each IMU energy signal and fingers is computed for each hand. Thus, a mean energy value is obtained for the four prosthetic hands. To compare results, Spearman rank correlation analysis is computed between the energy mean of each hand and the accuracy of the Tactile Feedback Perception Experiment. The impact force responses of the four hands are also compared.

V Tactile Feedback Recognition Experiment

To investigate whether acceleration data contain sufficient information to identify which finger is contacted, Artificial Neural Networks (ANNs) are trained. Due to their capability of handling time series data, Long-Short-Term Memory (LSTM) [23] models are used for finger detection and trained for each bionic hand. New acceleration signals inside the socket are recorded upon impacts on each fingertip to create large and balanced datasets.

V-A Datasets

New experiments are carried out to obtain four datasets, one for each robotic hand. The prosthetic hands with the reference socket are set by the same supporting structure in Section III and IV. The five IMUs are kept on the inner surface of the socket to measure the vibration pattern after each contact. The small hammer is used to contact each fingertip perpendicular to the coronal hand plane (Fig.2). In preliminary experiments, hammer impacts are tested to determine if the IMUs’ full scale is reached. Then, approximately the same impacts are manually delivered in all experiments. The experiments consist of one hundred impacts on each fingertip. The three acceleration components $a=(a_{x},a_{y},a_{z})$ are recorded from the five IMUs inside the socket during each impact. Then, the same data processing in Section IV-C is carried out to obtain the one-dimensional signal $A$ . Thus, four datasets are obtained with five hundred samples each, one hundred for each fingertip. Each sample comprises five one-dimensional signals $A$ from the five IMUs.

V-B LSTM Network Architecture and Hyperparameters

As the purpose of this experiment is to determine whether finger contact information can be extracted from acceleration data, various types of classification algorithms may be appropriate. LSTM networks are a particular type of Recurrent Neural Network (RNN) able to process entire data sequences and selectively remember information, widely used to capture time correlations efficiently [24, 23]. To avoid a separate feature extraction step, we choose LSTM over linear support vector machines (SVMs). We train four LSTM models to recognize the finger contacted from the acceleration signals $A$ of each IMU. Each dataset is randomly split into 80% training, 10% validation and 10% testing sets. We experimentally test different network parameters and hidden layers to evaluate the best performance. The four nets are composed of a first feed-forward layer with ReLU activation function and a second normalization layer which normalizes all five features together (the five signals from the IMUs). Then, the third layer is an LSTM layer. The output is fed to a dropout layer. The four layers repeat, and, lastly, the softmax layer gives the probability of each class (thumb, index, middle, ring, little), and the classification layer computes the cross-entropy loss. We set the dropout to 0.2. The hyperparameters are tuned by using the Bayesian optimization algorithm in MATLAB (MathWorks Inc., Natick, MA, USA), maximizing the validation accuracy on the validation sets. The hyperparameters selected are the initial learning rate, the number of units in a dense layer, the number of hidden units, the number of epochs, and the batch size.

V-C Data Analysis

Test sets are used to evaluate the performance of the four networks. Four confusion matrices with True Class and Predicted Class are computed. The results are compared in terms of accuracy, recall and precision metrics [19].

VI Results

VI-A Tactile Feedback Perception Experiment

We assessed the finger discrimination ability of the subject wearing CH, VP, IL and SH bionic hands. The subject’s cosmetic hand was also tested in the preliminary experiment. With the subject’s own prosthesis, the accuracy and mean recall reached $55\%$ and $58\%$ , respectively. Results of the experiments with the four specimens are shown in Table II. The accuracy related to the CH hand is $58\%$ while VP and IL accuracies are $52\%$ and $45\%$ , respectively. With SH, the accuracy reaches $37\%$ . The subject was able to discriminate the finger contacted with an accuracy that is always above the chance level. Mean precision and mean recall were also computed for each hand. Since the same number of contacts were delivered for each finger, the results are balanced, and the test accuracy also corresponds to the mean recall [19].

\MFUnocap

for\MFUnocapthe\MFUnocapof\MFUnocapand\MFUnocapin Bionic Hands Accuracy[%] Mean Precision[%] Thumb Recognition Accuracy[%] Index Recognition Accuracy[%] Cosmetic 58 62 81 83 VariPlus 52 63 82 65 I-Limb 45 49 93 72 Soft-Hand 37 45 61 55

TABLE II: \capitalisewordsSubject accuracy and mean precision of the Tactile Feedback Perception Experiment for each bionic hand.

All the mean precision results are higher than the mean recall metrics.

If it’s considered only the ability to discriminate the thumb finger with respect to the others, positive results are obtained for the accuracy [19] of each hand: $81\%$ for CH, $82\%$ for VP, and $93\%$ and $61\%$ for IL and SH, respectively. Regarding index recognition accuracy compared to the other fingers, the results are: $83\%$ for CH, $65\%$ for VP, $72\%$ for IL and $55\%$ for SH.

In Fig. 7, confusion matrices of the four bionic hands are shown.

Each row summary and column summary displays the percentage of correctly classified and incorrectly classified observations for each true or predicted class. In particular, the first column summary is the recall metric of each class, while the first row summary is the precision metric of each class. About the VP hand, higher accuracy and recall are highlighted for the thumb, the index and the middle fingers, which are more rigid with respect to the other fingers. This could be due to the presence of the three metallic fingers under the glove. A $95\%$ of recall and a $73\%$ of precision for the thumb of IL hand reflects the lower mechanical play in that direction of impact with respect to the other fingers. With the SoftHand, the overall discrimination ability of the subject decreased. The worst recall metric was achieved with the little finger in all hands, while precision is quite high for the same finger. Thus, the subject was rarely able to detect the little finger, but most of the predictions were correct.

Based on the questionnaire, it emerged that the subject feels a vibration on the inner socket of the prosthetic device, which is smooth with the SH and more evident with the CH, and can attribute the vibration to a contact zone.

VI-B Tactile Feedback Transmission Experiment

We recorded the acceleration signals from IMUs glued inside the socket and the impact forces of the pendulum, which contacts the prosthetic fingertips. Fig. 8 shows the different impact forces of the sensorized pendulum with the index finger of each bionic hand.

It can be noticed that the impact forces of the index finger of less compliant hands (as VP) are intense and shorter in time. In such a case, the impact between the pendulum and the prosthesis is almost instantaneous, and the energy is, thus, quickly transmitted. With more compliant hands, the response is less intense and slowly transmitted.

The bar charts in Fig. 9 show the computed energy of the one-dimensional signals $A$ for each IMU.

IMUs’ numbers refer to Fig. 5. Since the socket is always the same, transmission differences depend on the type of hand used. The energy associated with socket accelerations is higher when the rigid VariPlus and Cosmetic hands are worn (especially in the metallic fingers of VP) and decreases with the other hands. Indeed, high-frequency signals are better transmitted in less compliant and less articulated hands. Since the IL thumb is stiffer in the direction of impact, its energy results are greater than the other fingers. For the softer and more articulated hands (SH and IL), the energy is much lower for each IMU. In SH, IMUs energy values are also similar between a finger and another. Indeed, it has several dampening elements, such as the wrist, which prevent vibration transmission through the prosthetic device.

The behaviour of the five IMUs in response to contacts is slightly different from each other in all fingers. This can be noticed especially in the more rigid and less articulated fingers.

The Spearman rank coefficient between the mean energy value of each hand and the accuracy of Tactile Feedback Perception Experiment was 0.8. Thus, a strong monotonic association was found.

VI-C Tactile Feedback Recognition Experiment

We trained four LSTM networks to recognize the fingers contacted based on acceleration data recorded inside the socket, one net for each bionic hand. Nets’ hyperparameters were tuned using Bayesian optimization. Table III shows the hyperparameters tuning results, and Table IV shows validation and test accuracy results and mean precision metrics for the prosthetic hand used.

\MFUnocap

for\MFUnocapthe\MFUnocapof\MFUnocapand\MFUnocapin\MFUnocapfrom Hyperparameters LSTM ${}_{VP}$ LSTM ${}_{CH}$ LSTM ${}_{IL}$ LSTM ${}_{SH}$ Initial Learning Rate 0.0011 0.0016 0.0027 0.0033 Epochs 68 168 185 82 Units in a Dense Layer 40 37 21 18 Hidden Units 40 40 39 39 Batch Size 64 32 64 64

TABLE III: \capitalisewordsNets’ hyperparameters resulted from tuning the four training and validation sets in the Tactile Feedback Recognition Experiment.

Since all the datasets were balanced, the test accuracy results also correspond to the mean recall [19].

\MFUnocap

for\MFUnocapthe\MFUnocapof\MFUnocapand\MFUnocapin\MFUnocapfrom Net Validation Accuracy[%] Test Accuracy[%] Test Mean Precision[%] LSTM ${}_{CH}$ 84 78 79 LSTM ${}_{VP}$ 96 86 86 LSTM ${}_{IL}$ 88 88 89 LSTM ${}_{SH}$ 76 78 78

TABLE IV: \capitalisewordsNets’ validation and test accuracy in the Tactile Feedback Recognition Experiment.

All networks were able to detect the finger contacted with positive results. Validation and test accuracy of the LSTM model trained on the CH socket acceleration data reached $84\%$ and $78\%$ , while on VP data reached $96\%$ and $86\%$ , respectively. For SH, the validation and test accuracy are $76\%$ , $78\%$ , and for the IL, they both are $88\%$ . Mean precision and recall are also balanced. Accuracy, recall and precision metrics are all higher than the Tactile Feedback Perception Experiment results.

Confusion matrices of the four LSTM networks evaluated on test sets are shown in Fig. 10.

From the confusion matrices, it should be noted that each class was well predicted from all LSTM models. Precision and recall metrics are also balanced for each class. The overall performances decrease for the SoftHand LSTM model.

VII Discussion

The Tactile Feedback Perception Experiment proved that the subject was able to perceive and recognize the contact on all the bionic hands. The subject detected the finger contacted with positive results. Indeed, the accuracy-related to all the bionic hands is much higher than the chance level of 20%. Despite the daily use of the subject’s cosmetic prostheses, better results were achieved with the CH hand and the reference socket. This might be explained by the reference socket used in the experiments being tighter than the user’s one. Also, the material of the reference socket is stiffer than the personal one and could have affected the perception. Thus, vibrations may be better transmitted to the subject. Results prove that the subject learns, if unknowingly, to associate the transmitted vibration patterns to a finger position with better outcomes for rigid and less articulated hands. Indeed, wearing the Cosmetic and the VariPlus hands, accuracy metrics are 58% and 52% with respect to 45% and 37% of the I-Limb and the SoftHand bionic hands. Regarding the index finger recognition compared to the other fingers, the subject achieved impressive results, with higher accuracy for the CH hand (83%) than VP and IL (65% and 72% accuracies, respectively). With the SH, only a 55% accuracy was achieved, slightly higher than the chance level. Considering the discrimination ability to identify a contact on the thumb from an impact on the other fingers, the higher recognition accuracy for the I-Limb (93%) compared to the CH (81%) and the VP(82%) can be explained by considering that the I-limb thumb has a transverse mechanical play. Thus, during the impact, set on the hand sagittal plane, the thumb is more rigid compared to the other fingers and may make it easier for the subject to detect high-frequency vibration associated with that finger compared to the CH and VP. The SoftHand decreases the performance (accuracy of 61%), but still with a result above the chance level. Obviously, the findings are subjective, and further studies are needed to confirm them for subjects with different amputation types and levels. Moreover, different types of prosthetic impacts and impact directions could be further investigated. The experimental materials affected the results obtained, but we successfully highlighted the differences between the various hands by keeping the same socket.

In the Tactile Feedback Transmission Experiment, we have effectively quantified impacts at the socket level and studied the response of each hand. The results quantitatively prove that softer and more articulated hands dampen contacts while the hands with stiff components and less articulated better transmit high-frequency vibrations. Indeed, more intense accelerations were detected at the socket level for the more rigid hands with respect to the softer ones. The findings reflect the Tactile Feedback Perception Experiment results despite being carried out under different conditions. In the presence of IMUs, the subject could not wear the prosthesis, and the vibration transmission behaviour is different with respect to when the arm is present. Nevertheless, hanging the prosthesis enabled us to characterize each robotic hand and understand how vibration is transmitted regardless of the person wearing the prosthesis. Obviously, when comparing the stiffness differences between a human arm and the air, it is evident that the air-prosthesis connection influenced vibration transmission. Thus, the differences in the experimental setup might have contributed to the lower classification accuracies observed in the Tactile Feedback Perception Experiment. We acknowledge this as a limitation of the study. Future research will address this concern by exploring methods to perform measurements while a prosthetic user wears the socket. Nevertheless, the Spearman rank coefficient of 0.8 demonstrates that user perception and measured accelerations inside the socket are monotonically correlated. The hands with lower energy socket vibrations are those with lower accuracy results and vice versa.

In the Tactile Feedback Recognition Experiment, we proved that LSTM models are able to detect the finger contacted from the high-frequency socket acceleration signals with quite a high accuracy for all hands. Thus, the recorded vibratory patterns contain the information necessary to discriminate the impacted finger, even if registered on the inner surface of the socket. The discrimination ability of ANNs is substantially better than the subject’s one in all prosthetic hands. For acceleration dimensional reduction we used the DTF321 algorithm in both Tactile Feedback Recognition Experiment and Tactile Feedback Transmission Experiment. As mentioned, the DTF321 algorithm is considered one of the best 321 approaches for offline processing when perceptual similarity is prioritized [22]. Still, DTF321 is not rotation invariant, and a different orientation of the IMU yields different DFT321 values even if the signals captured by the IMU are identical in the new orientation of the sensor [21, 22]. Thus, future online implementation will consider the Principal Component Analysis (PCA) algorithm to avoid orientation dependencies [22, 25]. Nevertheless, as a result of our findings, the recorded acceleration signals and ANNs have the potential to compensate for the loss of tactile perception given by the softer and articulated hands. Based on the evidence we provided, the subject already knows how to relate those vibrations to the corresponding finger contacted, especially with rigid hands. Although prosthetic hands with soft and compliant components are adaptable, intuitive to use and more natural in the execution of activities of daily living with respect to rigid hands [26, 27], they do reduce vibration transmission, demoting perception. Thus, enhancing socket vibration transmission or replicating the acceleration signals associated with rigid hands could be a possible solution to improve soft prosthesis finger discrimination performance. Future studies will investigate the use of vibrotactile actuators associated with ANNs to intensify natural high-frequency vibration transmission and perception of tactile cues.

The overall results prove that the characteristics of the prosthetic hand affect users’ perception. Even without a feedback system, the subject was able to perceive impacts and discriminate them with an accuracy higher than the chance level. With softer and more articulated hands, however, accuracy, recall, and precision metrics indicate a lower user’s finger discrimination ability. Softer prosthetic hands provide fewer cues to human discrimination of tactile feedback, but artificial enhancement of vibrotactile feedback is feasible with ANNs trained on socket acceleration data. The findings of this study provide a foundation for new prosthetic design and engineering for artificial feedback. By recognizing prosthetic contact cues with ANN with sensors that are not attached to the prosthetic hand, the integration of a haptic feedback device may become - at least theoretically- feasible, independent of the prosthetic hand used, while not compromising the prosthesis’ wearability or usability. As a further implication, actuators that are not directly attached to the skin could be used to intensify socket vibration, thus preventing any discomfort to a user. Finally, by enhancing the natural transmission of tactile cues, the feedback could be more intuitive and easy to interpret with respect to state-of-the-art non-invasive artificial feedback solutions.

VIII Conclusions

This work successfully tested tactile perception and transmission in four prosthetic hands with different mechanical and kinematics characteristics. The results proved that a prosthetic user perceived tactile stimuli with the four bionic hands without a haptic system. Interestingly, finger discrimination performances were all higher than the chance level, with decreased performances for the softer hands. The four bionic hands were characterized by impact forces and high-frequency stimuli transmission through the socket upon impacts with each finger. It has been quantitatively shown that rigid hands promote tactile transmission while more articulated hands are softer and absorb impacts. These features are often appreciated by users, but they do reduce finger discrimination ability. We have shown that ANNs trained on signals from vibration sensors on the socket can recover maximum finger discrimination ability even with softer articulated hands. The results of this work encourage us to design a new intuitive haptic feedback device able to intensify prosthetic natural vibration transmission performances.

ACKNOWLEDGMENT

The authors would like to thank Manuel Barbarossa, Mattia Poggiani, Marina Gnocco and Emanuele Sessa for their valuable support in the experiments.

References

[1] S. J. Bensmaia, D. J. Tyler, and S. Micera, “Restoration of sensory information via bionic hands,” Nature Biomedical Engineering, pp. 1–13, 2020.
[2] U. Wijk, I. Carlsson, C. Antfolk, A. Björkman, and B. Rosén, “Sensory feedback in hand prostheses: A prospective study of everyday use,” Frontiers in Neuroscience, vol. 14, p. 663, 07 2020.
[3] P. D. Marasco, K. Kim, J. E. Colgate, M. A. Peshkin, and T. A. Kuiken, “Robotic touch shifts perception of embodiment to a prosthesis in targeted reinnervation amputees,” Brain, vol. 134, no. 3, 2011.
[4] C. Dietrich, K. Walter-Walsh, S. Preißler, G. O. Hofmann, O. W. Witte, W. H. Miltner, and T. Weiss, “Sensory feedback prosthesis reduces phantom limb pain: proof of a principle,” Neuroscience letters, vol. 507, no. 2, 2012.
[5] M. N. Nemah, C. Y. Low, O. H. Aldulaymi, P. Ong, A. A. Qasim, et al., “A review of non-invasive haptic feedback stimulation techniques for upper extremity prostheses,” International Journal of Integrated Engineering, vol. 11, no. 1, 2019.
[6] A. Akhtar, J. Cornman, J. Austin, and D. Bala, “Touch feedback and contact reflexes using the psyonic ability hand,” 2020.
[7] J. Schofield, K. Schoepp, J. Carey, and J. Hebert, “Applications of sensory feedback in motorized upper extremity prosthesis: A review,” Expert review of medical devices, vol. 11, pp. 1–13, 06 2014.
[8] J. W. Sensinger and S. Dosen, “A review of sensory feedback in upper-limb prostheses from the perspective of human motor control,” Frontiers in neuroscience, vol. 14, 2020.
[9] D. Farina and S. Amsuss, “Reflections on the present and future of upper limb prostheses,” Expert review of medical devices, vol. 13, 02 2016.
[10] Y. Shao, V. Hayward, and Y. Visell, “Compression of dynamic tactile information in the human hand,” Science Advances, vol. 6, no. 16, 2020. [Online]. Available: https://www.science.org/doi/abs/10.1126/sciadv.aaz1158
[11] L. E. Miller, L. Montroni, E. Koun, R. Salemme, V. Hayward, and A. Farnè, “Sensing with tools extends somatosensory processing beyond the body,” Nature, vol. 561, no. 7722, pp. 239–242, 2018.
[12] G. Lundborg and B. Rosen, “Sensory substitution in prosthetics,” Hand clinics, vol. 17, no. 3, 2001.
[13] R. Jacobs, R. Brånemark, K. Olmarker, B. Rydevik, D. v. Steenberghe, and P.-I. Brånemark, “Evaluation of the psychophysical detection threshold level for vibrotactile and pressure stimulation of prosthetic limbs using bone anchorage or soft tissue support,” Prosthetics and orthotics international, vol. 24, no. 2, 2000.
[14] E. Amoruso, L. Dowdall, M. Kollamkulam, O. Ukaegbu, P. Kieliba, T. Ng, H. Dempsey-Jones, D. Clode, and T. Makin, “Intrinsic somatosensory feedback supports motor control and learning to operate artificial body parts,” Journal of Neural Engineering, vol. 19, no. 1, p. 016006, 2022.
[15] A. Ajoudani, S. B. Godfrey, M. Bianchi, M. G. Catalano, G. Grioli, N. Tsagarakis, and A. Bicchi, “Exploring teleimpedance and tactile feedback for intuitive control of the pisa/iit softhand,” IEEE Transactions on Haptics, vol. 7, no. 2, pp. 203–215, 2014.
[16] A. S. Ivani, F. Barontini, M. G. Catalano, G. Grioli, M. Bianchi, and A. Bicchi, “Vibes: Vibro-inertial bionic enhancement system in a prosthetic socket,” 2023, pp. 1–6.
[17] S. B. Godfrey, K. D. Zhao, A. Theuer, M. G. Catalano, M. Bianchi, R. Breighner, D. Bhaskaran, R. Lennon, G. Grioli, M. Santello, A. Bicchi, and K. Andrews, “The softhand pro: Functional evaluation of a novel, flexible, and robust myoelectric prosthesis,” PLOS ONE, vol. 13, no. 10, 10 2018. [Online]. Available: https://doi.org/10.1371/journal.pone.0205653
[18] C. Piazza, G. Grioli, M. Catalano, and A. Bicchi, “A century of robotic hands,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 2, 2019.
[19] M. Grandini, E. Bagli, and G. Visani, “Metrics for multi-class classification: an overview,” arXiv preprint arXiv:2008.05756, 2020.
[20] C. Della Santina, C. Piazza, G. Gasparri, M. Bonilla, M. Catalano, G. Grioli, M. Garabini, and A. Bicchi, “The quest for natural machine motion: An open platform to fast-prototyping articulated soft robots,” IEEE Robotics and Automation Magazine, vol. PP, 02 2017.
[21] N. Landin, J. M. Romano, W. McMahan, and K. J. Kuchenbecker, “Dimensional reduction of high-frequency accelerations for haptic rendering.” Springer, 2010.
[22] H. Lee, G. I. Tombak, G. Park, and K. J. Kuchenbecker, “Perceptual space of algorithms for three-to-one dimensional reduction of realistic vibrations,” IEEE Transactions on Haptics, vol. 15, no. 3, pp. 521–534, 2022.
[23] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, 1997.
[24] R. C. Staudemeyer and E. R. Morris, “Understanding lstm–a tutorial into long short-term memory recurrent neural networks,” arXiv preprint arXiv:1909.09586, 2019.
[25] Y. Shao, H. Hu, and Y. Visell, “A wearable tactile sensor array for large area remote vibration sensing in the hand,” IEEE Sensors Journal, vol. 20, no. 12, pp. 6612–6623, 2020.
[26] B. Stephens-Fripp, M. J. Walker, E. Goddard, and G. Alici, “A survey on what australians with upper limb difference want in a prosthesis: justification for using soft robotics and additive manufacturing for customized prosthetic hands,” Disability and Rehabilitation: Assistive Technology, 2019.
[27] P. Capsi Morales, C. Piazza, M. Catalano, G. Grioli, L. Schiavon, E. Fiaschi, and A. Bicchi, “Comparison between rigid and soft poly-articulated prosthetic hands in non-expert myo-electric users shows advantages of soft robotics,” Scientific Reports, vol. 11, 12 2021.