Ortiz-Catalan, M., Håkansson, B., and Brånemark, R.

, Real-time classification of simultaneous hand and wrist motions using Artificial Neural Networks with variable
threshold outputs, in Proceedings of the XXXIV International Conference on Artificial Neural Networks (ICANN), Amsterdam, May 15-16, 2013, 77:1159-1164.

Real-time classification of simultaneous hand

and wrist motions using Artificial Neural
Networks with variable threshold outputs
Max Ortiz-Catalan, Bo Håkansson, and Rickard Brånemark

Abstract—Limb motions normally involve more than one degree The presented pattern recognition task can be named in
of freedom combined in a coordinated manner. Although prosthetic different ways: multi-class, multi-label, mixed-label, etc…
hardware today could be combined for a highly motorized limb Multi-label classification problems are normally formulated by
replacement, the control options available to amputees are so limited
having a feature set bellowing to each of the different labels
that this approach is rarely used. In this work, we introduce a
classification strategy for the real-time simultaneous prediction of the [8], e.g. a scientific article is associated to different keywords
individual movements present in natural motions. The real-time (labels). Analogously, a movement involving different DoF
evaluation of this strategy based on a Multi-Layer Perceptron (MLP) would have the associated labels of each DoF. A difference in
with variable threshold outputs resulted in high motion completion this case however, is that the feature vectors are also mixed,
rates. Moreover, the MLP alone showed higher offline accuracy than and therefore the terminology of mixed-label is here more
previously reported. This classifier was developed and evaluated in
appropriated. A mixed-label problem also implies that at least
BioPatRec, an open source framework for advanced prosthetic
control strategies based in pattern recognition algorithms. The source two classes are involved, and therefore its multi-class nature
code and the data obtained in this study are freely available to be can be deduced.
used for further algorithms development and benchmarking. The feasibility of decoding simultaneous motions has been
shown by Yatsenko et al. with offline accuracies up to 75%
Keywords—Artificial Neural Networks, Electromyography, using a grid of 22 electrodes [9]. Their algorithm employed
Mixed-label, Mixed-class, Simultaneous Prosthetic Control. principal component analysis (PCA), whitening, and
orthonormalization of the feature vectors assuming linear
I. INTRODUCTION relationships in the combined MES. Based on the same
principle, Jiang et al. proposed the biologically inspired

M MYOELECTRIC signals (MES) produced during

muscle contraction contain valuable information about
their resulting motions. In the case of amputees, the reaming
Nonnegative Matrix Factorization (NMF) algorithm [10]. The
NMF was tested for wrist movements satisfactorily predicting
2 out of 3 DoF. Additionally, it was compared to a multi-layer
muscles in the stump can still produce MES useful for the perceptron (MLP), which showed slightly but consistently
prediction of motion intent [1]. As an effort to advance limb better performance. This was argued to be due to the MLP
prosthetics, several pattern recognition algorithms have been capabilities to handle non-linear relationships by Muceli et al.,
used to predict limb movements by decoding associated who also used MLP for the prediction of hand kinematics
surface MES [1–7]. The majority of this work, however, has including “hand close” as an additional movement [11].
been focused on the prediction of individual movements. In this study, the “hand open” and “hand close” movements
Unfortunately, since only one movement is predicated at the are included together with 4 wrist motions for a total of 3 DoF.
time, this scheme is restraint to the serial control of different Additionally, the classification strategy presented in this work
degrees of freedom (DoF), which is cumbersome, slow, and only use surface MES as oppose to the previous work where
unnatural. On this study, an artificial neural network (ANN) additional hardware such as motion capture systems [11], [12],
with variable thresholds outputs is proposed as a solution for and force transducers [10] are required. Systems using motion
the simultaneous prediction of the different motions involved capture hardware are mainly designed for unilateral amputees,
in more natural and complex movements. where the contralateral limb is still available. Conversely, the
strategy here presented is suitable for both; unilateral and
M. Ortiz-Catalan is with the Dept. of Signals and System, Chalmers bilateral amputees.
University of Technology, and the Centre of Orthopaedic Osseointegration, Previous classifiers for simultaneous movements have been
Dept. of Orthopaedics, Sahlgrenska University Hospital, Gothenburg, Sweden
(e-mail: maxo@ chalmers.se). evaluated using pre-recorded data only (offline). Contrary to
B. Håkansson is with the Dept. of Signals and System, Chalmers common sense, it has been shown that offline accuracy does
University of Technology, Gothenburg, Sweden (e-mail: boh@ chalmers.se). not necessarily reflects real-time performance [1], [6], [7],
R. Brånemark is with the Centre of Orthopaedic Osseointegration, Dept. of
Orthopaedics, Sahlgrenska University Hospital, Gothenburg, Sweden (e-mail: [13]. This work is the first to evaluate the real-time
rickard.branemark@orthop.gu.se). performance of a simultaneous limb motion classifier.
This work was funded by VINNOVA (2010-00482) and Integrum AB.
 Ortiz-Catalan, M., Håkansson, B., and Brånemark, R., Real-time classification of simultaneous hand and wrist motions using Artificial Neural Networks with variable
threshold outputs, in Proceedings of the XXXIV International Conference on Artificial Neural Networks (ICANN), Amsterdam, May 15-16, 2013, 77:1159-1164.

This study was approved by the Swedish Regional Ethics vectors was divided in 40% training, 20% validation and 40%
Committee in Gothenburg (626-10, T688-12). testing. The feature vectors on the testing set were not
presented to the MLP during the training process, and were
II. METHODS only used to compute the offline accuracy once the network
was trained. One hundred trainings per subject were conducted
A. Signal Acquisition and Processing to compute the average offline performance. Cross-validation
was performed by randomizing the feature vectors belonging
Eight pairs of disposable Ag/AgCl electrodes (Ø = 1 cm) in to the training, validation and testing sets before each
a bipolar configuration (2 cm inter-electrode distance) were network’s training.
placed equally distributed around the proximal third of the Additionally, we have previously found in similar
forearm; one distal and one proximal. The first pair (channel experiments that using half of the randomized testing and
1) was consistently placed along the extensor carpi ulnaris and validation sets produced similar accuracies while reducing
the rest following the lateral direction around the forearm. The training time and improving cross-validation [6]. Therefore 24
bioelectric amplifier was an in-house design (MyoAmpF2F4- and 12 feature vectors per movement were used for training
VGI8) with 66 dB gain, and embedded active filtering: 4th and validation respectively, while 49 were used for testing.
order high-pass filter at 20 Hz; 2nd order low-pass filter at 400 During preliminary research we found that a concurrent
Hz; and, Notch filter at 50 Hz. The signals were digitalized at issue was the false positives of mixed movements while
2 kHz with 16-bits resolution. performing individual ones. For example, supination was
The subjects were guided by the software (BioPatRec [6]) to easily predicted as intended motion during the execution of
execute and hold the motion during 3s, and relax during 3s hand open or close.
between each contraction. Three repetitions of each movement An analysis of the output firing strength suggested that
result in 9s of raw MES information. The movements were misclassification could be considerable reduced by adjusting
hand open and close, wrist flexion and extension, and the activation threshold of the output neurons. The artificial
pro/supination, as well as all their possible combinations neuron’s firing is governed by its activation function, which
resulting in 26 motions plus rest. traditionally has a limited range, and although the firing
Seven subjects participated in this study and 4 of them had strength can vary through the activation range, the prediction
previous experience with the task, however, the remaining 3 of a motion is made binary for simplicity. Especially if
performed the task for the first time. The recording sessions, proportional control can later be implemented using the
together with the relevant subject’s information, are available average signal strength [14]. A visual indicator was created to
in the BioPatRec’s bioelectric signals repository under the individually setup the activation thresholds for each output
folder “6mov8ChFUS_MLP_Th” [6]. neuron during real-time classification. A similar visual strategy
We have previously found that under the presented has been employed by Hargrove et al. to setup binary
recording method, 70% of the contraction time (cTp) normally classification threshold to decode individual motions using
eliminates periods of absent MES while conserving the linear discriminant analysis (LDA) [3]. This approach is
isometric part of the contraction. This resulted in 121 time reasonably useful in the clinical settings as it allows easy
windows of 200 ms per movement (50 ms time increment), see customization of parameters according to individual needs.
[6] for further explanation on the signal processing and feature False positives have been observed to be more detrimental
extraction. to controllability than false negatives [3]. Therefore as an
Four time-domain signal features (mean absolute value, additional measure for the reduction of false positives during
wave length, zero crossings, and slope sign changes) were real-time prediction, the floor noisy during the relaxation
extracted from each time window in order to from the feature period of the recording session was used as a minimum value
vectors later used to feed the classifier. to overcome before further proceeding with classification. The
floor noise value corresponded to the average of the mean
B. Classifier Topology and Training absolute value of all channels. If the strength of the signal was
lower than the floor noise, the classifier predicted the “rest”
class by default.
As opposed to previous work where independent MLPs
were used per DoF [11], [12], a simplified single MLP was
employed in this study. The MLP had 32 input neurons (4 C. Real-time Evaluation
features x 8 channels), 2 hidden layers of 32 neurons each, 7
output neurons (six motions plus rest), and a sigmoid The “motion test” introduced by Kuiken et al. [4] was used
activation function. The training method was backpropagation as the real-time valuation evaluation tool. Its implementation
with η ꞊ 0.1 learning rate and α ꞊ 0.1 momentum. The training in BioPatRec and further description can be found in [6] . It
was stochastic by randomly supplying 70% of the available briefly consists on requesting the subject to execute the
training sets per learning iteration. A maximum of 200 different motions in a randomized order while evaluating the
iterations was allowed for convergence. The total of feature following key performance indicators.
 Ortiz-Catalan, M., Håkansson, B., and Brånemark, R., Real-time classification of simultaneous hand and wrist motions using Artificial Neural Networks with variable
threshold outputs, in Proceedings of the XXXIV International Conference on Artificial Neural Networks (ICANN), Amsterdam, May 15-16, 2013, 77:1159-1164.

design allows a seamless implementation of algorithms on

• Selection time. This is the time between the first signal processing; feature selection and extraction; pattern
prediction different than rest, and the first correct recognition; and, real-time control. It includes all the necessary
prediction. It includes the time window length. routines for the myoelectric control of virtual limbs, prosthetic
• Completion time. Using the same trigger as the devices, or game control; from data acquisition to real-time
selection time, the completion time elapses on the 20th evaluations, including a virtual reality environment.
correct prediction. Furthermore, it provides a repository of bioelectric signals for
• Completion rate. This is the percentage of motions that algorithm’s benchmarking on common data sets.
achieved 20 correct predictions before timeout.
• Real-time accuracy. This is reported as the percentage
of correct predictions over the total number of III. RESULTS
predictions during the completion time. The results are presented in box plots where the central line
represents the median value; the edges of the box are the 25th
The motion test was conducted after modifying the output and 75th percentiles; the whiskers give the range of data values
threshold for the movements that were easily misclassified without outliers (~ ±2.7σ); and diamond markers represent the
when not intended. The subject was asked to execute each mean values.
movement individually while the thresholds of misclassified
movements were adjusted accordingly. Then, the subject was
asked to perform the motions with modified threshold in order A. Offline Performance
to verify that the adjustment will still allow that specific
movement to be predicted when intended, see Fig. 1. The average offline accuracy was 94.7% (±3%), 92.1%
The motion tests consisted in 2 trials of 3 repetitions of each (±4%) and 92.6% (±3%) for 1 (individual), 2, and 3 mixed
movement. The time out for motion completion was 10 s. A movements respectively. The average accuracy for all subjects
prediction was made every 50 ms and the average processing and movements was 92.9% (±3%). The rest motion was
time was 20 ms, therefore the fastest selection time can be 220 consider together with the individual motions. These results
ms and the fastest completion time 1.17 s considering 20 are illustrated in Fig. 2.
correct predictions.

Fig. 2. Offline accuracy grouped by the number of mixed movements.

Fig. 1. Graphical user interface used the set up the activation

threshold for the output neurons. The height of the bar indicate the B. Real-time Performance
prediction strength and the color shows the neuron’s state: active
(green) or not (blue). The default activation value is 0.5. The subject
The average selection time was 0.50s (±0.2s), 0.79s (±0.3s)
was performing Close Hand + Extend Hand + Pronation in this
example.
and 0.89s (±0.3s) for 1 (individual), 2, and 3 mixed
movements respectively. The average selection time for all
subjects and movements was 0.76s (±0.3s). These results are
D. Classifier Implementation illustrated in Fig. 3, where it can be seen the selection time
increasing together with the number of mixed movement. This
The classification strategy presented in this work was is mainly because the first prediction was normally missing
implemented in BioPatRec, an open source framework for the one of the requested motions.
development of advanced prosthetic control strategies based in
pattern recognition algorithms [6]. BioPatRec’s modular
 Ortiz-Catalan, M., Håkansson, B., and Brånemark, R., Real-time classification of simultaneous hand and wrist motions using Artificial Neural Networks with variable
threshold outputs, in Proceedings of the XXXIV International Conference on Artificial Neural Networks (ICANN), Amsterdam, May 15-16, 2013, 77:1159-1164.

Fig. 3. Selection times grouped by the number of mixed movements. Fig. 5. Completion rate grouped by the number of mixed movements.

The average completion time was 2.2s (±0.7s), 2.3s (±0.3s)

and 2.5s (±0.6s) for 1 (individual), 2, and 3 mixed movements
respectively. The average completion time for all subjects and
movements was 2.3s (±0.5s). These results are illustrated in
Fig. 4.

Fig. 6. Illustration of the cumulative completed motions versus time.

The subjects with more exposure to the task (S1-S4) show faster and
higher completion rates in comparison to first timers (S5-S7).

Fig. 4. Completion times grouped by the number of mixed

movements.

The average completion rate was 0.99 (±0.03), 0.98 (±0.03)

and 0.96 (±0.04) for 1 (individual), 2, and 3 mixed movements
respectively. The average completion rate for all subjects and
movements was 0.97 (±0.02). These results are illustrated in
Fig. 5.

The cumulative completion rate is shown in Fig. 6 where it

can be seen that 80% of the motions were completed by the
more experienced subjects before 2.32 seconds, as oppose to
only 52% by the first timers. Fig. 7. Real-time accuracy grouped by the number of mixed
movements.
The average real-time accuracy was 65.2% (±13.6%),
59.6% (±6.5%), and 54.4% (±9.4%) for 1 (individual), 2, and C. Applications
3 mixed movements respectively. The overall accuracy was
59.3% (±8.1%). These results are illustrated in Fig. 7.
In order to demonstrate the applicability of the presented
classification strategy, it was incorporated into BioPatRec to
allow the simultaneous control of a multifunctional prosthetic
 Ortiz-Catalan, M., Håkansson, B., and Brånemark, R., Real-time classification of simultaneous hand and wrist motions using Artificial Neural Networks with variable
threshold outputs, in Proceedings of the XXXIV International Conference on Artificial Neural Networks (ICANN), Amsterdam, May 15-16, 2013, 77:1159-1164.

device and a virtual hand (virtual and augmented reality), Fig The results presented in this work should only be used as an
8. It was also employed to substitute keyboard strokes for indication of the feasibility of such a system. The absolute
game control, which is aimed to be used a neuromuscular values of speed and accuracy are dependent on the processing
rehabilitation tool. Videos of these demonstration are available hardware and the employed real-time test respectively. The
online on BioPatRec’s project website [15]. same algorithm running in different hardware will have
different response time. The subject’s attention and motivation
during the motion test might also alter the absolute results, as
one can be easily distracted or misinterpret requested motions,
thus resulting in delays and accuracy decay. Furthermore,
additional subject training is known to improve their
classification results [16]. We observed such impact in this
study as illustrated in Fig. 6 where considerable difference can
be observed between subjects with previous experienced in the
task, and first timers.
All the discussed variables so far suggest that the results
presented in this work are likely to be lower than the actual
potential of the classification strategy. For example, more
practice, electrodes and their selective placement including
additional muscles, are likely to improve these results.
However, to truly evaluate the real performance of any
Fig. 8. Demonstrations on the classifier used for the simultaneous
control of a multi-functional prosthesis (top-left inset); a virtual hand prosthetic control strategy, further testing on clinical settings is
in augmented reality (top-ring inset); and game control (bottom-right necessary.
inset). The bottom-left inset shows a trans-radial amputee controlling
a virtual hand to demonstrate the viability in subjects with missing V. CONCLUSION
limbs. The decoding of mixed movements is a necessary step
towards a more natural control of artificial limbs. In this work,
IV. DISCUSSION we demonstrate the feasibility of simultaneous motion
A concern worth of attention when planning to substitute classification for the real-time control of artificial limbs.
individual by simultaneous control, is the loss of stability when Furthermore, the proposed MLP implementation using
aiming to move a single DoF. Our results show the highest variable threshold outputs can be potentially used in the
real-time accuracy and completion rate, as well as the fastest clinical settings due to its operation simplicity and practicality.
selection and completion time, for single motions over the
mixed ones, thus reassuring the approach of simultaneous ACKNOWLEDGMENT
classification without compromising individual control. The authors thank Nichlas Sander and Morten B.
Further research on control algorithms aiming to reduce Kristoffersen for their contribution to the virtual and
spurious classifications and improve controllability is currently augmented reality environments. The authors also thank to all
performed by our group as well as tests on amputee subjects. the participants in this study.
In this work, the electrodes were only placed around the
forearm as oppose to previous implementations where REFERENCES
additional electrodes were also paced in the biceps muscle
[1] G. Li, A. E. Schultz, and T. Kuiken, “Quantifying pattern recognition-
[10]. The biceps provide independent information on the based myoelectric control of multifunctional transradial prostheses”,
supination movement, thus potentially facilitating IEEE Trans Neural Syst Rehabil Eng, vol. 18, no. 2, pp. 185–92, Apr.
classification. Furthermore, less electrodes were used in 2010.
[2] E. J. Scheme and K. Englehart, “Electromyogram pattern recognition for
comparison with previous work [9]. However, despite the control of powered upper-limb prostheses: State of the art and
simplifications made in this study, the MLP with variable challenges for clinical use”, J Rehabil Res Dev, vol. 48, no. 6, p. 643,
threshold outputs achieved higher offline accuracies than 2011.
[3] L. J. Hargrove, E. J. Scheme, K. B. Englehart, and B. S. Hudgins,
previously reported. It is worthy of notice however, that no fair “Multiple binary classifications via linear discriminant analysis for
performance comparison can be made between related studies improved controllability of a powered prosthesis.,” IEEE Trans Neural
due to the several variables involved in the methodology. The Syst Rehabil Eng, vol. 18, no. 1, pp. 49–57, Feb. 2010.
[4] T. A. Kuiken, G. Li, B. A. Lock, R. D. Lipschutz, L. A. Miller, K. A.
differences range from the electrode type and positioning;
Stubblefield, and K. B. Englehart, “Targeted muscle reinnervation for
through acquisition protocols; to subject’s anatomy, skills and real-time myoelectric control of multifunction artificial arms,” JAMA:
experience. This stresses the importance of using common data the Journal of the American Medical Association, vol. 301, no. 6, pp.
sets and a common platform for the comparison of different 619–628, 2009.
[5] M. Ortiz-Catalan, R. Brånemark, and B. Håkansson, “Biologically
algorithms. BioPatRec and the data generated in this study Inspired Algorithms Applied to Prosthetic Control,” in Proceedings of
provide an openly available solution for this problem. the IASTED International Conference, Biomedical Engineering,
BioMed, Innsbruck, 15-17 Feb 2012, pp. 7–15.
 Ortiz-Catalan, M., Håkansson, B., and Brånemark, R., Real-time classification of simultaneous hand and wrist motions using Artificial Neural Networks with variable
threshold outputs, in Proceedings of the XXXIV International Conference on Artificial Neural Networks (ICANN), Amsterdam, May 15-16, 2013, 77:1159-1164.

[6] M. Ortiz-Catalan, R. Brånemark, and B. Håkansson, “BioPatRec: A

modular research platform for the control of artificial limbs based on
pattern recognition algorithms,” Source Code for Biology and
Medicine, 2013, 8:11.
[7] E. J. Scheme, K. B. Englehart, and B. S. Hudgins, “Selective
classification for improved robustness of myoelectric control under
nonideal conditions,” IEEE Trans Biomed Eng, vol. 58, no. 6, pp.
1698–705, 2011.
[8] G. Tsoumakas and I. Katakis, “Multi Label Classification: An
Overview,” International Journal of Data Warehousing and Mining,
vol. 3, no. 3, pp. 1–13, 2007.
[9] D. Yatsenko, D. McDonnall, and K. S. Guillory, “Simultaneous,
proportional, multi-axis prosthesis control using multichannel surface
EMG,” in Conf Proc IEEE Eng Med Biol Soc, Lyon, 23-26 Aug 2007,
pp. 6134–7, 2007.
[10] N. Jiang, K. B. Englehart, and P. Parker, “Extracting simultaneous and
proportional neural control information for multiple-DOF prostheses
from the surface electromyographic signal,” IEEE Trans Biomed Eng,
vol. 56, no. 4, pp. 1070–80, 2009.
[11] S. Muceli and D. Farina, “Simultaneous and Proportional Estimation of
Hand Kinematics from EMG during Mirrored Movements at Multiple
Degrees of Freedom,” IEEE Trans Neural Syst Rehabil Eng, vol. PP,
no. 99, 2011.
[12] N. Jiang, J. L. Vest-Nielsen, S. Muceli, and D. Farina, “EMG-based
simultaneous and proportional estimation of wrist/hand dynamics in
uni-Lateral trans-radial amputees,” J Neuroengineering Rehabil, vol. 9,
no. 1, p. 42, 2012.
[13] B. A. Lock, K. Englehart, and B. Hudgins, “Real-time myoelectric
control in a virtual environment to relate usability vs. accuracy,” in
Proceedings of the 2005 MyoElectric Controls/Powered Prosthetics
Symposium, Fredericton, 17-19 Aug 2005.
[14] A. M. Simon, K. Stern, and L. J. Hargrove, “A Comparison of
Proportional Control Methods for Pattern Recognition Control,” in
Proc. 33rd Annu. Int. Conf. IEEE EMBS, Boston, 30 Aug – 3 Sep 2011,
pp. 3354–3357, 2011.
[15] M. Ortiz-Catalan, “BioPatRec [http://code.google.com/p/biopatrec],”
Accessed Jan, 2012.
[16] N. E. Bunderson and T. Kuiken, “Quantification of feature space
changes with experience during electromyogram pattern recognition
control.,” IEEE Trans Neural Syst Rehabil Eng, vol. 20, no. 3, pp. 239–
46, 2012.