Ergonomics Measurements Using Kinect With A Pose Correction Framework
Ergonomics Measurements Using Kinect With A Pose Correction Framework
Ergonomics Measurements Using Kinect With A Pose Correction Framework
Correction Framework
Pierre Plantard, Hubert Shum, Franck Multon
Abstract
Evaluation of potential risks of musculoskeletal disorders in real workstations is challenging as the environment
is cluttered, which makes it difficult to correctly and accurately assess the pose of a worker. Being marker-free
and calibration-free, Microsoft Kinect is a promising device to assess these poses, but it can deliver unreliable
poses especially when occlusions occur. To overcome this problem, we propose to detect badly recognized body
parts and to replace them by an appropriate combination of example poses gathered in a pre-recorded pose. The
main contribution of this work is to organize the database as a filtered pose graph structure that enables the system
to select relevant candidates for the combination: candidates that ensure continuity with the previous pose and
similarity with the available reliable information. We applied the proposed method in a realistic environment that
involved sub-optimal Kinect placement and several types of occlusions. An optoelectronic motion capture system
was concurrently used to obtain ground truth joint angles. In an ergonomics context, we also computed Rapid
Upper Limb Assessment RULA scores. This kind of ergonomics tool requires to rate the pose of the worker based
on an estimation of the joint angles. These latter are then used to provide a global risk score. Results showed that
when occlusions occur, the inaccurate raw Kinect data could be significantly improved using our correction
method, leading to acceptable joint angles. As RULA calculation is based on angular thresholds, which tends to
minimize the effect of joint angle errors, when these error values are not close to thresholds. However, for realistic
scenarios with occlusions that lead to very large joint angle errors, the correction method also provided
significantly better RULA scores. Our method opens new perspectives to define new fatigue or solicitation indexes
based on continuous measurement contrary to classical static images used in ergonomics. As the computation time
is very low, it also enables real-time feedback and interaction with the operator.
2
P. Plantard, Ergonomics Measurements using Kinect with a Pose Correction Framework
process to replace unreliable information by For example, the upper arm score ranges from 1 to 4
plausible combined one while preserving continuity if the shoulder flexion is within [-20; +20], < -20 or
and similarity to the reliable information (Figure within [20; 45], within [45; 90], or >90 respectively.
1.b). Finally, a physical model is used to filter the The same type of threshold are applied to the other
resulted pose and avoid jerky motions (Figure 1.c). upper-body joints angles. This approach leads to a
For the first step, reliability of Kinect joint is discretization of the score that may be less sensitive
computed with the method described in (Shum et al. to noise than methods that are based on continuous
2013) which provides a reliable value between 0 and scores. One has to notice that additional conditions
1 for each joint. can increase the local body part scores, such as when
In the second step, the unreliable part of the Kinect the shoulder is raised or the upper arm is abducted.
pose is corrected using a local optimization process. These scores are combined to provide a global risk
Firstly, the method selects the pose candidates that score for the left and right body parts, ranging from
are relevant with respect to the reliable Kinect 1 (posture is acceptable) to 7 (workstation requires
information in the Filtered Pose Graph. The Filtered investigation and changes immediately).
Pose Graph allowed us to use large database while
maintaining good real time performance during this 3.1.2 Computation of joint angles using the Kinect
online candidate selection. Moreover, selecting data
poses connected to the current one helps us to
consider only candidates that ensures continuity with To use the RULA method, relevant joint angles have
the current pose, while unorganized databases to be computed based on the Kinect skeleton data
cannot guarantee continuity. (see Figure 2). A Kinect pose is defined as 𝑝 =
Then, a combined pose is obtained by minimizing a {𝑥𝑗 , 𝑦𝑗 , 𝑧𝑗 }𝑗=1..𝑁 where N stands for the number of
set of energy functions in an optimisation process.
joints in the pose, and 𝑥𝑗 , 𝑦𝑗 , 𝑧𝑗 stand for the 3D
More precisely, optimization aims at finding the
most appropriate weights used to linearly combine a Cartesian coordinates of the 𝑗𝑡ℎ joint. According to
set of pose candidates while minimizing a set of the estimated joint positions, joint angles should be
constraints modelled as energy terms: 1) Minimizing computed using the ISB recommendation (Wu et al.
the difference between the optimizing pose and the 2002, Wu et al. 2005). However, the Kinect skeleton
observed Kinect pose for the joints considered as is not fully compatible with this recommendation as
reliable. 2) The style term minimizes the difference it does not provide all the required anatomical
between the optimizing pose and the selected pose in landmarks.
the Filtered Pose Graph. 3) Avoiding changes in We consequently slightly adapted the joint angle
bone length when combining various joint centers definition to take the available Kinect joints (maned
positions. 4) Ensuring continuity to minimize high with letters in Figure 2 a)) into account.
frequency jittery movements.
The optimization score is evaluated as a weighted
sum of the energy terms. The optimization process
continues until an optimal solution is found, or the
number of iterations reaches a predefined limit.
In the last step we filter the optimized joint positions
using a dynamic model to accurately maintain
kinematics features such as segment lengths.
Readers are referred to Shum et al. (2013) for more
details.
3
P. Plantard, Ergonomics Measurements using Kinect with a Pose Correction Framework
The pelvis coordinate is define relatively to the 40 cm per 30 cm per 17 cm box with the two hands,
recommendation (Wu and al. 2002). Y-axis is along place it in front of the abdomen, wait few seconds
the trunk axes represented by the vector from the hip and put it back to the original position. The box
center (a) to the spine (b). The X-axis is defined as (attached to a magnet) had two target placements, in
the normal of the plan formed by the Y-axis, the left order to generate two different motions. The first
(m) and the right (n) hips. Finally the Z-axis is placement named F (i.e. Front) the target was located
computed as the normal of the X-axis and Y-axis. in front of the subject, at 1.70 m high, 0.35 m left and
For the trunk coordinate system, the Y-axis is 0.50 m in front. In the second placement S (i.e. Side)
represented by the vector from the spine joint (b) to the target was located on the left of the subject,
the shoulder center joint (c). The X-axis is defined as aligned with the two shoulders at the same height and
the normal of the plan formed by the Y-axis, the left 0.55m left.
(e) and the right (i) shoulders. Finally the Z-axis is To simulate workplace environmental constraints,
computed as the normal of the X-axis and Y-axis. three experimental setups were defined, including
The shoulder coordinate system is defined according manipulation box (to add occlusions during the
to the ISB recommendation. The Y-axis is given by manipulation task) and various Kinect placements:
the vector from elbow joint (f or j) to shoulder joint - [NB – No Box condition]: the manipulation
(e or i). The Z-axis is the normal of the plan formed of the box was simulated by the subject
by the Y-axis and the lower arm defined from wrist without using actually a box to avoid
joint (g or k) to elbow joint (f or j). The X-axis is the occlusions. The Kinect was placed in front
normal of the plane formed by the two previous axis. of the subjects, as recommended by
These three coordinate systems were placed to the Microsoft. It enabled us to test the
hip center (a), shoulder center (c) and shoulder joints robustness of the Kinect sensor under
(e or i) respectively, as depicted in Figure 2 b). favourable conditions. In this condition, the
The trunk and shoulder joint angles were then subject simply reached to the position of the
computed according to the ISB recommendation. attachment where the box would usually be
We changed the matrix decomposition sequences of located.
the shoulder joint angle computation from YXY to - [B – Box]: the manipulation is actually
ZXY, to obtain abduction values and to limit gimbal preformed with the box to created
lock problems as suggested in (Senk and Chèze occlusions of parts of the body, as in real
2006). working situation. The Kinect was again
The Kinect skeleton also does not provide enough placed in front of the subject, as
points to compute the neck and elbow local recommended by Microsoft.
coordinate systems. We alternatively computed the - [B45 – Box and 45° sensor placement] As
elbow joint angles using the vector convention in the B condition the subject actually
detailed by (Bonnechère et al. 2014). manipulated the box but the Kinect was
The neck joint angles were computed by planar placed 45° left forward of the subject, as in
projection of the neck vector (c to d) expressed into real cluttered environments. In this
the local trunk coordinate system. condition, occlusion was more important.
As there is not enough available information to The subject repeated each gesture 5 times: getting,
compute the wrist angles, the wrist, and wrist twist and putting, for each conditions and box placement
RULA scores are set manually. Finally, all the (FNB, FB45, FB, SNB, SB45 and SB): 5 × 3 × 2 = 30
threshold values that are not provided by the RULA motions were recorded for each subject.
method are set to 20°, such as (Aptel et al. 2000) for In this experimentation, the correction was
the shoulder joint abduction. performed using a Filtered Motion Graph made-up
with 130 professional example gestures leading to
3.2. Experimental set-up
532,624 poses. The poses were then filtered into
In this section, we present the experimental protocol 2,048 nodes with an average of almost 7.81 links per
used to evaluate the relevance of the proposed node. The filtration intensity was chosen relative to
method in constrained conditions, such as work the optimal condition used in (Shum et al. 2013).
conditions. To this end, we carried-out an
experimental protocol with 12 male participants 4. Results
(age: 30.1±7.0 years, height: 1.75±0.046 m, mass:
Firstly, we evaluated the accuracy of the joint angles
62.2±7 kg). They were equipped with 47 reflective
measured with raw Kinect data or corrected ones, as
markers positioned at standardized anatomical
these angles were used in RULA. To this end, we
landmarks, as suggested in (Wu et al. 2005). The
computed the root mean square error (RMSE)
motion of the participants was recorded by both
between the joint angles measured with the reference
Microsoft Kinect 2 sensor and 15 cameras Vicon
Vicon motion capture system, and those computed
optical motion capture system.
with the raw and corrected Kinect data, as described
The subject had to perform getting and putting
in section 3.1.2. This RMSE has been applied to the
motions. More precisely, the subject had to carry a
8 main angles used in the RULA score: αT, γT, for the
4
P. Plantard, Ergonomics Measurements using Kinect with a Pose Correction Framework
5
P. Plantard, Ergonomics Measurements using Kinect with a Pose Correction Framework
6
P. Plantard, Ergonomics Measurements using Kinect with a Pose Correction Framework
6. Conclusion
Figure 6: Example of ergonomics application based on our
Kinect data correction framework. This paper presents an evaluation of the quality of
angular and RULA score values when using a Kinect
5. Discussion with software correction. The result showed that
both corrected and uncorrected Kinect data enabled
The results showed correction of Kinect data to compute acceptable to reliable angular and RULA
allowed us a significant improvement of the joint score data in occlusion-free conditions. However, in
angle accuracy, particularly when the body was more challenging environments with occlusions,
partially occluded. RULA score was more reliable kinematic data provided by the Kinect was more
with corrected data, although angular thresholds noisy, leading to inaccurate estimation of the joint
tended to minimize the effect of noise when using angles. The proposed correction framework enables
Kinect raw data. us to consider encumbered capture area (e.g.
The skeleton delivered by the Kinect did not contain production chain) that leads to such occlusions or
all the required information to compute all the joint bad sensor placements. Uncorrected Kinect data
angles as accurately as using Vicon data with ISB exhibited much higher errors than corrected ones,
recommendations. Particularly, Kinect delivered which may lead to difficulties when using the system
very noisy and unreliable information about the in real working environments.
hand. Hand configuration is a key point in
ergonomics, as reported in the RULA assessment Finally, one has to notice that correction runs in real
scores. As it is not correctly measured by the Kinect time and allows the possibility to implement real-
most of the time, further research would be necessary time user feedback, with potential application in
to address this particular point. Hence, motion training or virtual prototyping, as suggested by
involving dexterous manipulation and fine motion of (Vignais et al. 2013).
the wrist cannot be studied with such a system. The
method used for correction involves that a minimum Acknowledgement
set of reliable information is delivered by the Kinect,
which is not guaranteed for the wrist in Kinect v1 This work was partially funded by the Cifre
and v2. convention N1222/2012 and Faurecia Company. It
Another limitation of the method is the use of a was also partially supported by the Engineering and
database that may not correspond to the actual use of Physical Sciences Research Council (EPSRC) (Ref:
the system. In this paper, we used a database trained EP/M002632/1).
with working motions, similar to those performed by
the subject. For other type of motions, involving References
poses that have never been recorded before,
Aptel M, Lahaye S, Gerling A, 2000. Un outil
especially for larger ranges of motions, the
d'évaluation des facteurs biomécaniques: OREGE
performance of the correction method would not be
(Outil de Repérage et d'Evaluation des Gestes). In
so good.
INRS (eds), Documents pour le médecin du travail,
The method is also based on a set of parameters, such
83, 217-223.
as the number of candidates used to run the
Auvinet E, Multon F, Aubin C E, Meunier J, Raison
optimization, or the thresholds applied to prune the
M, 2014. Detection of gait cycles in treadmill
database and eliminate redundant information. It
walking using a kinect. Gait & posture 41(2), 722-
would be interesting to evaluate the actual impact of
725.
these parameters on the performance of the
Auvinet E, Multon F, Meunier J, 2012. Lower limb
correction method.
movement asymmetry measurement with a depth
Despite the reported limitations, the results of the
camera. In: Engineering in Medicine and Biology
current study are promising for the ergonomic
Society (EMBC), 2012 Annual International
evaluation of workstations. Kinect has already been
Conference of the IEEE, 6793-6796.
considered as a promising tool to evaluate
Bonnechre B, Jansen B, Salvia P, Bouzahouene H,
ergonomics on-site (Diego-Mas et al. 2014; Patrizi
Omelina L, Moiseev F, Sholukha V, Cornelis J,
et al. 2015), but only with very simple and inaccurate
7
P. Plantard, Ergonomics Measurements using Kinect with a Pose Correction Framework
Rooze M, Jan S V S, 2014. Validity and reliability Cybernetics, IEEE Transactions on 43(5), 1357-
of the kinect within functional assessment activities: 1369.
Comparison with standard stereo-photogrammetry. Vieira E R, Kumar S, 2004. Working postures: A
Gait & Posture 39 (1), 593-598. literature review. Journal of Occupational
Cassola F, Morgado L, de Carvalho F, Paredes H, Rehabilitation 14(2), 143-159.
Fonseca B, Martins P, 2014. Online-gym: A 3d Vignais N, Miezal M, Bleser G, Mura K, Gorecky D,
virtual gymnasium using kinect interaction. Procedia Marin F, 2013. Innovative system for real-time
Technology 13(0). In: Research conference on ergonomic feedback in industrial manufacturing.
virtual worlds learning with simulations, 130-138 Applied ergonomics 44 (4), 566-574.
Chai J, Hodgins J K, 2007. Constraint-based motion Wang Y, Dubey R, Magnenat-Thalmann N,
optimization using a statistical dynamic model. Thalmann D, 2013. An immersive multi-agent
ACM Trans. Graph, 26(3). system for interactive applications. Vis Comput 29,
Clark, R A, Pua Y H, Fortin K, Ritchie C, Webster 323-332.
K E, Denehy L, Bryant A L, 2012. Validity of the Wu G, Siegler S, Allard P, Kirtley C, Leardini A,
Microsoft Kinect for assessment of postural control. Rosenbaum D, and Others, 2002. ISB
Gait & Posture 36, 372-377. recommendation on definitions of joint coordinate
Diego-Mas J A, Alcaide-Marzal J, 2014. Using system of various joints for the reporting of human
kinect sensor in observational methods for assessing joint motion—part I: ankle, hip, and spine. Journal
postures at work. Applied Ergonomics 45(4), 976- of biomechanics 35(4), 543-548.
985. Wu G, van der Helm F C, Veeger H D, Makhsous
Galna B, Barry G, Jackson D, Mhiripiri D, Olivier P, M, Roy P V, Anglin C, Nagels J, Karduna A R,
Rochester L, 2014. Accuracy of the microsoft kinect McQuade K, Wang X, Werner F W, Buchholz B,
sensor for measuring movement in people with 2005. ISB recommendation on definitions of joint
parkinson's disease. Gait & Posture 39(4), 1062- coordinate systems of various joints for the reporting
1068. of human joint motion part ii: shoulder, elbow, wrist
Gameiro J, Cardoso T, Rybarczyk Y, 2014. Kinect- and hand. Journal of Biomechanics 38(5), 981-992.
sign, teaching sign language to "listeners" through
agame. Procedia Technology 17(0). In: conference
on Electronics, Telecommunications and
Computers, 384-391,
Kurillo G, Chen A, Bajcsy R, Han J J, 2013.
Evaluation of upper extremity reachable workspace
using Kinect camera. Technol. Health Care 21, 641-
656.
Liu Z, Zhou L, Leung H, Shum H P H 2015. Kinect
Posture Reconstruction based on a Local Mixture of
Gaussian Process Models. IEEE Transactions on
Visualization and Computer Graphics 99.
McAtamney L, Corlett E N, 1993. RULA: a survey
method for the investigation of work-related upper
limb disorders. Applied Ergonomics 24(2), 91-99.
Pedersoli F, Benini S, Adami N, Leonardi R, 2014.
Xkin: an open source framework for hand pose and
gesture recognition using kinect. Vis Comput 30,
1107-1122.
Patrizi A, Pennestrì E, Valentini P P, 2015.
Comparison between low-cost marker-less and high-
end marker-based motion capture systems for the
computer-aided assessment of working ergonomics.
Ergonomics 1-8.
Plantard P, Auvinet E, Le Pierres A-S, Multon F,
2015. Pose estimation with a kinect for ergonomic
studies: Evaluation of the accuracy using a virtual
mannequin. Sensors 15, 1785-1803.
Senk M, Chèze L, 2006. Rotation sequence as an
important factor in shoulder kinematics. Clinical
Biomechanics 21(1). In: proceedings of the 5th
Meeting of the International Shoulder Group.
Shum H P, Ho E S, Jiang Y, Takagi S, 2013. Real-
time posture reconstruction for microsoft kinect.