Journal of Biomechanics 104 (2020) 109718

Contents lists available at ScienceDirect

Journal of Biomechanics
journal homepage: www.elsevier.com/locate/jbiomech
www.JBiomech.com

Accuracy of image data stream of a markerless motion capture system in

determining the local dynamic stability and joint kinematics of human
gait
Saikat Chakraborty a,⇑, Anup Nandy a, Takazumi Yamaguchi b, Vincent Bonnet c, Gentiane Venture b
a
Machine Intelligence and Bio-motion Research Lab, Department of Computer Science and Engineering, National Institute of Technology, Rourkela, India
b
GV Lab., Department of Mechanical Systems Engineering, Tokyo University of Agriculture and Technology, Tokyo, Japan
c
Laboratory of Images, Signals and Intelligent Systems, University of Paris-Est, Creteil (UPEC), Creteil, France

a r t i c l e i n f o a b s t r a c t

Article history: Assessment of gait parameters is commonly performed through the high-end motion tracking systems,
Accepted 18 February 2020 which limits the measurement to sophisticated laboratory settings due to its excessive cost. Recently,
Microsoft Kinect (v2) sensor has become popular in clinical gait analysis due to its low-cost. But, deter-
mining the accuracy of its RGB-D image data stream in measuring the joint kinematics and local dynamic
Keywords: stability remains an unsolved problem. This study examined the suitability of Kinect(v2) RGB-D image
Kinect v2 data stream in assessing those gait parameters. Fifteen healthy participants walked on a treadmill during
Gait
which lower body kinematics were measured by a Kinect(v2) sensor and a optophotogrametric tracking
Extended Kalman filter
Local dynamic stability
system, simultaneously. Extended Kalman filter was used to extract the lower extremity joint angles from
Maximal Lyapunov exponent Kinect, while inverse kinematics was used for the gold standard system. For both systems, local dynamic
stability was assessed using maximal Lyapunov exponent. Sprague’s validation metrics, root mean square
error (RMSE) and normalized RMSE were computed to confirm the difference between the joint angles
time series of the two systems while relative agreement between them was investigated through
Pearson’s correlation coefficient (pr ). Fisher’s Exact Test was performed on maximal Lyapunov exponent
to investigate the data independence while reliability was assessed using intraclass correlation coeffi-
cients. This study concludes that the RGB-D data stream of Kinect sensor is efficient in estimating joint
kinematics, but not suitable for measuring the local dynamic stability.
Ó 2020 Elsevier Ltd. All rights reserved.

1. Introduction from high-end motion tracking cameras combined with the force
plate. But, due to excessive cost, and requirement of experts to
Local dynamic stability (LDS) is a quantitative measurement of operate those systems, their uses is bounded to high-end laborato-
the resilience of the coordinative pattern of human gait against the ries. A low-cost gait assessment system is highly required for the
internal perturbation that generates from the neuromotor noise people having gait pathology and as well as for clinics. It will also
during walking (van Emmerik et al., 2016). Assessment of LDS is aid the patients to conduct in-home gait assessment. Recently,
extremely important in determining gait abnormality and fall- Microsoft Kinect (v2) has become popular in clinical gait analysis
risk in geriatric people (Bizovska et al., 2018) as well as in other domain due to its affordable cost and portability. But, the efficiency
pathological population (Speedtsberg et al., 2018; Punt et al., of this sensor in measuring LDS remains an unsolved problem.
2016). Among several available techniques for LDS estimation Therefore, this study aimed to investigate the capability of Kinect
(i.e. Maximal Lyapunov Exponent (MLE), Floquet multipliers etc.), (v2) in estimating LDS. Again, kinematic time series is an important
MLE was reported to be comparatively more appropriate for gait prerequisite for assessing the LDS. The validity of Kinect skeletal
data (van Emmerik et al., 2016). tracking system in estimating joint kinematics was investigated
Generally, LDS is estimated from kinematic time series data in different studies (Xu et al., 2015; Guess et al., 2017; Mentiplay
(Piórek et al., 2017; Piorek, 2015) which is obtained commonly et al., 2015). Although spatiotemporal data were reported to have
a good agreement with a gold standard system, the error for joint
angles was significantly high (Xu et al., 2015; Mentiplay et al.,
⇑ Corresponding author.
2015). Guess et al. examined the knee and hip joint angle extracted
E-mail address: 516CS1005@nitrkl.ac.in (S. Chakraborty).

https://doi.org/10.1016/j.jbiomech.2020.109718
0021-9290/Ó 2020 Elsevier Ltd. All rights reserved.
2 S. Chakraborty et al. / Journal of Biomechanics 104 (2020) 109718

from the Kinect skeletal tracker system during a drop vertical jump between the two reference points. Considering embedding state
and a hip abduction motion (Guess et al., 2017). The authors reori-

!
vector RðtÞ, MI can be written as (Liebert and Schuster, 1989):
ented the pelvis segments of the default skeletal model and com-
puted relative osteokinematic Cardan joint angles of hip and Im ðsÞ ¼ mH0  Hm ðsÞ ð2Þ
knee to compare with a gold standard system. The authors where,
reported a good agreement between the Kinect and gold standard
Xm 

!
system but under the constraints of limited activities and optimal
Hm ¼ ð1=mÞ log P Rðt i Þ ð3Þ
capture volume. Mentiplay et al. ascertained that the Kinect (v2) i¼1
skeletal tracker system is accurate for analyzing spatiotemporal 

!
parameters but not suitable to measure lower limb kinematics Hm is the entropy of m-dimensional vector and P  Rðt Þ is the prob-
(Mentiplay et al., 2015). In case of rehabilitation, Capecci et al.

!
ability of finding another m-dimensional Rðt Þ within a sphere of
examined the accuracy of the Kinect (v2) skeletal system and

!
described its inefficiency in determining kinematic parameters radius  centered around the state vector Rðt Þ (Liebert and


!
(Capecci et al., 2016). Recently, Vilas-Boas et al. validated 3-D body Schuster, 1989). Probability P  Rðt Þ can be written as (Liebert
joint data of Kinect v1 & v2 for joint kinematics against the gold
and Schuster, 1989):
standard in two different walking directions (do Carmo Vilas-
Boas et al., 2019). But, except the knee angle they did not get sat- 

! X
N 

!


! 
isfactory result. On the contrary, the RGB-D color image data P Rðti Þ ¼ ð1=NÞ h   jRðti Þ  R tj j ð4Þ
j¼1
stream of Kinect(v2) having high resolution (1920 x 1080) and neg-
ligible measurement error, seems to be promising for estimating where N is the number of sampled data points. Now, Hm ðsÞ can be
joint angles. In this study, we have proposed a method to extract written as (Liebert and Schuster, 1989):
joint angles from the RGB-D image data stream of Kinect and com-
pared the results with a gold standard system. q ð; sÞ
Hm ðsÞ ¼ lim log C m ð5Þ
q!1
The rest of the paper is divided in the following parts: Section 2
demonstrates the theoretical approach of nonlinear time series where,
analysis in determining LDS using MLE. The proposed method !
X
m 

! 1=q1
q ðÞ ¼
Cm ð1=mÞ Pq1 ð Þ ð6Þ
and experimental set-up are described in Section 3 while Section 4  R t i
exhibits the results. The paper ends with an elaborate discussion i¼1
and conclusion in Section 5 and 6 respectively.
Eq. (6) refers to the generalized correlation integral (Pawelzik
and Schuster, 1987). With the assumption of independence of s
2. Nonlinear analysis method of gait kinematic time series data to H0 , Liebert et al. (Liebert and Schuster, 1989) proved that the
minima of Im ðsÞ are equivalent to the minima of log C m1 ð; sÞ.
Human gait pattern resembles a non-linear dynamical system To approximate the value of m, FNN technique (Kennel et al.,
(Austin, 2001). The underlying dynamics of human gait can be 1992) was used. The main concept of this algorithm is that: a true
unfolded using non-linear approaches. Determination of dynamic

!
neighbor of a point on an orbit RðtÞ will still appear as a neighbor
stability of human gait attributed to the non-linear assessment of after increasing the dimension by one. Due to the very small size
gait time series data. This non-linear assessment is divided into of the embedding space, a point may be misinterpreted as a neigh-
two steps: state space reconstruction and Maximal Lyapunov bor of a reference point. To approximate the minimum embedding
Exponent estimation. dimension, the percentage of false neighbors is used to compute
(until a specified threshold) under the condition of increasing
2.1. Reconstruction of state space dimension. At the threshold point (of false neighbors), the dimen-
sion of the system is declared as the minimum embedding
State space reconstruction is an important step in MLE compu- dimension.
tation. This step approximates the underlying dynamics of a non-
linear dynamical system (Boeing, 2016). The reconstruction of 2.2. Maximal Lyapunov Exponent
state space associates two successive methods: time-delay embed-
ding and false nearest neighbors (FNN) technique. According to The measurement of divergence of neighboring trajectories over
Taken’s embedding theorem (Takens, 1981), state space of a dynam- the time in a state space is used to compute the LDS. The mean
ical system can be reconstructed by time-delayed measurement of divergence of initially infinitesimally close vectors over the time
a single time series data in m-dimensional Euclidean space. State can be quantified by MLE (Brown, 1996). The sign of MLE determi-
space embedding vector can be represented as: nes the local stability of the system. The system is said to be locally
stable in a particular direction if MLE is negative but if MLE is pos-
! itive then that system is locally unstable in that direction (Bradley
R ðt Þ ¼ ½ yðtÞ; yðt  sÞ; yðt  2sÞ; . . . ; yðt  ðm  1ÞsÞ ð1Þ
and Kantz, 2015). The mean divergence can be expressed mathe-
where yðt Þ is a single time series data, s is time lag and m is embed- matically as (Rosenstein et al., 1993):
ding dimension of the system.
dðt Þ ¼ Cek1 t ð7Þ
Calculation of the optimal value of s and m is a challenging task
(Bradley and Kantz, 2015). In this study, time lag s was estimated where d(t) is the mean divergence, C is an initial gap between the
from the first minimum of the logarithm of the correlation integral infinitesimally close vectors, and k1 is the MLE. In the literature, dif-
(CI) (Liebert and Schuster, 1989). The correlation integral method ferent studies (Rosenstein et al., 1993; Wolf et al., 1985; Kantz,
actually correlates to the intuition of mutual information (MI) 1994) have estimated the value of MLE. We have taken Rosenstein’s
Im ðsÞ of reconstructed coordinates. MI between two consecutive algorithm (Rosenstein et al., 1993) to meet that purpose. Taking the
points of a time series represents the shareable information information of reconstructed state space and embedding dimen-
between those two data points. The first local minimum of CI cor- sion, Rosenstein’s algorithm estimates the mean divergence of
responds to the locally determined maximum independence initially infinitesimally close trajectories. Depending on the
 S. Chakraborty et al. / Journal of Biomechanics 104 (2020) 109718 3

assumption of the location of neighboring pair points as the initial both systems. Space synchronization of two coordinate systems
condition of two trajectories, Rosenstein’s algorithm (Rosenstein is also a necessary step before starting data acquisition. Checker-
et al., 1993) computes the mean divergence on logarithm scale board calibration technique (Zhang, 2000) was used for that
which yields the following equation: purpose.
A kinematic model (Fig. 2) of human lower limb was developed
ln dj ðiÞ  ln C j þ k1 ðiDt Þ ð8Þ
to estimate the joint angles based on the idea of Futamure et al.
th
where dj ðiÞ is the approximate divergence of j pair of nearest (Futamure et al., 2017). Modified Denavit-Hartenberg (MDH)
neighbors, C j represents the initial separation between them, and parameter was used for the design purpose of that model. The
th kinematic model estimates 3 DoF for pelvis, 3 DoF for each hip
iDt is the i time stamp having interval length of Dt. Actually, Eq.
joint, and 1 DoF for each knee joint. GX ; GY , and GZ represent the
(8) represents a set of parallel lines of all pairs of nearest neighbors.
global coordinate system. The explanation of the symbols of
The MLE value is estimated from the slope of the linear best-fit line
Fig. 2 is given in Table 1.
to the average logarithmic divergence of the adjacent trajectories.
The average logarithmic divergence can be expressed as:
  3.3. Data analysis
yðiÞ ¼ ð1=Dt Þ ln dj ðiÞ ð9Þ
Color image and point cloud of Kinect sensor were used to esti-
where yðiÞ is the average line and average of all values of j is
mate the joint positions. The rectangular area of each AR marker
denoted by hi.
was detected using ArUco marker detection algorithm (Garrido-
Jurado et al., 2014). To track the marker positions, OpenTLD
3. Methods (Nebehay et al., 2011) was used which computed the 2-D positions
of the AR markers on the color image based on the information of
3.1. Participants the rectangular area. The 2-D positional information was converted
to 3-D using point cloud feature of Kinect. Joint position data of
Fifteen healthy youths (age (years): 23  2, height (cm): 155  Kinect were converted to joint angle using Extended Kalman Filter
10, weight (kg): 53  11) without having any musculoskeletal dis- (EKF), while classical Inverse Kinematics (IK) was used for the posi-
orders were selected. They were explained the experimental proto- tional data of the reference system.
col and agreed to participate in the experiments according to the EKF is generally used to estimate next state in a nonlinear sys-
ethics board of the local institution. tem (Simon, 2006). In our experiment, to estimate joint angle, it
was implemented as:
3.2. Experimental setup
 State prediction:
The experiment was performed at GV Lab 1 of Tokyo University
m ^ k1
^ k ¼ F m ð10Þ
of Agriculture and Technology (TUAT). To accustom with treadmill
walking a brief training session was given to the participants. Two  Prediction of error covariance:
different case studies were performed by changing the Kinect posi-
tion. In case 1, a tripod (Slik F153) was used to place the Kinect P k ¼ FPk1 F T þ Q ð11Þ
(backside of the treadmill) at a height of 3 ft.5 inch from the floor  Calculation of Kalman gain:
with a tilt angle of 2 . In case 2, Kinect was placed laterally (right
side) to the subjects at the same height and tilt angle like case 1. Dis- ^ k1 ÞPk HT ðm
Sk ¼ Hðm ^ k1 Þ þ R ð12Þ
tances between the treadmill and Kinect were  6 ft.6 inch and  5
ft.9 inch in case 1 and 2 respectively. The mentioned positions and ^ k1 ÞS1
K k ¼ Pk HT ðm k ð13Þ
tilt of Kinect were set empirically to visualize full body of the sub-
jects. In both cases, the subjects were asked to walk on the treadmill  Update:
 
at a speed of 5 km/h which is generally considered as a normal aver- v k ¼ zk  h ^ k
m ð14Þ
age walking speed of human (Carey, 2005). Five consecutive trials
(each having 30 s duration) for each subject were taken with an m ^ k þ K k v k
^k ¼ m ð15Þ
interval of 2 min between the trials (resting period). To validate
the Kinect (v2), eight Optotrak motion tracking cameras were used  Error covariance update:
which captured 3-D motion data of pelvis, hip and knee joints at P k ¼ Pk  K k Sk K Tk ð16Þ
100 Hz. Plug-in-Gait protocol (Davis et al., 1991) was used to place
the markers (20 retro-reflective) on the pelvis, hip, knee and ankle
of each subject. Four Augmented Reality (AR) markers were attached Where m represents state vector, F is the transition dynamics of
on both sides of the pelvis and both heels (Fig. 1). Placement of AR the system, z represents the measurement vector, h refers to the
markers were the same for both cases. Data were captured simulta- transformation function from state space to measurement space,
neously from the Kinect and Optotrak motion tracking system. All H is Jacobian matrix of h, noise matrix in prediction and update
the experimental protocols remained the same for both cases. are represented by Q and R respectively, k refers to time step, ^
To compare both systems, lower limb joint position data cap- and T represent the estimated variables and transpose respectively.
tured from Kinect were up-sampled to 100 Hz using Piecewise The diagonal elements of Q corresponding to segment length were
Cubic Hermite Interpolating Polynomial (PCHIP) (Shulin et al., set to zero since it is independent parameters. It was assumed that
2007). Each subject was asked to uplift their right leg once just all the diagonal elements of Q corresponding to joint variables are
after the starting time of data acquisition. This event was created different to each other. For the measurement noise matrix R, the
for time synchronization between Optotrak cameras and Kinect. diagonal elements were set as the same to each other because it
During that event, the maximum height of the right heel marker was assumed that there is no difference in amplitude of measure-
was captured and its time was defined as the reference time for ment noise among AR markers. To estimate joint angles, state vec-
tor was defined as:
1
http://web.tuat.ac.jp/gvlab.
4 S. Chakraborty et al. / Journal of Biomechanics 104 (2020) 109718

Fig. 1. Marker Placement (as per Vicon plug-in standard in addition with 4 Augmented Reality (AR) markers).

Table 1
Explanation of the symbols of Fig. 2.

Symbol Name Definition

q1 PELV pitch Pitch angle of pelvis joint
q2 PELV roll Roll angle of pelvis joint
q3 PELV yaw Yaw angle of pelvis joint
q4 RHJC pitch Pitch angle of right hip joint
q5 RHJC roll Roll angle of right hip joint
q6 RHJC yaw Yaw angle of right hip joint
q7 RKNE Pitch angle of right knee joint
q8 LHJC pitch Pitch angle of left hip joint
q9 LHJC roll Roll angle of left hip joint
q10 LHJC yaw Yaw angle of left hip joint
q11 LKNE Pitch angle of left knee joint

h iT
mk ¼ hi h_i h€i Lj ð17Þ

In Eq. (17), h represents joint angle, i is joint number from 1 to

11, L is segment length, and j is segment number from 1 to 4 (see
Fig. 2). The prediction step assumed that each segment length is
constant and joint variables evolve linearly. Thus, F was defined as:

22 2
3 3
1 dt dt2
66 7
...: 0 7
6 4 0 1 dt 5 0 7
6 7
6 0 0 1 7
6 7
6 .. 7
6 7
6 0 ...: 2 ... . 7 ð18Þ
6 3 7
6 1 dt dt2
2
7
6 . 6 7 7
6 .. . . . 4 0 1 dt 5 0 7
6 7
6 7
4 0 0 1 5
0 ... 0 I4 4
Fig. 2. Kinematic model used in this study to compute joint angle.
 S. Chakraborty et al. / Journal of Biomechanics 104 (2020) 109718 5

Where dt is the time difference between samples. In update occlusion problem). Thus, for meaningful comparison, those data
^ k is corrected by measurement. The mea-
phase, predicted state m (for the both devices) were not considered for MLE calculation.
surement vector was composed of 3D marker positions of AR Fisher’s Exact Test (FET) was performed to investigate the relation-
markers as follows: ship (in terms of LDS labels (i.e. ‘‘stable” or ‘‘unstable”)) between
h iT the two systems. Intraclass Correlation Coefficients (ICC) was also
zk ¼ G
plx G
ply G
plz ð19Þ computed to confirm the reliability. All the statistical tests were
performed using SPSS (IBM Corp. Released 2011. IBM SPSS Statis-
Where l and G are marker number and global coordinate respec- tics for Windows, Version 20.0. Armonk, NY: IBM Corp.).
tively. Transformation function h was defined by forward kinemat-
ics using kinematic model of lower limb described in Fig. 2 and its
Jacobian matrix H was computed symbolically. In order to get fea- 4. Result
sible solution including physical constraint, the designed EKF
includes constraints in joint angles as follows: For case 1, it is evident from the results that Kinect-based joint
angles follow almost the similar pattern as that of reference
Cmk 6 d ð20Þ
system-based counterpart (Fig. 4), but differs in terms of amplitude
Where C was set as identity matrix. Constraints in joint angles and phase (combined error  36:25%) (Fig. 5). PELV pitch angle
were set in d. Kalman gain including this constraint is written as exhibited the highest error which was more than 50% (combined
follows: error  85%) for both magnitude and phase, while the combined
 1   1 error for each of the knee joints was the lowest ( 20%). The mean
K Rk ¼ K k  C T CC T ^ kjk  b v Tk S1
Am k vk v Tk S1
k ð21Þ RMSE among all joint angles across all trials was  4 (see supple-
mentary figure sfigure.1). In Fig. 4, the legends, ‘‘upper bound” and
From Eq. (21), Eqs. (15) and (16) are calculated again with con- ‘‘lower bound” shows the upper and lower limit of each joint angle
strained Kalman gain K Rk as follows: respectively. The ROM of knee joint is generally maximum for
walking motion. This seemed to produce large RMSE values for
m ^ k þ K Rk v k
^k ¼ m ð22Þ those joints. But, those large error was estimated within the error
 T of 10% of ROM (seeing from the NRMSE (Fig. 6). On the contrary,
Pk ¼ Pk  K Rk Sk K Rk ð23Þ the RMSE of pitch angle of pelvis was about 3 , but it was esti-
mated in the error of more than 40% of ROM. Especially, pitch
Although Eq. (20) shows constraints for upper bound, constraint angle of pelvis and yaw angle of both hip joints showed high
Kalman gain was also computed for lower bound. This inequality NRMSE which confirms the results of Sprague’s validation metrics.
constraints were implemented using active set method (Gupta These results were also supported by the corresponding PCs (knee
and Hauser, 2007). joints (0.95 (p < 0:01)), PELV pitch (0.27 (p < 0:01))) (see supple-
However, there was a minor gap between the estimated knee mentary table sTable 1).
joint positions by EKF and the reference method (in both cases). In case 2, the deviation of Kinect from the reference system was
Point cloud feature of Kinect was used to correct it. At first, for each much higher (combined error  54:37%) (see supplementary fig-
frame, the knee position estimated by EKF, was extracted. Then, ure sfigure.2 and sfigure.3). Here also like case 1, PELV pitch angle
the region of interest was set around the knee joint and the cen- exhibited the highest error (combined error  94%), while the
troid of point cloud was extracted. According to the principle of combined error for the right knee was the lowest ( 15%). Notice-
depth sensor of Kinect, the location of the centroid was assumed ably, in case 2, the difference between the error (combined error)
on the back surface of knee. Thus, the point located at a distance for the left knee and right knee joints was  22:70%, which was
of  5 cm from the centroid on x axis of the global coordinate 19:40% higher than that of case 1. The mean RMSE ( 10 ) was also
was defined as refined knee joint position. That point was handled higher than case 1 (see supplementary figure sfigure.4). Pitch angle
as virtual knee marker by EKF. Like AR markers, same type of com- of pelvis and yaw angle of both hip joints exhibited high NRMSE
putation was performed on the virtual knee marker by EKF. Joint (see supplementary figure sfigure.5) like case 1. The corresponding
angle time series computed from the two systems were used for PCs (0.94, p < 0:01, for right knee and 0.24, p < 0:01, for PELV
LDS computation. The complete flowchart of the proposed method pitch) (see supplementary table sTable 2) support the above results
is demonstrated in Fig. 3. also.
To accurately quantify LDS, it is needed to acquire kinematic In case of MLE computation for case 1, Fig. 7a and b demonstrate
data in a continuous way over multiple gait cycles (Piórek et al., the histograms of time lag s and embedding dimension m (across all
2017). As per the recommendation of standard literature ((Bruijn the joints and for all trials) for the gold standard system. 36, 38, and
et al., 2013; Toebes et al., 2012)), all time series data were cropped 39 were the most frequent time lag values (in centiseconds)
to 10 strides to make the number of strides same for all the sub- whereas, 4 and 5 were the most dominant values (ca. 87.2% for all
jects. For each time series, the parameters for state space recon- time series) for embedding dimension. These support the results
struction (i.e s and m) were computed separately. Threshold for reported in the literature (Piórek et al., 2017; Dingwell and
false neighbors percentage was set to ‘zero’. All computations were Cusumano, 2000; Dingwell and Marin, 2006; England and Granata,
performed using MATLAB 2017a (The MathWorks, Inc., Natick, 2007). The histogram of s in case of the Kinect (Fig. 8a) seemed to
Massachusetts, United States). differ from the gold standard system, whereas the histogram of m
Relationship between the two systems in terms of the joint (Fig. 8b) was almost similar to its counterpart. In case 2, the his-
angle values was investigated initially. Errors in terms of magni- tograms of both s and m for Kinect differed from the reference sys-
tude and phase of the joint angle time series were computed using tem (see supplementary figure sfigure.6a and sfigure.6b
the validation metrics described by Sprague et al. (Sprague and respectively). The results of the subsequent stages of MLE computa-
Geers, 2003). Root mean square error (RMSE), Normalized RMSE tion (case 1) for the sagittal plane movement of left knee joint of
(NRMSE) (normalized by the range of motion (ROM) of the corre- subject ‘A’ in case of gold standard system is demonstrated in sfig-
sponding joints) and Pearson’s correlation coefficients (PC (pr )) ure.7 (see supplementary figure sfigure.7), whereas for the Kinect
were also computed. Left and right hip transverse plane joint angle sensor, those results for case 1 and 2 are illustrated in sfigure.8
data of Kinect (for case 1 & 2) were detected as an outlier (due to and sfigure.9 (see supplementary figure sfigure.8 & 9) respectively.
6 S. Chakraborty et al. / Journal of Biomechanics 104 (2020) 109718

Fig. 3. Flowchart of the proposed work. (a) Joint angle computation (module 1), and (b) LDS computation (module 2).

The red line on the graphs of time lag and minimum embedding the joint kinematics and LDS. A comparative analysis (in two differ-
dimension indicates the first minimum and the point where FNN ent cases) between the Kinect and gold standard system in terms of
percentage drops to ‘zero’ respectively. The box plots (Fig. 9) which joint angles and MLE was performed. Results demonstrate that the
comprise all combinations of joints and anatomical planes, exhibit efficiency of Kinect to measure the joint angles using RGB-D image
the distribution of MLE values for the Kinect and the reference sys- data stream, depends on its placement, but for LDS assessment, the
tem. Fig. 9 demonstrates that the Kinect sensor deviates significantly sensor is not suitable at all.
from the reference system in almost all joints. Similar result was In terms of joint kinematics, a substantial improvement (in both
observed for case 2 also (see supplementary figure sfigure.10), how- cases) compared to state-of-the-art skeleton tracking system (of
ever, the deviation of Kinect was comparatively higher. On the other Kinect) was observed. The mean RMSE error of Kinect (across hip
hand, Fisher’s Exact Test concluded with the statistically insignifi- and knee joints) reported by Xu et al. (Xu et al., 2015) was 20:15
cant result (p > 0:05) (for both cases) which confirms the null which is far greater than to that by our proposed method ( 4
hypothesis of independent data sets (i.e the MLE values of Kinect for case 1 &  10 for case 2). Our approach also perform better
and the gold standard system are independent to each other). Also, than the method described by Guess et al. (Guess et al., 2017)
low reliability (case 1: ICC ¼ 0:12 & case 2: ICC ¼ 0:09) (across all where the error was 11:5 . In our study, assessment of Kinect
joint angles) for MLE values was observed for the Kinect sensor. was performed on 3-dimensional joint kinematics which is often
avoided by the comparative state-of-the-art (e.g (Xu et al., 2015;
Pfister et al., 2014)) where only a single dimension was considered.
5. Discussion Again, the use of only 4 AR markers instead of a set of retro-
reflective markers made our system cost-effective as well as sub-
This multi-objective study investigated the suitability of RGB-D ject friendly.
image data stream of Kinect (v2) sensor when using it to measure
 S. Chakraborty et al. / Journal of Biomechanics 104 (2020) 109718 7

Fig. 4. Comparison between joint angles of two systems (case 1).

Fig. 5. Bar chart (error graph) of Sprague’s validation metrics across all the joints (case 1).

In case 1, for most of the joints, Kinect sensor seemed to follow parallel to yaw axis. It was assumed that these condition results
almost similar pattern as those for the reference system. Flexion in low PC and high NRMSE error. On the other hand, the yaw angle
and extension angles of knee and hip were almost similar for both of both hip joints seems to be affected mostly either by the self-
systems (Fig. 4). Also, a considerable agreement was observed for occlusion problem of Kinect or the marker placement strategy.
hip abduction and adduction. However, for PLEV pitch, a remark- Approximately 0 yaw angle was observed for the both hips
able difference was observed. This may be due to the geometrical throughout the time series which is quite abnormal. Hence, these
conditions by the placement of four AR markers. Two markers on two angles were treated as outlier. In case 2, errors for most of
both side of pelvis are lined up parallel to pitch axis. When the the joints were higher than case 1 (see supplementary figure sfig-
knee was stretched, both pelvis and heel markers were lined up ure.2). This might be due to the lateral placement (with respect to
8 S. Chakraborty et al. / Journal of Biomechanics 104 (2020) 109718

Fig. 6. NRMSE values of joint angles (case 1).

Fig. 7. Histogram (case 1) of (a) time delay s, and (b) embedding dimension m - the gold standard system. Histograms are distributed into 4 bins.

Fig. 8. Histogram (case 1) of (a) time delay s, and (b) embedding dimension m - Kinect (v2) camera. Histograms are distributed into 4 bins.

subjects) of Kinect which might have increased the self-occlusion of dissimilarity (in percentage) across the different joints (in both
problem. During a gait cycle, the right limb occluded the AR mark- cases) between the two systems. Though the average combined
ers of the other limb (determined from the tracking movie). Higher dissimilarity is low in case 1 (36:25%), it seemed to affect the
NRMSE and lower PC of the left knee compared to the right knee LDS at a great extent. A similar effect was observed for case 2 also
(see supplementary figure sfigure.5 and table sTable 2) also con- (see supplementary figure sfigure.10) where the average combined
firm this hypothesis. dissimilarity (of joints) was 18:12% higher than case 1. Even the
Although the mean RMSE error of Kinect (in case 1) is clinically knee joints which exhibited the highest between system agree-
insignificant (McGinley et al., 2009), it seemed to affect the cycle- ment (in terms of joint angles) in case 1, demonstrated a very dif-
to-cycle mean divergence substantially which is the central con- ferent distribution in case of LDS assessment (Fig. 9). The
cept of MLE estimation. This assumption is supported by the measurement LDS is closely related to the problem of detection
Fig. 7(a) and Fig. 8(a) which demonstrate a different pattern of dis- of deterministic chaos in the gait cycle (Piórek et al., 2017) which
tribution of s for both systems. The result of Sprague’s validation is characterized by its extreme sensitivity to the initial conditions.
metrics (Sprague and Geers, 2003) indicates a changeable degree Hence, the MLE which literally measures the average exponential
 S. Chakraborty et al. / Journal of Biomechanics 104 (2020) 109718 9

Fig. 9. Box plots of MLE values for the gold standard system and the Kinect sensor (case 1). L hip f: Frontal plane of left hip joint, L hip s: Sagittal plane of left hip joint, R hip f:
Frontal plane of right hip joint, R hip s: Sagittal plane of right hip joint, knee l s: Sagittal plane of left knee joint, knee r s: Saigittal plane of right knee joint, Pelvis f: Frontal
plane pelvis joint, Pelvis s: Sagittal plane pelvis joint, Pelvis t: Transverse plane pelvis joint, Mok: Optotrack motion capture camera (gold standard camera).

rate of divergence of initially adjacent trajectories may be influ- due to the unavailability constraints, our experiment did not
enced highly by the presence of marginal error between the joint include pathological subjects as participants.
angle time series of the two systems.
In this experiment, only a single Kinect was used which has
6. Conclusion
failed to track (for some frames) accurately the AR markers on
the heel (determined from the tracking movie) due to its non-
The present study examined the accuracy of RGB-D image data
recognizable orientation during walking. Again, lateral placement
stream of the Kinect (v2) sensor in measuring joint kinematics and
of Kinect has increased the partial self-occlusion problem. This
the LDS in two different cases. Joint positions were estimated using
might have led to the erroneous time series. Comparison between
RGB-D image data stream and point cloud feature of Kinect. Joint
the RGB-D image and skeleton data streams of Kinect for joint
angles were computed using EKF. The kinematic time series was
kinematics and LDS could be performed using specialized software
used to compute the LDS. For the purpose of validation, the results
application (Cunha et al., 2016) and an alternate experimental pro-
of the Kinect sensor were compared with the gold standard cam-
tocol which is beyond the scope of the current study. Furthermore,
eras. The experimental results reveal the reasonable accuracy level
10 S. Chakraborty et al. / Journal of Biomechanics 104 (2020) 109718

of Kinect RGB-D image data stream in measuring the joint kine- van Emmerik, R.E., Ducharme, S.W., Amado, A.C., Hamill, J., 2016. Comparing
dynamical systems concepts and techniques for biomechanical analysis. J. Sport
matics but, low efficiency in the determining the LDS (in both
Health Sci. 5, 3–13.
cases). Also, placement of Kinect seemed to be an important factor England, S.A., Granata, K.P., 2007. The influence of gait speed on local dynamic
in estimating the joint kinematics. In future work, investigation on stability of walking. Gait Posture 25, 172–178.
multiple Kinects may be performed to reduce the tracking obscu- Futamure, S., Bonnet, V., Dumas, R., Venture, G., 2017. A sensitivity analysis method
for the body segment inertial parameters based on ground reaction and joint
rity problem. Again, simultaneous assessment of different data moment regressor matrices. J. Biomech. 64, 85–92.
streams of Kinect could be performed using multiple Kinects. Garrido-Jurado, S., Muñoz-Salinas, R., Madrid-Cuevas, F.J., Marín-Jiménez, M.J.,
Examination of LDS and joint kinematics in different walking 2014. Automatic generation and detection of highly reliable fiducial markers
under occlusion. Pattern Recogn. 47, 2280–2292.
speeds and with pathological population is also warranted in Guess, T.M., Razu, S., Jahandar, A., Skubic, M., Huo, Z., 2017. Comparison of 3d joint
future research. angles measured with the kinect 2.0 skeletal tracker versus a marker-based
motion capture system. J. Appl. Biomech. 33, 176–181.
Gupta, N., Hauser, R., 2007. Kalman filtering with equality and inequality state
Declaration of Competing Interest constraints. arXiv preprint arXiv:0709.2791.
Kantz, H., 1994. A robust method to estimate the maximal lyapunov exponent of a
The authors have no conflicts of interest related to this study. time series. Phys. Lett. A 185, 77–87.
Kennel, M.B., Brown, R., Abarbanel, H.D., 1992. Determining embedding dimension
for phase-space reconstruction using a geometrical construction. Phys. Rev. A
Acknowledgment 45, 3403.
Liebert, W., Schuster, H., 1989. Proper choice of the time delay for the analysis of
chaotic time series. Phys. Lett. A 142, 107–111.
We would like to be extremely thankful to Department of
McGinley, J.L., Baker, R., Wolfe, R., Morris, M.E., 2009. The reliability of three-
Science and Technology (DST), Govt. of India and Japan Society dimensional kinematic gait measurements: a systematic review. Gait Posture
for the Promotion of Science (JSPS) to support this research work 29, 360–369.
(File No. DST/INT/JSPS/P-246/2017). Mentiplay, B.F., Perraton, L.G., Bower, K.J., Pua, Y.-H., McGaw, R., Heywood, S., Clark,
R.A., 2015. Gait assessment using the microsoft xbox one kinect: Concurrent
validity and inter-day reliability of spatiotemporal and kinematic variables. J.
Appendix A. Supplementary data Biomech. 48, 2166–2170.
Nebehay, G., Micusik, B., Picus, C., Pflugfelder, R., 2011. Evaluation of an online
learning approach for robust object tracking. AIT Austrian Institute of
Supplementary data associated with this article can be found, in Technology, Technical Report AIT-DSS-TR-0279.
the online version, at https://doi.org/10.1016/j.jbiomech.2020. Pawelzik, K., Schuster, H., 1987. Generalized dimensions and entropies from a
measured time series. Phys. Rev. A 35, 481.
109718.
Pfister, A., West, A.M., Bronner, S., Noah, J.A., 2014. Comparative abilities of
microsoft kinect and vicon 3d motion capture for gait analysis. J. Med. Eng.
References Technol. 38, 274–280.
Piorek, M., 2015. Chaotic properties of gait kinematic data. In: IFIP International
Austin, G.P., 2001. Motor control of human gait: a dynamic systems perspective. Conference on Computer Information Systems and Industrial Management.
Orthopaedic Physical Therapy Clinics of North America. Springer, pp. 111–119.
Bizovska, L., Svoboda, Z., Janura, M., Bisi, M.C., Vuillerme, N., 2018. Local dynamic Piórek, M., Josiński, H., Michalczuk, A., Świtoński, A., SzczČŠsna, A., 2017.
stability during gait for predicting falls in elderly people: A one-year Quaternions and joint angles in an analysis of local stability of gait for
prospective study. PloS One 13. e0197091. different variants of walking speed and treadmill slope. Inf. Sci. 384, 263–
Boeing, G., 2016. Visual analysis of nonlinear dynamical systems: Chaos, fractals, 280.
self-similarity and the limits of prediction. Systems 4, 37. Punt, M., Bruijn, S.M., van Schooten, K.S., Pijnappels, M., van de Port, I.G., Wittink, H.,
Bradley, E., Kantz, H., 2015. Nonlinear time-series analysis revisited. Chaos: An van Dieën, J.H., 2016. Characteristics of daily life gait in fall and non fall-prone
Interdisciplinary. J. Nonlinear Sci. 25. 097610. stroke survivors and controls. J. Neuroeng. Rehabilitat. 13, 67.
Brown, T.A., 1996. Measuring chaos using the lyapunov exponent. Chaos Theory in Rosenstein, M.T., Collins, J.J., De Luca, C.J., 1993. A practical method for calculating
the Social Science, pp. 53–66. largest lyapunov exponents from small data sets. Physica D 65, 117–134.
Bruijn, S., Meijer, O., Beek, P., Van Dieën, J., 2013. Assessing the stability of human Shulin, L., Haifeng, Z., Hui, W., Rui, M., 2007. Application of improved emd algorithm
locomotion: a review of current measures. J. Roy. Soc. Interface 10, 20120999. for the fault diagnosis of reciprocating pump valves with spring failure. In:
Capecci, M., Ceravolo, M.G., Ferracuti, F., Iarlori, S., Longhi, S., Romeo, L., Russi, S.N., ISSPA 2007. 9th International Symposium on Signal Processing and Its
Verdini, F., 2016. Accuracy evaluation of the kinect v2 sensor during dynamic Applications, 2007. IEEE, pp. 1–4.
movements in a rehabilitation scenario. In: 2016 IEEE 38th Annual International Simon, D., 2006. Optimal State Estimation: Kalman, H Infinity, and Nonlinear
Conference of the Engineering in Medicine and Biology Society (EMBC). IEEE, pp. Approaches. John Wiley & Sons.
5409–5412. Speedtsberg, M.B., Christensen, S.B., Stenum, J., Kallemose, T., Bencke, J., Curtis, D.J.,
Carey, N., 2005. Establishing Pedestrian Walking Speeds. Portland State University. Jensen, B.R., 2018. Local dynamic stability during treadmill walking can detect
do Carmo Vilas-Boas, M., Choupina, H.M.P., Rocha, A.P., Fernandes, J.M., Cunha, J.P.S., children with developmental coordination disorder. Gait Posture 59, 99–103.
2019. Full-body motion assessment: Concurrent validation of two body tracking Sprague, M.A., Geers, T.L., 2003. Spectral elements and field separation for an
depth sensors versus a gold standard system during gait. J. Biomech. 87, 189– acoustic fluid subject to cavitation. J. Comput. Phys. 184, 149–162.
196. Takens, F., 1981. Detecting strange attractors in turbulence. In: Dynamical systems
Cunha, J.P.S., Rocha, A.P., Choupina, H.M.P., Fernandes, J.M., Rosas, M.J., Vaz, R., and turbulence, Warwick 1980. Springer, pp. 366–381.
Achilles, F., Loesch, A.M., Vollmar, C., Hartl, E. et al., 2016. A novel portable, low- Toebes, M.J., Hoozemans, M.J., Furrer, R., Dekker, J., van Dieën, J.H., 2012. Local
cost kinect-based system for motion analysis in neurological diseases. In: 2016 dynamic stability and variability of gait are associated with fall history in
38th Annual International Conference of the IEEE Engineering in Medicine and elderly subjects. Gait Posture 36, 527–531.
Biology Society (EMBC). IEEE, pp. 2339–2342. Wolf, A., Swift, J.B., Swinney, H.L., Vastano, J.A., 1985. Determining lyapunov
Davis III, R.B., Ounpuu, S., Tyburski, D., Gage, J.R., 1991. A gait analysis data exponents from a time series. Physica D 16, 285–317.
collection and reduction technique. Human Movement Sci. 10, 575–587. Xu, X., McGorry, R.W., Chou, L.-S., Lin, J.-H., Chang, C.-C., 2015. Accuracy of the
Dingwell, J.B., Cusumano, J.P., 2000. Nonlinear time series analysis of normal and microsoft kinect for measuring gait parameters during treadmill walking. Gait
pathological human walking. Chaos: An Interdisciplinary. J. Nonlinear Sci. 10, Posture 42, 145–151.
848–863. Zhang, Z., 2000. A flexible new technique for camera calibration. IEEE Trans. Pattern
Dingwell, J.B., Marin, L.C., 2006. Kinematic variability and local dynamic stability of Anal. Machine Intell. 22, 1330–1334.
upper body motions when walking at different speeds. J. Biomech. 39, 444–452.