sensors

Article
A Foot-Arch Parameter Measurement System Using a
RGB-D Camera
Sungkuk Chun 1 ID
, Sejin Kong 2 , Kyung-Ryoul Mun 3 and Jinwook Kim 3, *
1 Spatial Optical Information Research Center, Korea Photonics Technology Institute, Gwangju 61007, Korea;
k612051@kopti.re.kr
2 R&D Division, LS Networks, Seoul 04386, Korea; sjkong@lsnetworks.com
3 Center for Imaging Media Research, Korea Institute of Science and Technology, Seoul 02792, Korea;
krmoon02@gmail.com
* Correspondence: jwkim@imrc.kist.re.kr; Tel.: +82-2-958-6776

Received: 12 June 2017; Accepted: 2 August 2017; Published: 4 August 2017

Abstract: The conventional method of measuring foot-arch parameters is highly dependent on the
measurer’s skill level, so accurate measurements are difficult to obtain. To solve this problem, we
propose an autonomous geometric foot-arch analysis platform that is capable of capturing the sole
of the foot and yields three foot-arch parameters: arch index (AI), arch width (AW) and arch height
(AH). The proposed system captures 3D geometric and color data on the plantar surface of the foot in
a static standing pose using a commercial RGB-D camera. It detects the region of the foot surface in
contact with the footplate by applying the clustering and Markov random field (MRF)-based image
segmentation methods. The system computes the foot-arch parameters by analyzing the 2/3D shape
of the contact region. Validation experiments were carried out to assess the accuracy and repeatability
of the system. The average errors for AI, AW, and AH estimation on 99 data collected from 11 subjects
during 3 days were −0.17%, 0.95 mm, and 0.52 mm, respectively. Reliability and statistical analysis
on the estimated foot-arch parameters, the robustness to the change of weights used in the MRF,
the processing time were also performed to show the feasibility of the system.

Keywords: biomedical image processing; RGB-D camera; foot-arch; arch width; arch height; arch
index; computer aided analysis

1. Introduction
The foot-arch, which plays a key role in supporting the weight of the body and providing
propulsion force during push-off, is important because it enables a more natural and aesthetic gait and
protects the foot from injury. It is well-known that filling the void space between the foot-arch and
shoe reduces the plantar pressure, alleviates impact force, and improves shoe comfort [1,2]. Therefore,
understanding the geometric shape of an individual’s foot and foot-arch is necessary to provide
direct and useful information, not only for clinical and rehabilitative purposes, but also for designing
personalized and comfortable footwear [3].
The arch index (AI), the arch width (AW), and the arch height (AH) are the representative
parameters showing the foot characteristics of healthy individuals as well as subjects with foot
functional abnormalities. These parameters are defined based on the shape of the footprint. The AI is
defined by the ratio of the midfoot area to the entire foot area (excluding the toes) from the measured
footprint [4]. The AW and the AH are defined as the vertical and horizontal distances from the
midpoint of the medial border line (MBL), which is the line connecting the most medial border of the
metatarsal and heel region of the foot [5], in the arch region of the footprint to the foot surface. Based
on these definitions, the traditional methods measure these parameters manually using a footprint on

Sensors 2017, 17, 1796; doi:10.3390/s17081796 www.mdpi.com/journal/sensors

Sensors 2017, 17, 1796 2 of 26

a grid paper. Therefore, the accuracy of the measurement is dependent on the measurement skill of the
experimenter, and reliability and repeatability are thus usually poor.
In order to overcome this problem, many scientific researchers have attempted to measure and
analyze the foot shape via vision-based measurement (VBM) approaches. VBM involves producing
measurement results using 2D/3D visual images and computational methods. The main idea of
this approach in the context of biometric measurement of geometric shape of human body parts is
that a camera or a scanner captures the human body surface and measurement is done by analyzing
the captured visual and geometric data using hardware and software technologies [6]. Recent rapid
advances in imaging devices and computing systems have allowed VBM to be easily used to measure,
detect, monitor, recognize, and record physical phenomena in a variety of automated applications and
scenarios: measurement of human body parts [7], human motion tracking [8], hand recognition [9],
face recognition [10], gait analysis [11], palmprint-based identifications [12], and 3D human body
reconstruction [13].
In this study, we aimed to develop an autonomous geometric foot-arch measurement platform
that is capable of capturing the sole of the foot and estimating three foot-arch parameters: AI, AW and
AH. The proposed system captures 3D geometric and color data on the plantar surface of the foot in
a static standing pose using a commercial RGB-D camera installed below the transparent footplate.
As explained in [4,14,15], three foot-arch parameters can be calculated based on the foot axis and
the MBL in the footprint image representing the contact region of foot. Therefore, in this paper, we
describe the process of detecting the contact region of foot and computing the foot-arch parameters
using the 3D geometric and color data obtained from the RGB-D camera and validate the results of the
estimated foot-arch parameters from human experiment. The main contributions of this work are:

• A new methodology for sole of foot analysis: The proposed system, which automatically analyzes the
plantar surface of the foot in a static standing pose, utilizes a commercial RGB-D camera installed
below the transparent acrylic plate of the scanning stage. Some existing methods using separately
designed wearable devices or visual markers involved adding an extra factor to the foot surface,
but this is undesirable in a clinical environment. Also, some use camera-projector systems or
multiple cameras to reconstruct the surface of the foot. However, these are more expensive than
commercial off-the-shelf RGB-D cameras and require heavy computational processing. In contrast,
our system can measure the plantar surface of the foot efficiently by using an RGB-D camera,
which provides accurate 3D geometric and visual information.
• An automatic foot-arch parameter computation method: To define and recognize numerically the
characteristics of individual feet, the system automatically calculates the foot-arch parameters—AI,
AW, and AH. The system detects the contact region from an input color and depth image set by
applying image segmentation methods such as data clustering and Markov Random Field (MRF)
techniques, and generates the foot-arch parameters by analyzing the 2D and 3D shape of the
contact region. In contrast to other existing systems that focus mainly on surface reconstruction
and footprint image generation, our system is capable of not only capturing the sole of the foot
but also determining the three foot-arch parameters.

2. Related Literature
VBM-based foot shape analysis systems can be classified into passive and active 3D shape
measurement techniques, depending upon the sensing method used [16]. The first is based on
matching of corresponding pixels in multiple images captured by multiple cameras. The second
involves measurement of the 3D shape by emitting and receiving light. As one of the earliest passive
3D shape measurement systems, Lee et al. proposed a foot model reconstruction using 12 cameras [17].
From the 12 captured images, the system calculates major foot parameters, such as foot length and
ball width, to scale the foot model in their database. Reconstruction is completed by morphing and
deforming the foot models in the database similar to the user’s foot. Coudert et al. proposed a six
cameras-based method of measuring 3D foot deformation during the gait cycle [18]. The final foot
Sensors 2017, 17, 1796 3 of 26

model is reconstructed by image matching and triangulation techniques. Amstutz et al. used ten
cameras to reconstruct the 3D foot shape [19]. They fitted the initial 3D model to the real foot by
projecting the initial model to the 2D images of the real foot. In [20], Al-Baghdadi et al. applied a dense
surface modeling algorithm to automatically obtain a 3D point cloud of a human foot. A mounting
platform for three video cameras and a glass-top step-on platform were used to capture the foot.
Alshadli et al. introduced a video based 3D foot modeling system to map the shape changes of the
medial longitudinal arch during gait [21]. The imaging system they proposed consists of 4 high
definition video camcorders and a force plate. Using this system, multiple images and force data
synchronized with the camcorders can be captured simultaneously. The 3D foot shape is reconstructed
by using multiple images. In [22], Alshadli proposed the method to calculate the foot mobility
magnitude (FMM) and arch height index (AHI) using the system proposed in [21]. And the author
tried to find the relationship between the dynamic FMM and AHI and the foot posture index (FPI).
These camera-based reconstruction methods have limitations in that a texture of the foot surface for
correspondence matching between multiple images is required for an accurate foot shape acquisition.
To overcome these limitations, they proposed special socks or painting of the texture on the foot skin.
Although these solutions were suitable for their reconstruction process, they are undesirable in a
clinical environment and not comfortable.
The active shape measurement methods typically acquire 3D shape by the structured light method
or time-of-flight (ToF) method. The structured light method calculates the distance by using the shape
and location of the projected pattern. The ToF method calculates the distance by measuring the
time-of-flight of a light signal to the subject. These methods are widely used, as users do not need to
wear or attach sensors or markers to their feet. In [23,24], they introduced a camera-projector system
to reconstruct the plantar surface. A pattern comprising small squares of random colors is projected
onto the foot sole and the reflected patterns are captured by the camera; these are used to reconstruct
the 3D geometric shape of the sole. JezerĹAek ˛ et al. presented a multiple laser plane triangulation
technique for high-speed foot measurement [16]. They used four measuring modules, each of which
comprised a laser projector and a digital camera. In [25], Novak et al. developed a 3D foot scanning
system with a rotational laser-based measuring head. In their system, the measuring head, comprising
a three laser line projection unit and two cameras, rotates around the center of the platform on which
the customer stands, and measures both feet simultaneously. Herrewegen et al. used four structured
light scanning modules to capture the foot shape [26]. To analyze multi-segmental foot kinematics,
the proposed system tracks four segments (shank, calcaneus, metatarsus, and hallux) during walking
using the iterative closest point (ICP). Using the ToF camera-based foot shape measurement system
proposed in [27], Samson et al. proposed a new method of analyzing foot roll-over under dynamic
conditions [28]. The system generates sequential images of lowest height data (LHD), which represent
the distance from the 3D foot shape to the ground plane by projecting the foot surface. For each frame
during foot roll-over motion, the change in mean height and projected surface in seven regions of
interest (ROI) are computed. In [29], Chen et al. proposed a hand held RGB-D camera based 3D foot
parameter estimation method. The user firstly rotates around the foot and captures the consecutive
color and depth images using a hand held RGB-D camera. The system reconstructs the 3D shape
of the foot with reference to the AR code located around the feet. And then using the 3D shape of
foot, the system calculates the foot length, width, and ball girth. Even though their system uses 3D
shape information and provides useful information for suitable shoe selection, they do not provide any
essential information to calculate AI, AW, and AW such as the contact region of foot. Although some
existing systems can measure the entire surface of the foot and some provide analytic information
related to foot measurement, such as the deformation of cross-sections [23], global foot dimensions
(foot length, width, height, and girth) [25,29], and changes in mean height and projected surface [28],
the studies do not include any information regarding the AI, AW, and AH which can be used for
clinical purposes and design of ergonomic personalized footwear.
Sensors 2017, 17, 1796 4 of 26

In this paper, among many features of the foot, we focus on automatic estimation of three foot-arch
parameters, AI, AW, and AH. The AI is measured by counting the squares in the graph paper with
which the sole of foot is in contact. The AW and AH are measured manually using a ruler. These
conventional methods are highly dependent on the measurer’s skill and undesirable in a clinical
aspect since the subject must cover his/her foot in ink to create the footprint image. In [30], Chu et al.
proposed an image processing-based system to improve the accuracy and repeatability of the footprint
acquisition and AI calculation of the traditional method. However, it is impossible to compute and
predict the AI, AW, and AH simultaneously using the footprint-based method. This is because the
footprint does not contain 3D information of the foot, such as the height of the plantar surface—it
represents simply the 2D contact region of the sole of the foot.
The system proposed in this paper is a commercial RGB-D camera-based foot-arch measurement
system for AI, AH, and AW computation. Compared with the passive 3D measurement methods,
the system is easily able to obtain 3D shape of sole of foot from the geometric and visual data captured
by the RGB-D camera, without special socks or painting the pattern on the foot skin. Another advantage
that the proposed system includes that the used camera is compact, relative low cost, and the fast
data acquisition frame rate. The frame rate and image resolution of the system in [28] are 40 Hz and
176 × 144. The frame rate and image resolution of the used camera in the proposed system are 60 Hz
and 640 × 480. Even though the system in [23] has higher image resolution (1024 × 768) than the
proposed system, the frame rate is slow (14 Hz). In [29], they use the low-cost RGB-D camera similar
with the RGB-D camera used in the proposed system. However, the frame rate and image resolution
are relatively low as 30 Hz and 320 × 240, and also their system cannot capture the plantar shape
of the foot. In addition, unlike existing VBM studies that analyze the overall shape change of the
foot, the proposed system calculates AI, AW, and AH that reflect the characteristics of the foot-arch.
Therefore, for the foot-arch parameters estimation, the proposed system has advantages of better
usability and convenience than the existing systems.
In particular, the conventional footprint-based foot-arch parameter measurement method
depends on manual operation, but the proposed system efficiently computes them through an image
segmentation technique and computational 3D shape analysis of foot. Moreover, since the proposed
system is able to estimate the AI, AW, and AH simultaneously using the advantages of the 3D shape
information of foot, it is more convenient to measure than the conventional method. And also another
advantage of the proposed system compared with the 2D footprint based method is the calculation
time. The conventional method takes an average of 3 min or more, from painting ink on the foot to
calculating the AI based on the inked area of the footprint and measuring the AH using a ruler or a
caliper. The 2D digital image processing based method proposed in [30] takes 10 s or more to calculate
AI. However, the proposed system is able to calculate not only AI but also AW and AH simultaneously
within 10 s (Section 5.5). The other advantage is that the proposed method can be applied to the
dynamic foot motion analysis. The camera used in the proposed system can capture shape and color
data for continuous foot motion. This can be used to analyze how the user's feet change in the stance
phase of the gait. For example, successive 3D shape and color information of foot can be obtained to
analyze how foot shape changes at each stage of the stance phase, which consists of heel strike, foot
flat, midstance, heel off, and toe off. However, 2D footprint based technologies cannot be employed
for dynamic foot analysis, since the acquired data does not reflect the 3D dynamic foot shape change.
In this system, we focus on the static foot measurement, and do not dynamic foot analysis.

3. Foot-Arch Parameters
To identify the important considerations in system development, foot-arch parameters must be
examined. In this section, we briefly describe the considerations of our system with respect to the
computational method for estimation of foot-arch parameters.
The AI developed by Cavanagh and Rogers represents the ratio of the area of the middle third of
a footprint relative to the total area (excluding the toes) [4]. The definition of AI is as follows: The line
Sensors 2017, 17, 1796 5 of 26

Sensors 2017, 17, 1796 5 of 24

of the foot axis between the centers of the heel (point K in Figure 1a) and the second toe is measured.
Next,
of theafootprint.
perpendicular line is drawn
The intersection tangential
point to the most
is then marked (pointanterior point
L in Figure of The
1a). the main
line L-Kbody of the
is divided
footprint.
into threeThe intersection
equal parts. The point is then
main body ofmarked (pointisLdivided
the footprint in Figure 1a).
into Theareas
three line by
L-Kthose
is divided
points into
with
three equal parts. The main body of the footprint is divided into three areas by those points
the perpendiculars from the foot axis. The AI, the ratio of the middle area relative to the total with theis
area,
perpendiculars
computed [14]. from the foot axis. The AI, the ratio of the middle area relative to the total area, is
computed [14].

(a) (b)
Figure1.1. Definitions
Figure Definitions of
of foot-arch
foot-arch parameters:
parameters:(a)
(a)arch
archindex
index(AI)
(AI)measurement
measurementfrom thethe
from footprint, (b)
footprint,
arch width (AW) and height (AH) measurement.
(b) arch width (AW) and height (AH) measurement.

The AW and AH are clinically important, as they are closely related to foot type. Low arches or
The AW and AH are clinically important, as they are closely related to foot type. Low arches or
flat feet can cause heel pain, arch pain and plantar fasciitis. High arches can cause plantar fasciitis as
flat feet can cause heel pain, arch pain and plantar fasciitis. High arches can cause plantar fasciitis as
the plantar fascia is stretched away from the calcaneus or heel bone. The AW and AH are generally
the plantar fascia is stretched away from the calcaneus or heel bone. The AW and AH are generally
measured using the footprint and a ruler, as follows. The MBL is first drawn. Then, a perpendicular
measured using the footprint and a ruler, as follows. The MBL is first drawn. Then, a perpendicular
line is drawn from the mid-point of the MBL in the arch area to the mid-foot. The length of this line is
line is drawn from the mid-point of the MBL in the arch area to the mid-foot. The length of this line is
the AW [15]. The AH is defined as the length of a perpendicular line from the mid-point of the MBL
the AW [15]. The AH is defined as the length of a perpendicular line from the mid-point of the MBL to
to the plantar surface of the foot. Figure 1b shows the lines and points required to measure AW and
the plantar surface of the foot. Figure 1b shows the lines and points required to measure AW and AH.
AH.
To calculate the AI, AW, and AH, the region of the foot in contact with the floor, which can be
To calculate the AI, AW, and AH, the region of the foot in contact with the floor, which can be
easily measured using the footprint, must be defined. Also, the key points and lines, such as the center
easily measured using the footprint, must be defined. Also, the key points and lines, such as the
of the heel and the second toe and the MBL, are required. Therefore, from the next section, we describe
center of the heel and the second toe and the MBL, are required. Therefore, from the next section, we
how the proposed system calculates the foot-arch parameters by solving the following two technical
describe how the proposed system calculates the foot-arch parameters by solving the following two
problems: how to recognize the region in contact with the footplate using the color and depth image
technical problems: how to recognize the region in contact with the footplate using the color and
set and how to define the key points and lines.
depth image set and how to define the key points and lines.
4. Method
4. Method
The system consists of measurement and analysis modules. The measurement module includes
The system consists of measurement and analysis modules. The measurement module includes
the scanning stage and a RGB-D camera underneath transparent acrylic board mounted on the
the scanning stage and a RGB-D camera underneath transparent acrylic board mounted on the
scanning stage (Section 4.1). And the analysis module is programmed to process the obtained
scanning stage (Section 4.1). And the analysis module is programmed to process the obtained color
color and geometric data of foot and to extract the foot-arch parameters through the following
and geometric data of foot and to extract the foot-arch parameters through the following three
three submodules: pre-processing (Section 4.2), contact region detection (Section 4.3), and foot-arch
submodules: pre-processing (Section 4.2), contact region detection (Section 4.3), and foot-arch
parameter computation (Section 4.4). The system flow is shown in Figure 2.
parameter computation (Section 4.4). The system flow is shown in Figure 2.
Sensors 2017, 17,
Sensors 2017, 17, 1796
1796 66 of
of 26
24
Sensors 2017, 17, 1796 6 of 24

Figure
Figure 2. System flow
2. System flow of
of the
the proposed
proposed system.
system.
Figure 2. System flow of the proposed system.
4.1. Measurement Module
4.1. Measurement Module
4.1. Measurement Module
The
The scanning
scanning stage used to capture the plantar surface
surfaceof ofthe
thefoot is is
45 cmcm in in
height and 70 70
cm in
The scanningstage
stageused
usedto tocapture
capture the the plantar
plantar surface of the foot
foot is 4545cm height
in height andand
70 cmcm
width.
in in
width.A transparent
AA transparent acrylic board of 45 cm × 35 cm is embedded at the middle of scanning stage.
width. transparentacrylic
acrylicboard
board of of 45
45 cm
cm × 35 cm
× 35 cm isis embedded
embeddedatatthe themiddle
middle ofof scanning
scanning stage.
stage.
This
This acrylic board is used as the footplate where the targeted foot is measured on. For stable lighting
Thisacrylic
acrylicboard
boardisisusedused as as the
the footplate
footplate where
where the the targeted
targeted foot footisismeasured
measuredon.on.ForFor stable
stable
condition,
lighting a led desk lamp is installed inside of the scanning stage. A RealSense F200 RGB-D camera
lightingcondition,
condition,a aled leddesk
desk lamp
lamp is is installed inside of
installed inside of the
the scanning
scanningstage. stage.AARealSense
RealSense F200F200
(Intel,
RGB-D
RGB-D
Santa
camera Clara,
camera(Intel,
CA, Santa
USA)Clara,
(Intel,Santa
is installed
Clara,CA, CA, USA)
30 cm
USA) is
beneath
installedthe
is installed 30cm
30
footplate
cm beneath
beneath
tothe
measure
the the
footplate
footplate
sole
toto
of thethe
measure
measure
foot
the
(Figure
sole of 3).
the foot (Figure
sole of the foot (Figure 3).3).
The
The camera
The camera
camera captures
captures and
captures sends
and
and sends the the
sends input data data
the input in thein
data form
in theof
the a color
form
form ofofaand depth
acolor
color and image
and setimage
depth
depth containing
image setset
visual and
containing
containing geometric
visual
visualand information
andgeometric on
geometricinformationthe plantar
information on the surface
the plantarof the
plantarsurfacefoot to
surfaceofofthe a server
thefoot via a USB
foottotoa server
a servercable.
viavia aUsing
a USB USB
the data
cable. obtained,
Using the the
datasystem analyzes
obtained, the the
system foot shape
analyzes in
theterms
foot of the
shape contact
cable. Using the data obtained, the system analyzes the foot shape in terms of the contact region,in terms region,
of the arch index,
contact width,
region,
and
arch height.
arch index,
index, These
width,
width, outputs
and can be
andheight.
height. Thesetransferred
These outputs to
outputs canother
be foot analysis
be transferred
transferred systems.
totoother
otherfoot footanalysis
analysis systems.
systems.

(a) (b) (c)

(a) (b) (c)
Figure
Figure 3. 3. System
System installation:(a)
installation: (a)scanning
scanning stage
stage of
of the
the proposed
proposedsystem,
system,(b)
(b)RGB-D
RGB-Dcamera beneath
camera beneath
Figure 3. System acrylic
the transparent installation:
board,(a)
(c)scanning stage
sole of foot of the proposed
measurement system,
using our (b) RGB-D camera beneath
system.
the transparent acrylic board, (c) sole of foot measurement using our system.
the transparent acrylic board, (c) sole of foot measurement using our system.
4.2. Pre-Processing Module
4.2. Pre-Processing Module
4.2. Pre-Processing Module
The preprocessing module (PM) performs noise removal measured from an input depth image
The preprocessing module (PM) performs noise removal measured from an input depth image
andThemapspreprocessing
the input color moduleimage(PM)to theperforms noise Each
depth image. removal
pixelmeasured
in an input from an image
depth input depth image
represents
and maps the input color imagetotothe the depthalong
image. Each pixel inof
an input depth image represents
and themaps the input
distance from thecolor image to the
camera depth image.
surface theEach pixel
optical axisin antheinput depth
camera, and image represents
can easily be
theconverted
distance into
from3D thepoints
camera to the
( ,to, the surface
). However, along the
directthe optical
useoptical axis
of the axis
depthof the camera, and can easily be
the distance from the camera surface along of image are notand
the camera, recommended
can easily be
converted into 3Dnoise,
due to optical
converted
points
into 3D points ( x,(depth
lost y, z). However,
, , ).information
However,on
directshiny
use ofsurfaces,
direct use
the depth
of the and
image
depth
are notartifacts.
flickering
image are
recommended due to
To increase
not recommended
optical noise,
thetoquality lost depth information on shiny surfaces, and flickering artifacts. To increase the quality
due opticaland stability
noise, of the information
lost depth depth image,onthe PM surfaces,
shiny reduces the and noise in the artifacts.
flickering input depth To image
increase
and
the quality and stability of the depth image, the PM reduces the noise in the input depthmedian
stability
using median of the depth
filtering to image,
preserve thefoot
PMedges
reduces theremoving
while noise in noise
the input
[31]. depth
In the image using
proposed system, a
image
3 ×
filtering3 to preserve
sliding window footisedgesused, while
and removing
the median noise
depth [31].
value Inamong
the proposed
nine system,
pixels in the 3 × 3 sliding
a window is
using median filtering to preserve foot edges while removing noise [31]. In the proposed system, a
window is the
used, and the
3 ×taken as
3 sliding filtered
window ismedian
depth value.
used, depth value among nine pixels in the window is taken as the filtered
and the median depth value among nine pixels in the window is
depth
taken as value.
To use visual and geometric information simultaneously, their coordinate systems must be
the filtered depth value.
unified
To since the color and the depth images captured from theirthe RGB-D camera have
mustdifferent
To use
usevisual
visualandandgeometric
geometric information
information simultaneously,
simultaneously, coordinate
their systems
coordinate systems bemust
unified
be
coordinates.
since the color The
and calibration
the depth process captured
images is necessaryfromto find
the the intrinsic
RGB-D camera parameters
have of each
different camera,
coordinates.
unified since the color and the depth images captured from the RGB-D camera have different
The such as lens distortion
calibration and focal length, as well
the as extrinsic parametersofrepresenting relative
suchposes
coordinates. Theprocess is
calibration necessary
process to find
is necessary intrinsic
to find parameters each camera,
the intrinsic parameters of each as lens
camera,
such as lens distortion and focal length, as well as extrinsic parameters representing relative poses
Sensors 2017, 17, 1796 7 of 26

distortion and17,focal
Sensors 2017, 1796 length, as well as extrinsic parameters representing relative poses between 7 oftwo
24
cameras. In our system, the camera is calibrated using the method proposed in [32]. Using the
betweenintrinsic
calculated two cameras. In our system,
and extrinsic the camera
parameters, is calibrated
the system mapsusing the method
the input proposed
color image to thein depth
[32].
image. As a result, a pixel in the depth image becomes a 6D point ( x, y, z, r, g, b), where (x, y, z) is to
Using the calculated intrinsic and extrinsic parameters, the system maps the input color image the
3D the depthofimage.
position As ainresult,
the point a pixel
the depth in thecoordinate,
camera depth image becomes
and (r, g, b)ais6D color( information
thepoint , , , , , ),mapped
where
( , , ) is the 3D position of the point in the depth camera coordinate, and ( , , ) is the color
from the input color image.
information mapped from the input color image.
Additionally, the system first filters out the points whose distances from the camera are more than
Additionally, the system first filters out the points whose distances from the camera are more
the predefined threshold (600 mm in this system) to remove the points which does not corresponds to
than the predefined threshold (600 mm in this system) to remove the points which does not
the foot-point (Figure 4c). then retains only the largest component by applying connected component
corresponds to the foot-point (Figure 4c). then retains only the largest component by applying
labeling to remove
connected other unnecessary
component minor noisy
labeling to remove otherpoints [33]. Connected
unnecessary minor noisycomponent labeling
points [33]. involves
Connected
identifying
component all connected components
labeling involves in an image
identifying and assigning
all connected a unique
components in label to all pixels
an image in the same
and assigning a
component. The computation of size, position, orientation and bounding rectangle
unique label to all pixels in the same component. The computation of size, position, orientation can be carriedandout
using the result
bounding labels. The
rectangle can besystem applies
carried the labeling
out using algorithms
the result labels. Theto detect
system foot pointsthe
applies by labeling
retaining
only the largest
algorithms tocomponent and filtering
detect foot points out the
by retaining others.
only the largest component and filtering out the others.

(a) (b)

(c) (d)
Figure
Figure 4. Results
4. Results of of preprocessing
preprocessing module:(a)
module: (a)input
inputcolor
color image,
image, (b)
(b) input
input depth
depthimage
imagecolor-coded
color-coded
by depth value, (c) depth image filtered by depth thresholding, (d) foot point image filtered by the
by depth value, (c) depth image filtered by depth thresholding, (d) foot point image filtered by the
connected component labeling and color mapped image (left bottom).
connected component labeling and color mapped image (left bottom).

To find the largest component as the group of foot points, the system first converts the filtered
To find
depth the to
image largest
a binarycomponent
image. Inas thethe group
binary of foot
image, points, the system
two-pass-based connectedfirst component
converts the filtered
labeling
depth image to a binary image. In the binary image, two-pass-based connected
is applied. In the first pass, the system scans the binary image left to right and top to bottom. If the component labeling
is applied.
pixel is 1,Inthe
thesystem
first pass, the asystem
assigns label toscans the binary
the pixel image
as follows: (1) left to left
If the right andistop
pixel 1 andto bottom.
the top pixelIf the
pixel is assign
is 0, 1, the system
the labelassigns a label
of the left pixel.to(2)
theIfpixel
the topas pixel
follows: (1) Ifthe
is 1 and theleft
leftpixel
pixel is is
0, 1assign
and the
the top
labelpixel
of
is 0,the
assign the label
top pixel. (3) Ifofthe
thetop
left pixel.
and (2) If the
left pixels are top pixel the
1, assign is 1label
and the leftleft
of the pixel is 0,pixel.
or top assign the two
If the label
of the topofpixel.
labels (3)and
the left If the
toptop andare
pixels leftdifferent,
pixels are 1, assign
record thetwo
that the label of the
labels areleft or top pixel.
equivalent. In theIfsecond
the two
pass,
labels of the
the system
left andassigns a unique
top pixels label to all
are different, pixelsthat
record of athe
component
two labels using the lowest label
are equivalent. In theforsecond
each
equivalent set. Using the number of pixels in each label, the system finds the largest component,
Sensors 2017, 17, 1796 8 of 26

pass, the system assigns a unique label to all pixels of a component using the lowest label for each
equivalent set. Using the number of pixels in each label, the system finds the largest component,
and defines the corresponding pixels as the foot points. Figure 4d shows the detected foot points and
depth information. Invalid points are filtered out and only the foot points are retained.

4.3. Contact Region Detection Module

The foot-arch parameters are computed based on the contact region. To identify the contact region,
the system first detects the contact points and then defines the contact region by a Markov random
fields (MRF)-based method.

4.3.1. Contact Point Detection

A contact point is defined as a point whose distance from the footplate (acrylic board) is ideally
zero. If the RGB-D camera is ideally installed parallel with the footplate, the distance between the foot
point and the footplate could be calculated as the difference between the depth value of a point in
the depth image and the distance of the footplate from the camera. However, it is almost impossible
to install the camera in such a way, and the camera is generally rotated. Therefore, recognizing the
rotation and translation of the footplate in 3D space is required to calculate the distance from the foot
point to the footplate. One of the simplest ways to recognize the footplate is to model the plane using,
for example, a checkerboard. However, this plane modeling method must be performed whenever
the camera installation is changed. To overcome this problem, the proposed system automatically
estimates the footplate from the previously filtered foot points based on the following assumption:
among the foot points in the depth image, those in the contact region comprise the greatest proportion
and their surface normal vectors are identical. Using this assumption, the system calculates the unit
normal vectors of the foot points, clusters them, and finds the largest cluster. Finally, the system
estimates the plane equation using the points included in the detected cluster.
The point neighborhood-based surface normal vector estimation method is used in our system [34].
This approach first computes two 3D vectors between the left and right neighboring points and between
the upper and lower neighboring points in the depth image. The normal vector is computed using the
cross product of the two vectors. This method is considerably faster, but sensitive to noise. To alleviate
this problem, the proposed system applies a smoothing filter to the normal vectors by averaging them.
As mentioned above, we assume that most of the points are in the contact region and their surface
normal vectors have similar directions. To identify the point set with similar normal vectors, the system
performs clustering of the normal vectors [35]. The system constructs a 3D voxel grid and votes the
normal vectors to the corresponding grid cell. In our system, each side of the 3D grid has a range of
−1.0 to 1.0, and the grid consists of 20 × 20 × 20 cells. The size of each cell is 0.1 × 0.1 × 0.1. Each normal
vector is voted according to its x-y-z coordinates. All non-empty cells are initial clusters. The system
calculates the average normal vector of each initial cluster. To alleviate the discretization effects on
the cell size, the system merges neighboring clusters and updates the average normal vector, if the
difference between their averages is smaller than the cell size. After the merging process, the points in
the largest cluster are used to compute the plane equation estimation. Figure 5a shows the results of
our normal vector clustering algorithms.
Sensors 2017, 17, 1796 9 of 26
Sensors 2017, 17, 1796 9 of 24

(a) (b)

(c)
Figure
Figure 5.
5. Contact
Contact point
point detection
detection process:
process: (a)
(a) foot
foot points
points included
included in
in the
the largest
largest normal
normal vector
vector cluster
cluster
(green points),
(green points),(b)(b) detected
detected contact
contact points
points (green(green points),
points), (c) number
(c) number of corresponding
of corresponding points
points according
according to distance from
to distance from the plane. the plane.

To detect the contact points among the foot points, the system exploits the distance from the
The plane equation is calculated using the least-squares plane fitting algorithms proposed
estimated footplate to the point. The distance can be easily calculated using the estimated plane
in [36]. A plane equation can be specified by a point (x0 , y0 , z0 ) on the plane and the normal vector
equation: =
(T − ̅ ) + ( − ) + ( − ̅). Ideally, the distances of all contact points would
n = n , n , nz of the plane. Any point ( x, y, z) on the plane satisfies n x ( x − x0 ) + ny (y − y0 ) +
be zero,x buty some are not zero due to the noise of the input data and the estimation error in the
nz (z − z0 ) = 0. The best-fit plane to m given data points ( xi , yi , zi ), where m ≥ 3, passes through the
plane-fitting process. For this reason, the system identifies the set of contact points based on the
centroid ( x, y, z) of the data, and this specifies a point on the plane. The first step is to find the average
following assumption: among the foot points, the points in the contact region comprise the greatest
( x, y, z) of the points. A matrix A is formulated such that its first column is xi − x, second column is
proportion among the foot points. The system calculates the distance from each point first and then,
yi − y, and third column is zi − z. Then the matrix A is decomposed by singular value decomposition.
finds the distance to the footplate of the largest number of the foot points. Finally, the system
The singular vector corresponding to the smallest singular value is chosen as the normal vector n
defines the foot points with distances less than the detected distance as the contact points. Figure 5b
of the plane. Finally the plane with the best fit to the given data points is specified by x, y, z, and n:
shows the recognized contact points and Figure 5c shows the change in the number of foot points
n x ( x − x ) + ny (y − y) + nz (z − z) = 0.
according to distance. Algorithm 1 explains the procedures of contact point detection.
To detect the contact points among the foot points, the system exploits the distance from the
estimated footplate to the point. The distance can be easily calculated using the estimated plane
Algorithm 1 Procedures of Contact Point Detection
equation: d = n x ( x − x ) + ny (y − y) + nz (z − z). Ideally, the distances of all contact points would
Input data: foot points in the input depth image
be zero, but some are not zero due to the noise of the input data and the estimation error in the
Output data: contact points
plane-fitting process. For this reason, the system identifies the set of contact points based on the
Variables
following assumption: among the foot points, the points in the contact region comprise the greatest
- : index of the foot points in the depth image
proportion among the foot points. The system calculates the distance from each point first and then,
- : i-th foot point in the depth image
finds the distance to the footplate of the largest number of the foot points. Finally, the system defines
- : normal vector of
the foot points with distances less than the detected distance as the contact points. Figure 5b shows the
- : distance from the footplate to
recognized contact points and Figure 5c shows the change in the number of foot points according to
- _ : length of one side of a cell in the 3D voxel grid
distance. Algorithm 1 explains the procedures of contact point detection.
Procedures:
1. Constructing a 3D voxel grid:
create a 3D voxel grid having a range of −1.0 to 1.0 and divide the grid into cells of equal
Sensors 2017, 17, 1796 10 of 26

Algorithm 1 Procedures of Contact Point Detection

Input data: foot points in the input depth image
Output data: contact points
Variables
- i: index of the foot points in the depth image
- f i : i-th foot point in the depth image
- ni : normal vector of f i
- di : distance from the footplate to f i
- c_size: length of one side of a cell in the 3D voxel grid
Procedures:

1. Constructing a 3D voxel grid:

create a 3D voxel grid having a range of −1.0 to 1.0 and divide the grid into cells of equal size.
2. Computing a normal vector:

for all fi ,

calculate the normal vector ni and apply a smoothing filter.

3. Voting normal vectors:

for all ni ,

assign ni to the cell in the grid

4. Setting initial clusters:

set the non-empty cells as the initial clusters and calculate the average normal vector in the initial cluster.
5. Merging adjacent clusters:

for all inital clusters,

calculate the distance between the adjacent cluster and merge them if the distance is smaller than c_size.
6. Estimating footplate equation:

find the cluster with the largest number of normal vectors of the foot points and estimate the plane
equation using the correspoindng foot points.
7. Calculating distances of foot points:

for all fi ,

calculate the distance from the footplate using the plane equation.
8. Constructing a histogram:

create a hisgtorm of the distance from the foot points and the footplate.
9. Detecting the contact points:

find the bin with the highest frequency and define the points corresponding to the bin as the
contact points.

4.3.2. Contact Region Detection

As mentioned above, the foot-arch parameters are computed based on the contact region in the
plantar surface of the foot. Therefore, the system must define the contact region before computing
the foot-arch parameters. In this module, the proposed system extracts the dense and connected set
of points in contact with the footplate. To solve this problem, a MRF is applied to detect the contact
region [37]. A MRF enables incorporation of shape, color and distance from the footplate cues in a
single unified model. Using the MRF, the system segments the foot point image into the following
three classes: the contact, non-contact, and background regions. The contact and non-contact region
Sensors 2017, 17, 1796 11 of 26

are respectively the set of foot points in contact and not in contact with the footplate. The background
region is the non-foot points in the image.
Given a color image x, set of distances d, and set of included angles between the normal vectors
of the points and the normal vector of estimated footplate θ, the energy function of the proposed MRF
model for the class labels c is defined as:
 
∑ wψ ψ ∑



E(c, π, w, x, d, θ) = ci , xi ; πψ + wλ λ(ci , di ; πλ ) + wρ ρ ci , θi ; πρ + φ c i , c j , xi , x j , (1)
i (i,j)∈Υ

where Υ is the set of the edges in the four-connected grid, π = πψ , πλ , πρ are the model parameters,

w = wψ , wλ , wρ are the weights for each term, and i and j are the index nodes in the grid
(corresponding to positions in the image).
As shown in Figure 6a, the contact and non-contact regions in the foot point set have different
colors due to skin deformation caused by the effect of body weight on the contact region. The first term


ψ ci , xi ; πψ , known as the color potential, is based on this idea and represents the color distribution
of the class given the point color. This term is proportional to the likelihood of the input color xi given
ci , and is defined as:


ψ ci , xi ; πψ = − log P(xi |ci ), (2)
Sensors 2017, 17, 1796 11 of 24
where P(xi |ci ) is the normalized distribution given by the Gaussian mixture models (GMM) using
learned
where parameters
( | ) is the . A RGB color
πψnormalized model is given
distribution used for the Gaussian
by the color potential.
mixture models (GMM) using
The second term
learned parameters λ (
.Ac , d
i RGB i ; π )
color
λ , known as the distance potential,
model is used for the color potential. captures the distance distribution
of the class
The given
secondtheterm
distance ( ,of the; )point.
, knownThis term
as theis based on the
distance notion that
potential, many contact
captures points are
the distance
closer to the footplate than are the non-contact points, as shown in Figure 6b. This
distribution of the class given the distance of the point. This term is based on the notion that many term is proportional
tocontact
the likelihood ofcloser
points are the input
to the distance
footplate given
dithan arecithe
, and is definedpoints,
non-contact as: as shown in Figure 6b. This
term is proportional to the likelihood of the input distance given , and is defined as:
λ(ci , di ; πλ ) = − log P(di |ci ), (3)
( , ; ) = −log ( | ), (3)
whereP(d(i |c|i ) )isisthe
where thenormalized
normalized distribution givenby
distribution given byaasingle
singleGaussian
Gaussian model
model using
using thethe learned
learned
parameter
parameter λ .
π .

(a) (b)

(c) (d)
Figure 6. Example of MRF-based contact region detection: (a) color image of the foot points (b)
Figure 6. Example of MRF-based contact region detection: (a) color image of the foot points
visualization of distance between the foot points and the estimated footplate, (c) visualization of the
(b) visualization of distance between the foot points and the estimated footplate, (c) visualization
angle between the normal vectors of the footplate and the points, (d) MRF-based segmentation (red:
of the angle between the normal vectors of the footplate and the points, (d) MRF-based segmentation
contact region, green: non-contact region, and white: background).
(red: contact region, green: non-contact region, and white: background).

The third term , ; , known as the angle potential, captures the angle distribution of the
class given the included angles between the normal vectors of the points and the normal vector of
the footplate. The included angle can be calculated easily from the inner product of two normal
vectors. This term is based on the idea that many contact points have normal vectors similar to the
normal vector of the footplate, as shown in Figure 6c. This term is proportional to the likelihood of
Sensors 2017, 17, 1796 12 of 26

The third term ρ ci , θi ; πρ , known as the angle potential, captures the angle distribution of the
class given the included angles between the normal vectors of the points and the normal vector of the
footplate. The included angle can be calculated easily from the inner product of two normal vectors.
This term is based on the idea that many contact points have normal vectors similar to the normal
vector of the footplate, as shown in Figure 6c. This term is proportional to the likelihood of the input
angle θi given ci , and is defined as::

ρ ci , θi ; πρ = − log P(θi |ci ) and θi = cos−1 n p · ni ,

(4)

where P(θi |ci ) is the normalized distribution given by a single Gaussian model using the learned
parameter πρ , and n p and ni are the normal vectors of the footplate and the point, respectively.


The last term φ ci , c j , xi , x j , known as the smoothness potential, is a prior distribution that
discourages large differences in the labels of neighboring sites by assigning a low value to these
configurations. We model this term using a contrast-sensitive Potts model, as follows:
 
φ ci , c j , xi , x j = I ci 6= c j exp − βkxi − x j k2 ,



(5)

where I(·) is an indicator function that is 1 (0) if the input argument is true (false), and β is a fixed
parameter. The color difference between two neighboring points is used in this smoothness potential.
In practice, this enforces spatial continuity of labels since the output of the term becomes large if the
color difference is small and the corresponding labels are not identical.
Each of the potential terms is trained separately to produce a normalized model parameter.
The training sample of each class is chosen from the result of contact point detection. The samples for
the contact region class are randomly selected from the contact points. The samples for the non-contact
region class are randomly selected from the non-contact points among the foot points. However, some
points in the contact region are classified as non-contact points since the result point set of the contact
point detection is sparse, as shown in Figure 5b. To prevent selection of points in the contact region as
the samples for the non-contact region class, the system applies dilation to the contact points in the
result image, removes them, and selects the samples for the non-contact region class from among the
remaining foot points. The samples for the background region class are set to fixed values: [0, 0, 0]T ,
◦
0, and 180 for the color, distance, and angle potentials, respectively. The system adds small noises
to them. For the color potentials, an Expectation-Maximization (EM) algorithm is used to learn the
GMM parameter πψ . We use three Gaussian models to train the color distribution of a single class.
For the distance and angle potentials, we train the Gaussian models of each class using the mean and
standard deviation. Compared with the contact points in Figure 5b, the recognized contact region is
dense (Figure 6d). The weights, wψ , wλ , and wρ , for color, distance, and angle potentials are set to 2.9,
9.3, and 1.8, respectively.

4.4. Foot-Arch Parameter Computation Module

The foot-arch parameters—AI, AW, and AH—can be defined by using the contact region and
the key points and lines, such as centroid points of the second toe and heel and the MBL. In this
section, we describe computation of foot-arch parameters by 2D and 3D shape analysis of the estimated
contact region.

4.4.1. Arch Index Computation

To compute the AI from the estimated contact region, the system performs following three
processing: (1) foot axis definition, (2) toe removal, (3) AI computation.
The system defines the foot axis based on the two key points, the centers of the second toe and
heel. Some of existing method presented the automatic foot axis detection [28,29]. Their methods
compute the first principal axis of foot point distribution and define it as the foot axis. However, these
Sensors 2017, 17, 1796 13 of 26

methods are not suitable for finding the correct foot axis, since the extracted foot axis based on these
methods is generally not the line connecting the centers of the second toe and heel. And since the foot
data distribution depends on posture changes, the resulting foot axis also is sensitive to the posture
changes. In order to accurately define the foot axis in the proposed system, the center of the second toe
is manually selected by user input using a graphical user interface (GUI).
The center of the heel (point K in Figure 1a) is automatically defined using the boundary-tracing
technique. In [38], Chun et al. proposed a 3D human pose representation method based on the salient
points in the silhouette boundary. To detect the salient points, they first calculated the distances from the
centroid of the silhouette to the boundary points by boundary tracing clockwise or counter-clockwise
and applied a smoothing filter to the sequentially detected distances. In the sequential distances,
the system finds and defines the local maxima as the salient points. Our system performs boundary
tracing on the heel side image of the foot center and detects the salient points. Before this process,
to recognize the directions to the toe side and heel side, the system aligns the contact region image
based on the principle axis computed by PCA, as shown in Figure 7a. The system splits the contact
region into two parts based on the centroid. For both parts, the system calculates the maximum width
of the contact region perpendicular to the first principal axis. The part with a greater maximum width
is defined as the toe side part, and the other is defined as the heel side part. Finally, among the salient
points detected in the heel side part, the point farthest from the centroid is selected as the center of the
heel (red point in Figure 7b). The line connecting the centers of the second toe and heel is considered
the Sensors
foot axis
2017,(Figure
17, 1796 7c). Algorithm 2 explains the foot axis detection procedures. 13 of 24

(b)

(a) (c)
Figure
Figure 7. Key
7. Key point
point detection
detection process:
process: (a) (a) contact
contact region
region alignment
alignment using
using PCA,
PCA, (b) (b) center
center of heel
of heel
detection using boundary tracing, (c) definition of the foot axis as a line between the central points of of
detection using boundary tracing, (c) definition of the foot axis as a line between the central points
the the
heelheel
andand second
second toe.toe.

Algorithm 2 Procedures of Foot Axis Detection

Input data: contact region
Output data: foot axis connecting the centers
Variables
- : centroid of the contact region
- : index of the boudnary point in the heel side part
- : i-th boudnary point in the heel side part
- : distance between and
- : smoothed distance between and
′
- : first derivvatnormal vector of
- : index of the salient point
- : j-th salient point
Procedures
Sensors 2017, 17, 1796 14 of 26

Algorithm 2 Procedures of Foot Axis Detection

Input data: contact region
Output data: foot axis connecting the centers
Variables

- cn: centroid of the contact region

- i: index of the boudnary point in the heel side part
- bi : i-th boudnary point in the heel side part
- li : distance between cn and bi
- lˆi : smoothed distance between cn and bi
- l 0 i : first derivvatnormal vector of lˆi
- j: index of the salient point
- s j : j-th salient point

Procedures

1. Aligning the contact region:

calculate the centroid of contact region cn and rotate the contact region by using PCA
2. Defining the heel side part:

2.1. devide the aligned contact region into two parts.

2.2. for each part, find maximum width perpendicular to the first principal axis.
2.3. define a part with a larger maximum width as a heel side part.
3. Calculating the distances from cn to the boundary points in the heel side part:

for all bi ,

calculate distance li from cn. Smoothing the distances:

for all li ,

compute the smoothed distance lˆi using a 1D Gaussian smoothing filter.

4. Deriving the derivative of the smoothed distances:

for all lˆi ,

compute l 0 i the first derivative of the smoothed distance lˆi using the central difference method.
5. Detecting the salient points:

for all lˆi ,

find the local maximum of which l 0 i is a zero-crossing point, and define the local maximum as the salient
point s j .
6. Detecting the center of heel:

for all s j ,

find the farthest point from cn and define it as the center of heel.
7. Defining the foot axis:

detect the line connecting the detected center of heel and the center of second toe selected by a user.

The AI is the ratio of the area of the middle third of the main body of the contact region to the
entire contact region, excluding the toes. The next step is to segment the foot point image into the main
body part of the foot and toe area. The edge detection technique is applied for this work, based on the
significant color change between the main foot body and toe area due to skin deformation and the
shadow, as shown in Figure 6a. To detect the color change, the system first rotates the foot point image
 The AI is the ratio of the area of the middle third of the main body of the contact region to the
entire contact region, excluding the toes. The next step is to segment the foot point image into the
main body part of the foot and toe area. The edge detection technique is applied for this work,
based on the significant color change between the main foot body and toe area due to15skin
Sensors 2017, 17, 1796 of 26
deformation and the shadow, as shown in Figure 6a. To detect the color change, the system first
rotates the foot point image based on the foot axis, and divides the image into three parts. The
based on
system the foot
scans axis, andeach
horizontally divides
line the image
in the toeinto threeand
region, parts. The the
detects system scans
vertical horizontally
edges × 3
using 3 each
line in the toe region, and detects the vertical edges using 3 × 3 Sobel operator (Figure
Sobel operator (Figure 8a). And in order to find the strongest edge per each scan line, the system 8a). And in
order to find
performs thethe strongest edge
non-maximal per each scan
suppression. line,system
The the system
thenperforms
applies the
the non-maximal suppression.
smoothing filter to the
The system
positions of then applies
detected the by
edges smoothing
averaging filter
thetopositions
the positions of detected
of their edges
neighboring by averaging
edges. Using thesethe
positions of their neighboring edges. Using these edges, the main part and
edges, the main part and toe area are divided (Figure 8b). And by applying AND operation to the toe area are divided
(Figure 8b).main
segmented Andpartby applying AND operation
and the contact region, the to system
the segmented main
detects the partbody
main and part
the contact region,
of the contact
the system
region. detects the main body part of the contact region.

(b)

(a) (c)
Figure
Figure 8.
8. AI
AI computation:
computation: (a)
(a) foot
foot point
point image
image rotated
rotated by
by the
the foot
foot axis
axis (top)
(top) and
and the
the detected
detected edge
edge
(colored
(colored green) (bottom), (b) toe part and main body part segmented by the edge, (c) result of AI
green) (bottom), (b) toe part and main body part segmented by the edge, (c) result of AI
computation
computation from
from the
the segmented
segmented main
main body
body part
part of
of the
the contact
contact region.
region.

Finally, the system computes the AI by dividing the detected contact region into three parts
Finally, the system computes the AI by dividing the detected contact region into three parts and
and calculating the ratio of the area of the middle part to the entire region. Figure 8c shows the
calculating the ratio of the area of the middle part to the entire region. Figure 8c shows the example of
example of the AI computation.
the AI computation.
4.4.2.
4.4.2. Arch
Arch Width
Width and
and Height
Height Computation
Computation
The
The AW and AH are defined as
AW and AH are defined as the
the lengths
lengths of
of the
the lines
lines from
from the
the mid-point
mid-point of
of the
the MBL
MBL to
to the
the
contact region and
contact region and to
tothe
thefoot
footpoint
pointin in a direction
a direction perpendicular
perpendicular to footplate.
to the the footplate. Therefore,
Therefore, the
the MBL
must be measured first. The AW and AH are computed after detecting the mid-point of the MBL.
To find the MBL, the system exploits the convex hull detection algorithm [39]. This algorithm
finds the convex hull of a 2D point set. The convex hull is the smallest set that contains the entire 2D
point set. Through this algorithm, the system obtains the set of contour lines that contains the points
in the contact region image (Figure 9a). The line with the greatest depth among the contour lines is
defined as the MBL (Figure 9b). The mid-point of the MBL is computed by averaging the two 3D
points; i.e., the beginning and end points of the line.
The AW and AH are calculated by defining the two lines perpendicular to the MBL at the
mid-point of the MBL. The line for the AW estimation is on the plane (the estimated footplate),
begins at the mid-point of the MBL, and intersects with the contact region. The system draws a line
perpendicular to the MBL on the contact region image and defines the distance between the mid-point
of the MBL to the point on the line intersecting with the contact region as the AW (Figure 9b).
The line for the AH begins at the mid-point of the MBL and intersects with the foot point in
a direction perpendicular to the footplate. The system first estimates the line equation using the
mid-point of the MBL and the normal vector of the footplate. Using the line equation, the system
begins at the mid-point of the MBL, and intersects with the contact region. The system draws a line
perpendicular to the MBL on the contact region image and defines the distance between the
mid-point of the MBL to the point on the line intersecting with the contact region as the AW (Figure
9b).
The line for the AH begins at the mid-point of the MBL and intersects with the foot point in a
Sensors 2017, 17, 1796 16 of 26
direction perpendicular to the footplate. The system first estimates the line equation using the
mid-point of the MBL and the normal vector of the footplate. Using the line equation, the system
the foot
finds the footpoint
pointclosest
closesttotothe
theline,
line,and
and defines
defines thethe distance
distance between
between thethe closest
closest footfoot point
point andand
the
the mid-point
mid-point of MBL
of the the MBL as AH
as the the AH (Figure
(Figure 9c). 9c).

(a) (b)

(c)
Figure 9.
Figure 9. AW
AWandandAHAH computation:
computation: (a) (a) result
result of convex
of convex hull detection
hull detection algorithm
algorithm appliedapplied to the
to the contact
contact region image, (b) the medial border line (MBL) and the mid-point of the MBL
region image, (b) the medial border line (MBL) and the mid-point of the MBL detection and AW detection and
AW computation, (c) 3D visualization of the AW and AH computation.
computation, (c) 3D visualization of the AW and AH computation.

5. Experiments
5. Experiments
In this
In this section,
section, we
we present
present experimental
experimental results
results to
to show
show that
that the
the proposed
proposed system
system facilitates
facilitates
accurate and
accurate and stable
stable foot-arch
foot-arch parameter
parameter measurement
measurement in in repeat
repeat trials.
trials. In
In Section
Section 5.2, we show
5.2, we show thethe
accuracy and repeatability of the foot-arch parameters estimated by the proposed
accuracy and repeatability of the foot-arch parameters estimated by the proposed system. And in system. And in
Section 5.3, we show the reliability of the proposed method compared with the
Section 5.3, we show the reliability of the proposed method compared with the ground truth and ground truth and
analyze statistically
analyze statisticallythe
themeasured
measured data.
data. In In Section
Section 5.4,5.4, we discuss
we discuss the weight
the weight choicechoice
for ourfor
MRFourmodel
MRF
model used in the contact region detection module. In Section 5.5, the processing
used in the contact region detection module. In Section 5.5, the processing time of each module time of eachis
described. Before reporting the experimental results, we first explain the experimental setting.

5.1. Experiment Environment

The experiments to test the feasibility of the proposed system were performed on a 2.83 GHz CPU
with 4 Gbyte of memory running 64-bit Windows 8.1 Enterprise K. The Intel Realsense Development
Kit was used to capture the color and depth images. OpenCV library was used for image processing,
and the Matlab wrapper for graph cut was used to infer the result of MRF-based labeling [40]. Eleven
adult volunteers, nine males and two females, aged 25~35 years, participated in the validation.
All participants were healthy, with no history of surgery or abnormality that could affect their stand
pose. The subjects were asked to stand on the scanning stage and put their right foot on the acrylic
board. For each subject, we collected three sets of depth and color images at three different times
during a single day. We asked them to return to repeat the above scanning procedure for 3 consecutive
days, thus a total of 99 image datasets were obtained.
To evaluate the accuracy and repeatability of our system, we generated ground truth data for
comparison. All the measurements for the ground truth data were performed by a clinical professional,
who has more than four years of experience. We first painted the subject’s right foot with ink. Then,
right foot on the acrylic board. For each subject, we collected three sets of depth and color images at
three different times during a single day. We asked them to return to repeat the above scanning
procedure for 3 consecutive days, thus a total of 99 image datasets were obtained.
To evaluate the accuracy and repeatability of our system, we generated ground truth data for
comparison. All the measurements for the ground truth data were performed by a clinical
Sensors 2017, 17, 1796 17 of 26
professional, who has more than four years of experience. We first painted the subject’s right foot
with ink. Then, the subject placed his/her painted right foot on graph paper (Figure 10a). Using this
the subjectimage,
footprint placedwe his/her painted
measured theright
AI byfoot on graph
counting paper (Figure
manually 10a). Using
the number this footprint
of painted squares image,
on the
we
footprint image and the AW using a ruler. The AH data were collected by measuring image
measured the AI by counting manually the number of painted squares on the footprint and
the height
the
fromAW theusing
floora to
ruler.
the The
skinAH data were
externally collected
covering thebymetatarsal
measuringbones
the height
usingfrom the floor
a ruler or a to the skin
caliper, as
externally covering the metatarsal bones using a ruler or a caliper, as shown in Figure
shown in Figure 10b. We have collected 11 sets of ground truth data on the foot-arch parameters 10b. We have
collected 11 sets ofFor
from 11 subjects. ground truth dataground
the complete on the foot-arch
truth set, parameters
the AI was from 11 subjects.
23.59~31.14%, theFor
AW thewas
complete
28~41
ground
mm andtruth the AHset, was
the AI was
8~16 mm.23.59~31.14%, the AW was 28~41 mm and the AH was 8~16 mm.

(a) (b)
Figure 10.
Figure 10. Measuring
Measuring ground
ground truth
truth data:
data: (a)
(a) generation
generation of
of aa footprint
footprint for
for AI
AI and
and AW measurement,
AW measurement,
(b) AH
(b) AH measurement.
measurement.

5.2. Accuracy
5.2. Accuracy and
and Repeatability
Repeatability
The accuracy
The accuracy waswasevaluated
evaluatedby bycomparing
comparingthe thesimilarity
similarityofofthe
the measured
measured foot-arch
foot-arch parameters
parameters to
to the ground truth, and the repeatability was determined according to the
the ground truth, and the repeatability was determined according to the similarity of the data similarity of the data
taken at
taken at different time points
different time points over 3 days. over 3 days.
Table 11 shows
Table shows the
the results
results ofof the
the accuracy
accuracy and
and repeatability
repeatability tests.
tests. We
We performed
performed the the foot-arch
foot-arch
parameter computation on 99 datasets collected from 11 subjects over 3 days,
parameter computation on 99 datasets collected from 11 subjects over 3 days, and compared with and compared with
the ground
the ground truth
truth by
by calculating
calculating thethe average
average error
error (AE).
(AE). For
For the
the repeatability
repeatability test,
test, we
we computed
computed the the
standard deviation (STD) of nine datasets for each subject. For AI computation,
standard deviation (STD) of nine datasets for each subject. For AI computation, average of AE and average of AE and
STD were
STD were −−0.17%
0.17% and
and 0.70%.
0.70%. ForForAWAW andand AH,
AH, averages
averages ofof AEs
AEs were
were 0.95
0.95 mm
mm andand 0.52
0.52 mm
mm andand the
the
averages of
averages of STDs
STDs were
were1.63
1.63mm mmand and0.68
0.68mm,
mm,respectively.
respectively.TheThecoefficients
coefficientsofof variation
variation (CV,
(CV, mean
mean ÷
÷ standard deviation) that is widely used to express the precision and repeatability
standard deviation) that is widely used to express the precision and repeatability are 0.023% for AI, are 0.023% for
AI, 0.046%
0.046% for AW,
for AW, and and
0.061%0.061%
for AHfor respectively.
AH respectively.

Table 1. Accuracy
Table 1. Accuracy and
and repeatability
repeatability assessment.
assessment.

AI (%) AW (mm) AH (mm)

Subject
GT AE STD GT AE STD GT AE STD
1 29.06 1.63 0.89 29 1.86 1.77 8 −1.06 0.51
2 31.14 −0.40 0.37 28 −0.86 2.49 8 0.95 0.63
3 25.27 −1.35 1.12 30 0.67 0.99 11 1.93 0.87
4 27.90 −0.81 0.96 37 1.71 1.93 12 2.06 0.56
5 23.59 −1.29 0.28 28 −0.65 0.57 9 0.57 0.84
6 26.66 0.95 0.36 40 1.38 2.90 12 0.49 0.56
7 28.26 0.60 0.48 41 1.12 1.21 11 0.38 0.45
8 25.86 −1.34 0.85 35 1.76 1.97 14 −1.00 1.15
9 26.73 −0.02 0.79 38 2.24 0.41 16 1.29 1.07
10 29.30 −0.21 1.23 41 −0.80 1.93 10 0.42 0.43
11 27.75 0.33 0.38 39 2.05 1.74 11 −0.31 0.44
AVG 27.41 −0.17 0.70 35.09 0.95 1.63 11.09 0.52 0.68
CV(%) 0.023 0.046 0.061
GT: Ground Truth, AE: Average Error, STD: Standard Deviation, AVG: Average, CV: Coefficient of Variation.
Sensors 2017, 17, 1796 18 of 26

5.3. Reliability and Statistical Analysis

In this section, we investigate statistically the results of the proposed system to evaluate the
reliability and feasibility. To compare the reliability of the estimated foot-arch parameters, correlation
coefficients were calculated between the AI, AW, AH and the ground truth. And Spearman’s rank order
correlation was also performed between the estimated results and the ground truth. The relationships
among the foot-arch parameters, which are still controversial, were also tested.
The correlation coefficients and Spearman’s rank order correlations between the AI, AW, AH
estimated by the proposed system and the ground truth were calculated to evaluate the reliability of
the proposed method [41]. The correlation coefficients of AI, AW, and AH are 0.798, 0.851, and 0.811
respectively. All the p-value are less than 0.0001. These results show that the foot-arch parameters
are significantly correlated with the ground truth. In case of the Spearman’s rank order correlations,
the average AI, AW, AH of each subjects measured through the proposed system were used to rank.
The result correlation coefficients of AI, AW, and AH are 0.945, 0.923, and 0.95 respectively. All the
p-value are less than 0.0001. These results suggest that there are strong positive relationships between
the foot-arch parameter measurements collected using the conventional method and the proposed
method. Tables 2–4 list the individual subject rankings of AI, AW, AH for the ground truth and the
proposed method. In case of AI and AH, all of the absolute differences between the two ranks are less
than or equal to 2. In case of AW, the absolute difference between two ranks is less than 2 in 10 out of
11 subjects.

Table 2. Individual subject rankings based on AI.

Ground Truth Proposed Method

Subject Rank Difference
AI Rank AI Rank
5 23.59 1 22.3 1 0
3 25.27 2 23.92 2 0
8 25.86 3 24.52 3 0
6 26.66 4 27.61 6 −2
9 26.73 5 26.71 4 1
11 27.75 6 28.08 7 −1
4 27.9 7 27.09 5 2
7 28.26 8 28.86 8 0
1 29.06 9 30.69 10 −1
10 29.3 10 29.09 9 1
2 31.14 11 30.74 11 0

Table 3. Individual subject rankings based on AW.

Ground Truth Proposed Method

Subject Rank Difference
AW Rank AW Rank
2 28 1.5 27.14 1 0.5
5 28 1.5 27.35 2 −0.5
1 29 3 30.86 4 −1
3 30 4 30.67 3 1
8 35 5 36.76 5 0
4 37 6 38.71 7.5 −1.5
9 38 7 40.24 8 −1
11 39 8 41.05 9 −1
6 40 9 41.38 10 −1
7 41 10.5 42.12 11 −0.5
10 41 10.5 40.2 7.5 3
 3 30 4 30.67 3 1
8 35 5 36.76 5 0
4 37 6 38.71 7.5 −1.5
9 38 7 40.24 8 −1
11 39 8 41.05 9 −1
6 40 9 41.38 10 −1
Sensors 2017, 17, 1796 19 of 26
7 41 10.5 42.12 11 −0.5
10 41 10.5 40.2 7.5 3

Table 4. Individual subject rankings based on AH.

Table 4. Individual subject rankings based on AH.

Ground
Ground Truth
Truth Proposed Method
Proposed MethodRank Difference
Subject Subject Rank Difference
AH Rank AH Rank
AH Rank AH Rank
1 8 1.5 6.94 1 0.5
1 2 8 1.5 6.94 1 0.5
8 1.5 8.95 2 −0.5
2 8 1.5 8.95 2 −0.5
5 9 3 9.57 3 0
5 9 3 9.57 3 0
10 10 10 10 44 10.42
10.42 4 4 0 0
3 3 11 11 66 12.93
12.93 8 8 −2 −2
7 7 11 11 66 11.38
11.38 6 6 0 0
11 11 11 11 66 10.69
10.69 5 5 1 1
4 4 12 12 8.5
8.5 14.06
14.06 10 10 −1.5 −1.5
6 12 8.5 12.49 7 1.5
6 12 8.5 12.49 7 1.5
8 14 10 13 9 1
8 14 10 13 9 1
9 16 11 17.29 11 0
9 16 11 17.29 11 0

Figure 11 shows
Figure the distribution
11 shows of AIofmeasured
the distribution by thebyproposed
AI measured system
the proposed using using
system 99 data.
99 Cavanagh
data.
Cavanagh
and Rogers and Rogers
proposed criteriaproposed criteria foot
for classifying for classifying footnormal
type as high, type asand
high,
flatnormal
arches and
usingflat
AIarches
calculated
from using AI calculated
the footprint from method
[4]. Their the footprint [4]. Their
involves method
dividing theinvolves dividing
distribution of AItheinto
distribution
quartiles.ofThe
AI first
into quartiles. The first and third quartiles act as the boundaries to recognize the foot type.
and third quartiles act as the boundaries to recognize the foot type. For our system, the first and third For our
system, the first and third quartiles were 26.125 and 29.4375. Based on [4], these values suggest that
quartiles were 26.125 and 29.4375. Based on [4], these values suggest that a foot can be recognized as a
a foot can be recognized as a high arch foot if its AI < 26.125; if the AI > 29.4375, then a foot can be
high arch foot if its AI < 26.125; if the AI > 29.4375, then a foot can be recognized as a low arch foot.
recognized as a low arch foot. If its AI falls between these values, it can be recognized as a normal
If its arch
AI falls
foot.between these values, it can be recognized as a normal arch foot.

Figure11.
Figure 11.AI
AIdistribution
distribution for
forthe
the9999test data.
test data.

Figure 12 shows a linear regression analysis of the relationships among AI, AH and AW for 99
Figure
datasets12 shows abylinear
measured regression
the proposed analysis
system. of the relationships
The correlation amongAI
coefficient between AI,and
AHAHand
wasAW a for
99 datasets
negative value (r = −0.51) and was statistically significant (p < 0.0001). AI and AW were less stronglywas a
measured by the proposed system. The correlation coefficient between AI and AH
negative
correlated −0.51)
value(r(r==−0.06), andand
thewas statistically
result significant
was not significant (p =(p < 0.0001). AI and AW were less strongly
0.71).
correlated
Sensors 2017, −0.06),
(r = 17, 1796 and the result was not significant (p = 0.71). 19 of 24

(a) (b)
Figure 12. Linear regression analysis for 99 datasets: (a) scatter plot showing regression between AI
Figure 12. Linear regression analysis for 99 datasets: (a) scatter plot showing regression between AI
and AH (correlation coefficient = −0.51, < 0.0001), and (b) scatter plot showing regression
and AH (correlation coefficient r = −0.51, p < 0.0001), and (b) scatter plot showing regression between
between AI and AW (correlation coefficient = −0.06, = 0.71).
AI and AW (correlation coefficient r = −0.06, p = 0.71).
5.4. Weight for the MRF Model
In the contact region detection, the system extracts the dense and connected point set to be the
estimated region in contact with the footplate by minimizing iteratively the energy function in the
proposed MRF model. The energy function consists of four terms-color, distance, angle, and
smoothness potentials—and it has weights, = , , , for the first three terms as the control
Sensors 2017, 17, 1796 20 of 26

5.4. Weight for the MRF Model

In the contact region detection, the system extracts the dense and connected point set to be
the estimated region in contact with the footplate by minimizing iteratively the energy function
in the proposed MRF model. The energy function consists of four terms-color, distance, angle,

and smoothness potentials—and it has weights, w = wψ , wλ , wρ , for the first three terms as the
control values that enable regulation of the relative importance of each term. In this section, we
investigate the changes in foot-arch parameters according to the changes in weights, and propose the
optimal weights.
We search for an automatic criterion to find the optimal weights for the accurate estimation
of foot-arch parameters. We first define a cost function relating the errors of estimated foot-arch
parameters to changes in the weights:

1 m 1 1
F( w ) = ∑
2 i =1
( f i (w))2 = kf(w)k2 = f(w)T f(w), f i (w) = Gi − Ei (w)
2 2
(6)

where m is the number of samples, Gi and Ei (w) are the i-th ground truth and estimated result, and f
is a vector function: f : Rn → Rm with m ≥ n, n is the dimension of w. Here, we want to find the
optimal weights w∗ that minimize F(w) and equivalently minimize kf(w)k. To find the minimizer w∗ ,
we apply the Levenberg-Marquardt algorithm (LMA) [42]. We start with an initial guess, w0 . In the
iterations, w is updated by wnew = w + h only for the downhill step. The step h is calculated by solving
(J(w)T J(w) + µI)h = −J(w)T f(w), where J(w) ∈ Rm×n is the Jacobian of derivatives of f(w) with
respect to the weights and µ is the adaptive damping parameter. If 12 kf(wnew )k2 > 12 kf(w)k2 , then
reject the step, keep the old w and the old f(w), adjust damping parameter µ, and calculate the step h
again. If f(w) is converged during the iterations, return w as the minimizer of the cost function F(w).
The proposed system outputs three different estimation results for the AI, AW, and AH. Therefore,
there are three minimizers for each foot-arch parameter. To calculate them, we applied the above
LMA-based method for AI, AW, and AH separately. Among the 99 data, 22 were chosen randomly
and used for the sample (m = 22), and the remaining data were used for the evaluation. The calculated
optimal minimizers and the average error of the 77 evaluation data are shown in Table 5.

Table 5. Assessment optimal minimizer and error of foot-arch parameter estimation of the 77 data.

AI AW AH
minimizer
{2.9, 9.3, 1.8} {4.7, 6.7, 0.9} {6.2, 7.3, 1.2}
n o
w∗ = wψ ∗ , w∗ , w∗
λ ρ

average AE
{−0.17, 0.95, 0.52} {0.57, 1.03, 0.74} {0.54, 0.99, 0.62}
{AI(%), AW(mm), AH(mm)}

5.5. Processing Time

We tested the processing time using 99 dataset, and calculated the average time for each
submodule of analysis module. As shown in Table 6, the processing times of three submodules,
with the exception of the contact region detection, were less than 60 ms. The processing time of the
contact region detection was 8301.62 ms, which is the most time consuming.
Sensors 2017, 17, 1796 21 of 26

Table 6. Processing time of the analysis module.

Module Task Processing Time (ms)

Preprocessing Noise removal/coordinate mapping 53.51
Contact point detection 51.4
Contact region detection
MRF-based region segmentation 8301.62
Foot-arch parameter computation AI, AW, AH estimation 39.97
Total - 8446.5

6. Discussion
In this paper, we developed autonomous foot-arch parameter measurement system for estimating
three foot-arch parameters, AI, AW, and AH, through the use of a RGB-D camera. The system
makes use of well-known image processing techniques, such as normal vector clustering, MRF-based
segmentation, and 2/3D morphology and shape analysis, to detect the contact region and key features
of foot and to calculate the foot-arch parameters.
In Section 5.2, we described the accuracy and repeatability of the foot-arch parameters obtained
by the proposed system. And the mean error rates (= |AVG AE|/AVG GT × 100) of AI, AW, and AH
were about 0.6%, 2.7%, and 4.6%, respectively. This may result not only from computational errors in
the processing of the data but also from inherent measurement noise of the RGB-D camera. In fact,
the RGB-D used in the proposed system is known for having an around 0.5~0.6% detecting error on the
distance [21]. Therefore, it is expected to obtain more accurate and sophisticated foot-arch parameters
if the performance of the RGB-D camera has been improved.
As far as we know, lots of existing methods estimate the parameters related with the foot or arch,
but there is no system that calculates the AI, AW, and AH simultaneously. Therefore it is difficult to
directly compare the accuracy and repeatability of the proposed method with the previous studies.
The method proposed in [17] calculates the ball width, ball girth, instep height, and instep girth using
several 12 RGB cameras. The average estimation error of their method is about 2 mm. The method
proposed in [29] estimates the foot length, width and ball girth using a RGB-D camera. The standard
deviations of measurements of 10 times for each subject’s foot length and width are both 3.5 mm.
In the case of the ball girth, the variations are about 6 mm. The digital image processing based AI
computation study proposed in [30] shows that the coefficient of variation (mean ÷ standard deviation)
of AI measurements of 10 times is 1.16%. As shown in Table 1, although the comparison parameters
and evaluation dataset are different, the accuracy and repeatability of the proposed system for AI, AW,
and AH measurements are relatively better.
In Section 5.3, we firstly tested the reliability of the proposed system and showed that the foot-arch
parameters estimated through the proposed method are strongly correlated with the ground truth
measured by the conventional method. Also the ranks of foot-arch parameters using both methods
are similar. This means that the foot-arch parameters measured by the proposed method can be used
to identify the feature of individual foot as the foot-arch parameters measured by the conventional
method do. However, in case of AI, the correlation coefficient is relatively smaller than the others.
In the procedure of AI calculation, the system extracts the edge between main body part of the foot and
toe area, and then calculates the ratio of middle area of main body over total main body. The edge is
detected based on the color difference between the main body and toe area, and the system defines the
edge if the color difference between the neighboring pixels is greater than a predefined threshold (in the
proposed system, we defined the threshold as 50 explicitly.). Thus, the stable lighting condition is
important to solve this problem which is a common and significant issue in computer vision application
systems. In order to alleviate the problem caused by the different lighting condition, we installed
a LED desk lamp inside of the scanning stage. We located the lamp in front of the toe direction to
emphasize the color difference between the main body and toe area.
Sensors 2017, 17, 1796 22 of 26

Many studies reported the correlations among the foot-arch parameters have come to the
controversial conclusion, particularly that between the AI and AH [4,30,43–45]. Despite these
controversy, the correlation coefficient between the AI and AH generally demonstrated similar results
showing negative correlation (e.g., −0.70 in [30], −0.67 in [43], −0.39 in [44], and −0.42 in [45]).
As shown in Figure 12, the correlation between AI and AH obtained from our system is consistent
with these previous studies. On the other hand, according to the results of the correlation between AI
and AW, these two parameters are not related to each other. These results indicate that the analysis
result based on the data extracted from the proposed system is not different from the results of existing
researches, and show the availability of the proposed system to biomedical researches for foot analysis.
In Section 5.4, we introduced a method to determine the optimal weight of each term to find the
contact region by using the MRF method. We found that the optimal weights are different depending
on the foot-arch parameter. Nevertheless, it was confirmed that the difference in arch parameters was
calculated using different optimal weights. As shown in Table 5, the weight wλ for the second term,
distance potentials, is higher than the others in three cases, although their values are different. These
results indicate that distance potentials are significantly considered for accurate foot-arch parameter
estimation. This is also in accord with the definition of the contact region: the distance from the plane
of the contact points is close to 0. The average errors of the estimation results using three different
weight sets were less than 0.57%, 1.03 mm, and 0.74 mm, and the differences were less than 0.74%,
0.08 mm, and 0.22 mm, respectively.
In Section 5.5, we presented the processing time of each submodule in the analysis module.
The proposed system outputs the foot-arch parameters on average within 8.5 s. Compared with the
system proposed in [30] that takes 10~30 s for AI computation from a footprint image, the proposed
method is relatively fast. Among the submodules, the contact region detection consumes more
than 98% of the total processing time. This module solves the contact region detection problem by
iteratively minimizing the energy function and finding the optimal label set for each pixel. To reduce
the processing time of this module, GPU-based parallel processing can be applied. According to [46],
a GPU-based solver for pixel labeling problems is 10–12-fold faster than the CPU-based solver used in
this study. In the case of the proposed system, we did not apply a parallel-processing technique, since
the system dealt with a single depth and color image set for a static pose. However, if the number of
data processed by the system increases, the processing time of this module could be a problem. We are
considering a parallel-processing technique to improve the applicability of the system as future works.
From the automation perspective of foot-arch parameter computation, the proposed system
automatically performs the rest of the processes except for the foot axis definition for AI calculations.
In order to define the foot axis, the proposed system specifies the center point of the second toe through
the graphical user interface. For this purpose, a color image of the foot obtained is displayed to the
monitor, and the user sets the center point by clicking the mouse manually. In addition, it provides
the function to correct the designated point so that accurate foot axis detection is possible. Although
this manually selection does not require much time (less than 2 s on average in the experiment on
99 data), the usability of the proposed system will be more improved if this method becomes also
automated. However, as opposed to automatically detected the center point of heel as the salient
point on the boundary in the foot-heel, it is difficult to detect the center point of the second toe by
image processing or computer vision techniques due to the lack of visual, geometrical, and topological
features. In this case, machine learning-based detection method can be a good solution [8]. We will
apply the automatic detection of the center point of the second toe to the proposed system. We will
also apply the machine learning to estimate the position and area of important anatomic structures of
foot, such as the calcaneus, the talus, the navicular bone, and the metatarsal bones, which are known
only through X-ray, CT or MRI, from color and depth images taken by the proposed system.
One of the most difficult and important parts in the proposed system is the contact region
detection. Most existing foot-arch parameters such as AI, AW, AH, arch length index, footprint index,
arch angle, truncated arch index, and so on, are based on the shape of contact region [21]. This is since
Sensors 2017, 17, 1796 23 of 26

the individual musculoskeletal structures of the feet are reflected onto the contact region. Therefore,
the detection of the contact region is very important and can be applied to lots of foot-related researches.
The proposed system detects the contact region using the MRF as described in Section 4.3.2. Another
possible method to detect the contact region is Active shape models (ASMs) [47]. The ASMs, widely
used in facial image analysis and medical imaging, is a statistical model that iteratively deforms the
shape of the given model to fit it to the desired shape of an object. Assuming that each person's contact
region is not significantly different, the ASMs can be applied to contact region detection. In particular,
this will allow the system to detect the contact region faster than used MRF based method, since the
processing time of ASMs is very fast.
One of the representative advantages of the proposed system is that the system is able to capture
the full 3D geometric plantar shape of the foot sequentially. Unlike foot pressure-based method
which measures the pressure due to the body weight applied to the foot contact region, the proposed
system enables to obtain 3D information of the plantar shape of the foot including the contact region.
Also, unlike the existing 3D foot shape acquisition method, the proposed method can be used for
data measurement and analysis of 3D foot shape change according to continuous and dymamic
motions, such as gait, running, squat, and jumping, since continuous data acquisition is possible.
Especially, although the most of existing studies investigating the change of the arch according to the
posture change only consider the static postural changes such as sitting and standing, it is possible to
efficiently observe the continuous shape change of the foot-arch according to various motions using
the proposed system.

7. Conclusions
In this paper, we presented a RGB-D camera-based geometric foot-arch measurement system
that is able to capture the sole of the foot and estimate three foot-arch parameters: AI, AW and AH.
To achieve these goals, the proposed system provides the following: (1) 3D measurement of the plantar
surface of the foot, (2) detection of the contact region, and (3) AI, AW, and AH estimation via 2/3D
shape analysis of the contact region.
The feasibility of the system was proven by the four tests which are the average estimation error
measurements, statistical analysis, optimal weights used in the MRF, and the processing time. From
the tests it was validated that the proposed system can be used to obtain reliable geometric information
of the foot plantar surface and foot-arch parameters.
Our future work will focus on expanding the applicability of our system to dynamic foot
measurement and recognition, such as a gait and run analysis. More sophisticated methods that take
into account automatic foot part tracking and recognition will improve the feasibility and suitability of
the proposed system for dynamic foot analysis. As stated in the discussion, the parallel-processing
technique for the contact region detection module and the machine learning for automatic foot region
segmentation and landmark detection will be considered to reduce the processing time and increase the
efficiency of the system. The use of our system to identify novel characteristics of static and dynamic
foot analyses, such as the relationship between the foot-arch parameters and personal gait patterns,
is an important future research topic.

Acknowledgments: This research project was supported by the Sports Promotion Fund of Seoul Olympic Sports
Promotion Foundation from Ministry of Culture, Sports, and Tourism. And this work was supported by the
‘Civil-Military Technology Cooperation Program’ grant funded by the Korea government.
Author Contributions: Sungkuk Chun, Sejin Kong and Jinwook Kim conceived and designed the experiments;
Sungkuk Chun and Sejin Kong performed the experiments; Sungkuk Chun and Kyung-Ryoul Mun analyzed the
data; Sungkuk Chun and Sejin Kong contributed analysis tools; Sungkuk Chun, Kyung-Ryoul Mun and Jinwook
Kim wrote and revised the paper.
Conflicts of Interest: The authors declare no conflict of interest.
Sensors 2017, 17, 1796 24 of 26

References
1. Lee, Y.-H.; Hong, W.-H. Effects of shoe inserts and heel height on foot pressure, impact force, and perceived
comfort during walking. Appl. Ergon. 2005, 36, 355–362.
2. Hong, W.-H.; Lee, Y.-H.; Chen, H.-C.; Pei, Y.-C.; Wu, C.-Y. Influence of heel height and shoe insert on comfort
perception and biomechanical performance of young female adults during walking. Foot Ankle Int. 2005, 26,
1042–1048. [CrossRef] [PubMed]
3. Kouchi, M.; Kimura, M.; Mochimaru, M. Deformation of foot cross-section shapes during walking.
Gait Posture 2009, 30, 482–486. [CrossRef] [PubMed]
4. Cavanagh, P.R.; Rodgers, M.M. The arch index: A useful measure from footprints. J. Biomech. 1987, 20,
547–551. [CrossRef]
5. Razeghi, M.; Batt, M.E. Foot type classification: A critical review of current methods. Gait Posture 2002, 15,
282–291. [CrossRef]
6. Shirmohammadi, S.; Ferrero, A. Camera as the instrument: The rising trend of vision based measurement.
IEEE Instrum. Meas. Mag. 2014, 17, 41–47. [CrossRef]
7. Lee, S.H.; Yoon, C.; Chung, S.G.; Kim, H.C.; Kwak, Y.; Park, H.-W.; Kim, K. Measurement of shoulder range
of motion in patients with adhesive capsulitis using a kinect. PLoS ONE 2015, 10, e0129398. [CrossRef]
[PubMed]
8. Shotton, J.; Sharp, T.; Kipman, A.; Fitzgibbon, A.; Finocchio, M.; Blake, A.; Cook, M.; Moore, R. Real-time
human pose recognition in parts from single depth images. Commun. ACM 2013, 56, 116–124. [CrossRef]
9. Hernandez-Belmonte, U.H.; Ayala-Ramirez, V. Real-time hand posture recognition for human-robot
interaction tasks. Sensors 2016, 16, 36. [CrossRef] [PubMed]
10. Liu, Y.; Li, Y.; Ma, X.; Song, R. Facial expression recognition with fusion features extracted from salient facial
areas. Sensors 2017, 17, 712. [CrossRef] [PubMed]
11. Hu, R.; Shen, W.; Wang, H. Recursive spatiotemporal subspace learning for gait recognition. Neurocomputing
2010, 73, 1892–1899. [CrossRef]
12. Lin, C.-L.; Wang, S.-H.; Cheng, H.-Y.; Fan, K.-C.; Hsu, W.-L.; Lai, C.-R. Bimodal biometric verification using
the fusion of palmprint and infrared palm-dorsum vein images. Sensors 2015, 15, 31339–31361. [CrossRef]
[PubMed]
13. Mao, A.; Zhang, H.; Liu, Y.; Zheng, Y.; Li, G.; Han, G. Easy and fast reconstruction of a 3d avatar with an
rgb-d sensor. Sensors 2017, 17, 1113. [CrossRef] [PubMed]
14. Roy, H.; Bhattacharya, K.; Deb, S.; Ray, K. Arch index: An easier approach for arch height (a regression
analysis). Al Ameen J. Med. Sci. 2012, 5, 137–146.
15. Stavlas, P.; Grivas, T.B.; Michas, C.; Vasiliadis, E.; Polyzois, V. The evolution of foot morphology in children
between 6 and 17 years of age: A cross-sectional study based on footprints in a mediterranean population.
J. Foot Ankle Surg. 2005, 44, 424–428. [CrossRef] [PubMed]
16. JezerĹAek,
˛ M.; MoĹžina, J. High-speed measurement of foot shape based on multiple-laser-plane
triangulation. Opt. Eng. 2009, 48, 113604–113608. [CrossRef]
17. Lee, H.; Lee, K.; Choi, T. Development of a low cost foot-scanner for a custom shoe tailoring system.
In Proceedings of the 7th Symposium on Footwear Biomechanics, Cleveland, OH, USA, 27–29 July 2005.
18. Coudert, T.; Vacher, P.; Smits, C.; Van der Zande, M. A method to obtain 3d foot shape deformation during
the gait cycle. In Proceedings of the 9th International Symposium on the 3D analysis of Human Movement,
Valenciennes, France, 28–30 June 2006.
19. Amstutz, E.; Teshima, T.; Kimura, M.; Mochimaru, M.; Saito, H. Pca-based 3d shape reconstruction of human
foot using multiple viewpoint cameras. Int. J. Autom. Comput. 2008, 5, 217–225. [CrossRef]
20. Al-Baghdadi, J.; Chong, A.K.; McDougall, K.; Alshadli, D.; Milburn, P.; Newsham-West, R. A dense surface
modelling technique for foot surface imaging. In Proceedings of the Surveying & Spatial Sciences Biennial
Conference, Wellington, New Zealand, 21–25 November 2011; pp. 295–302.
21. Alshadli, D.; Chong, A.K.; McDougall, K.; Al-Baghdadi, J.; Milburn, P.; Newsham-West, R. Reliability
of a high accuracy image-based system for 3d modelling of the medial longitudinal arch during gait.
In Developments in Multidimensional Spatial Data Models; Springer: Berlin, Germany, 2013; pp. 85–101.
22. Alshadli, D. A 3D Image-Based Measurement Approach for Analysing Dynamic Foot Posture and Mobility;
University of Southern Queensland: Queensland, Australia, 2015.
Sensors 2017, 17, 1796 25 of 26

23. Kimura, M.; Mochimaru, M.; Kanade, T. Measurement of 3D foot shape deformation in motion.
In Proceedings of the 5th ACM/IEEE International Workshop on Projector Camera Systems, Marina del Rey,
CA, USA, 10 August 2008; ACM: New York, NY, USA, 2008; p. 10.
24. Yoshida, Y.; Saito, S.; Aoki, Y.; Kouchi, M.; Mochimaru, M. Shape completion and modeling of 3D foot shape
while walking. In Proceedings of the 2012 International Symposium on Optomechatronic Technologies
(ISOT), Paris, France, 29–31 October 2012; IEEE: Piscataway, NJ, USA, 2013; pp. 1–3.
25. Novak, B.; Babnik, A.; Možina, J.; Jezeršek, M. Three-dimensional foot scanning system with a rotational
laser-based measuring head. Stroj. Vestnik J. Mech. Eng. 2014, 60, 685–693. [CrossRef]
26. Van den Herrewegen, I.; Cuppens, K.; Broeckx, M.; Barisch-Fritz, B.; Vander Sloten, J.; Leardini, A.; Peeraer, L.
Dynamic 3D scanning as a markerless method to calculate multi-segment foot kinematics during stance
phase: Methodology and first application. J. Biomech. 2014, 47, 2531–2539. [CrossRef] [PubMed]
27. Liu, S.; Cui, Y.; Sanchez, S.; Stricker, D. Foot scanning and deformation estimation using time-of-flight
cameras. Footwear Sci. 2011, 3, S98–S99. [CrossRef]
28. Samson, W.; Van Hamme, A.; Sanchez, S.; Chèze, L.; Jan, S.V.S.; Feipel, V. Foot roll-over evaluation based on
3D dynamic foot scan. Gait Posture 2014, 39, 577–582. [CrossRef] [PubMed]
29. Chen, Y.-S.; Chen, Y.-C.; Kao, P.-Y.; Shih, S.-W.; Hung, Y.-P. Estimation of 3-D foot parameters using hand-held
RGB-D camera. In Proceedings of the Asian Conference on Computer Vision, Singapore, 1–5 November
2014; Springer: Berlin, Germany, 2014; pp. 407–418.
30. Chu, W.C.; Lee, S.H.; Chu, W.; Wang, T.-J.; Lee, M.-C. The use of arch index to characterize arch height:
A digital image processing approach. IEEE Trans. Biomed. Eng. 1995, 42, 1088–1093. [PubMed]
31. Matyunin, S.; Vatolin, D.; Berdnikov, Y.; Smirnov, M. Temporal filtering for depth maps generated by kinect
depth camera. In Proceedings of the 3DTV Conference: The True Vision-Capture, Transmission and Display
of 3D Video (3DTV-CON), Antalya, Turkey, 16–18 May 2011; IEEE: Piscataway, NJ, USA, 2011; pp. 1–4.
32. Zhang, C.; Zhang, Z. Calibration between depth and color sensors for commodity depth cameras. In Computer
Vision and Machine Learning with RGB-D Sensors; Springer International Publishing: Basel, Switzerland, 2014;
pp. 47–64.
33. He, L.; Chao, Y.; Suzuki, K. A run-based two-scan labeling algorithm. IEEE Trans. Image Process. 2008, 17,
749–756. [PubMed]
34. Holz, D.; Schnabel, R.; Droeschel, D.; Stückler, J.; Behnke, S. Towards semantic scene analysis with
time-of-flight cameras. In Robocup 2010: Robot Soccer World Cup Xi; Springer: Berlin, Germany, 2011;
pp. 121–132.
35. Holz, D.; Holzer, S.; Rusu, R.B.; Behnke, S. Real-Time Plane Segmentation Using RGB-D Cameras; Robot Soccer
World Cup, 2011; Springer: Berlin, Germany; pp. 306–317.
36. Forbes, A. Least-Squares Best-Fit Geometric Elements; National Physical Laboratory: Teddington, UK, 1989.
37. Kato, Z.; Pong, T.-C. A markov random field image segmentation model for color textured images.
Image Vis. Comput. 2006, 24, 1103–1114. [CrossRef]
38. Chun, S.; Hong, K.; Jung, K. 3D star skeleton for fast human posture representation. World Acad. Sci.
Eng. Technol. 2008, 2, 2603–2612.
39. Graham, R.L.; Yao, F.F. Finding the convex hull of a simple polygon. J. Algorithms 1983, 4, 324–331. [CrossRef]
40. Bagon, S. Matlab Wrapper for Graph Cuts. Available online: http://www.wisdom.weizmann.ac.il/~bagon/
matlab.html (accessed on 6 June 2017).
41. Pohl, M.B.; Farr, L. A comparison of foot arch measurement reliability using both digital photography and
calliper methods. J. Foot Ankle Res. 2010, 3, 14. [CrossRef] [PubMed]
42. Marquardt, D.W. An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math.
1963, 11, 431–441. [CrossRef]
43. McCrory, J.; Young, M.; Boulton, A.; Cavanagh, P.R. Arch index as a predictor of arch height. Foot 1997, 7,
79–81. [CrossRef]
44. Hawes, M.R.; Nachbauer, W.; Sovak, D.; Nigg, B.M. Footprint parameters as a measure of arch height.
Foot Ankle 1992, 13, 22–26. [CrossRef] [PubMed]
45. Queen, R.M.; Mall, N.A.; Hardaker, W.M.; Nunley, J.A. Describing the medial longitudinal arch using
footprint indices and a clinical grading system. Foot Ankle Int. 2007, 28, 456–462. [CrossRef] [PubMed]
Sensors 2017, 17, 1796 26 of 26

46. Vineet, V.; Narayanan, P. CUDA cuts: Fast graph cuts on the GPU. In Proceedings of the IEEE Computer
Society Conference on Computer Vision and Pattern Recognition Workshops, 2008 (CVPRW’08), Anchorage,
AK, USA, 23–28 June 2008; IEEE: Piscataway, NJ, USA, 2008; pp. 1–8.
47. Cootes, T.F.; Taylor, C.J.; Cooper, D.H.; Graham, J. Active shape models-their training and application.
Comput. Vis. Image Underst. 1995, 61, 38–59. [CrossRef]

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).