A Mechatronics Data Collection, Image Processing, and Deep Learning Platform For Clinical Posture Analysis
A Mechatronics Data Collection, Image Processing, and Deep Learning Platform For Clinical Posture Analysis
A Mechatronics Data Collection, Image Processing, and Deep Learning Platform For Clinical Posture Analysis
https://doi.org/10.1007/s13246-021-01035-w
SCIENTIFIC NOTE
Received: 24 October 2020 / Accepted: 13 July 2021 / Published online: 16 August 2021
© Australasian College of Physical Scientists and Engineers in Medicine 2021
Abstract
Static and dynamic posture analysis was a critical clinical examination in physiotherapy and rehabilitation. It was a time-
consuming task for clinicians, so a semi-automatic method can facilitate this process as well as provide well-documented
medical records and strong infrastructure for deep learning scenarios. The current research presents a mechatronics plat-
form for static and real-time dynamic posture analysis, which consisted of hybrid computational modules. Our study was
a developmental and applied research according to a system development life cycle. The designed modules are as follows:
(1) a mechanical structure includes patient place, 360-degree engine, mirror, laser, distance meter, and cams; (2) a software
module includes data collection, electronic medical record, semi-automatic image analysis, annotation, and reporting, and
(3) a network to exchange raw data with deep learning server. Patients were informed about the research by their healthcare
provider and all data were transformed into a Fourier format, in which the patients remained autonomous without a bit of
information. The results show acceptable reliability and validity of the instruments. Also, a telerehabilitation application was
designed to cover the patients after diagnosis. We suggest a longer time for data acquisition. It will lead to a more accurate
and fully automated dynamic posture analysis. The result of this study suggest that the designed mechatronics device used
in conjunction with smartphone application is a valid tool that can be used to obtain reliable measurements.
1
* Taha Samad‑Soltani Physiotherapy Department, Faculty of Rehabilitation, Tabriz
samadsoltani@tbzmed.ac.ir University of Medical Science, 29 Bahman St, Tabriz, Iran
2
Zahra Salahzadeh Department of Health Information Technology, School
salahzadeh@tbzmed.ac.ir of Management and Medical Informatics, Tabriz University
of Medical Sciences, Tabriz, Iran
Peyman Rezaei‑Hachesu
3
rezaeip@tbzmed.ac.ir Department of Artificial Intelligence, Faculty of Computer
Engineering, University of Tabriz, Tabriz, Iran
Yousef Gheibi
4
yousef.gheibi@gmail.com Department of Research and Development, SanamSahand
Health Promotion Industries, Tabriz, Iran
Ali Aghamali
5
seyedali1377525@gmail.com Department of Biomedical Engineering, Islamic Azad
University of Tabriz, Tabriz, Iran
Hamed Pakzad
Hamed.pakzad@yahoo.com
Saeideh Foladlou
saeideh_foladlou@yahoo.com
13
Vol.:(0123456789)
902 Physical and Engineering Sciences in Medicine (2021) 44:901–910
to the correction of other parts of the body in patients with processing and identifying the patient’s problems, extracting
spinal disorders has a significant effect on the successful knowledge from his or her images, and using that knowledge
treatment of this type of disorder [1, 2]. to make better decisions on the management of related ill-
Postural problems need to be evaluated in a timely and nesses as decision support systems, ultimately improving the
accurate manner because they may lead to greater problems quality of healthcare.
in the musculoskeletal system as they develop [3]. Two main
approaches for clinical posture analysis are observational [4,
5] and photogrammetry [6, 7] methods. Compared to other Methods
methods of postural analysis that may be invasive or harm-
ful to patients (e.g., X-ray) or costly (e.g., 3D models), the The majority of existing posture analysis systems have a
alternative is to use a better, modernized, and non-invasive static nature in terms of image acquisition, analysis, and
solution, i.e., computerized or digital photogrammetry that reporting. Line drawing and imaging analysis have been
uses a combination of digital cameras and image process- shown to be highly reliable. However, many arguments
ing techniques [8, 9]. Images constitute a significant set of affect the accuracy of computations, such as position chang-
important data records in medicine since they are of great ing due to rotation, inaccurate distancing between the patient
technical and computational importance in computer science and the camera, and the ineffective top view of the patient.
as the most accurate type of signal. By using, analyzing, and Our designed system overcomes these limitations using a
processing images, the quality of the data generated from a mechatronics approach embedded in traditional posture
diagnostic point of view will lead to a reduction in the cogni- analysis methods. To achieve our objectives, we have organ-
tive errors of physicians [10, 11]. A number of studies have ized the remainder of the paper in the following way. In the
been conducted with various objectives to detect, measure, next section, we provide a brief description of the hardware
and analyze posture using computerized methods. However, and software components. Then, the main outcomes of the
the majority of these studies have been limited to a small set system in early applications in rehabilitation centers are
of functions and measures. For example, Patel et al. used presented.
wireless and wearable technologies as a potential alternative
to clinical testing to assess the risk of falling in older adults. Components and capabilities of the posture analysis
They used a wearable inertial sensor which provided real- system
time kinematic data during routine activities by classifying
posture [12]. Afanasieva et al. state that the sensitivity and The designed system consisted of (1) a sturdy and strong
specificity of the computerized photo-geometric program structure in two parts, i.e., patient structure and special-
for posture analysis were in a satisfactory range, empha- ist panel, (2) two cameras installed for vertical and frontal
sizing the necessity of utilizing this program in healthcare imaging, and (3) a remote-controlled 360-degree patient
practice [13]. The available literature shows that informa- rotation engine. Each component has its sub-modules, which
tion technology provides the capability to objectively iden- are discussed in the next section.
tify postural risks among patients, even when clinical tests As shown in Fig. 1, three separate structures were
cannot. Other studies have shown that the results obtained designed to fix patient and clinician positions. The clini-
using traditional postural evaluation and postural biome- cian panel includes a computer to compute postural meas-
chanical evaluation methods are less satisfactory compared urements, a monitor to view the real-time software UI, a
to computerized biometrics. Therefore, a fast, semi-auto- button panel to control the hardware and the step motor,
matic, and digitized method for posture analysis can help and an optional printer to print reports. The patient structure
clinicians prevent human errors, while saving time and cost includes a station for the patient to settle in, a top camera,
[14, 15]. Moreover, these studies recommend using various a 360-degree step motor to rotate the patient, and a sloping
computational and technological tools to improve the results mirror on top of the patient. The third structure is a fixed
based on the differences among subjects in terms of gender base for the front camera, which is installed at a distance
or race [16]. Thus, we need a more intelligent approach for between the patient and the specialist. We used effective and
knowledge discovery and pattern recognition. The current affordable webcams instead of an expensive camera. Table 1
applied-developmental study aims to create a mechatronics presents a list of peripheral devices and their applications.
platform to provide easy access and to process medical pos- We developed a stepper motor for the 360-degree plat-
tural images in order to determine complications, problems, form by assembling the parts available in the market and
and the form of the spine. Consistent with this goal, a data- the exclusive design of a partner company. Figure 2 shows
base management system for the follow-up and for record- the designed platform.
ing patient information will also be developed. The main The computational and operational software of the system
objectives of the study include documenting patient data, consisted of (1) a preliminary intelligent processing module
13
Physical and Engineering Sciences in Medicine (2021) 44:901–910 903
Table 1 Peripheral devices and objects and their features for posture analysis
No (Fig. 1) Device Features Application
13
904 Physical and Engineering Sciences in Medicine (2021) 44:901–910
Fig. 2 The stepper motor of the 360-degree platform (without the top
cover)
13
Physical and Engineering Sciences in Medicine (2021) 44:901–910 905
Fig. 4 Various views of the software UI (a patient record search, b anterior view of patient imaging, c anterior measurement toolbox, and d a
sample of software reports)
13
906 Physical and Engineering Sciences in Medicine (2021) 44:901–910
Fig. 5 The architecture of the proposed deep learning platform (required to train Model 2)
from a convenient sample of five males and five females, Then, a data extraction protocol was designed based on
assessed previously using traditional methods (average age: common therapies in the country. The raters became quite
28.5, average weight: 84.3 KG, average height: 173.5 cm). familiar with the calibration and usage of the device. They
The data were recorded based on the captured photos and acquired sufficient knowledge to identify relevant postural
related reports stored in paper-based medical records. Expert landmarks suggested by the software using an early-trained
physiotherapists, as the raters, performed various posture deep learning approach used as a semi-automated image pro-
measurements as the person was standing on the structure. cessing method. It should be noted that because of the lim-
For each sample, a 360-degree scan was performed to take ited number of images in this trained model, the freedom of
four images of the patient. Prior to data extraction, the raters the user for changing the suggested landmarks was not lim-
completed a session to learn how to operate the software. ited. To achieve a fully-automated software, a cloud-based
13
Physical and Engineering Sciences in Medicine (2021) 44:901–910 907
solution is proposed in the current study to collect data from suitable for ergonomic and personal style improvement, it
the client software and train a powerful model. All the data is limited to a non-clinical analysis of the subject. Our ulti-
were imported into the software’s database for further intra mate objective was to investigate the reliability of the pos-
and inter-rater reliability analysis. To calculate the reliabil- ture parameters extracted from the input data (the image
ity parameters, the change in the mean, the typical error of stream) using the proposed device. Our device analyzed
measurement (TEM), and the intra-class correlation coef- the posture in four different views (i.e., anterior, lateral,
ficient (ICC) were computed using the method proposed by posterior, and top) without the use of reference markers.
Smit and Hopkins [19]. All the measures examined here demonstrated good to
excellent intra-rater reliability with small to moderate typi-
• Intra-rater reliability: A summarized report of the pos- cal errors of measurement based on commonly-employed
tural parameters extracted during 2 days by the same rater thresholds. In a study conducted by Hebert-Losier and
is provided in Table 2 with the relevant ICC. Overall, Abd-Rahman, the reliability of a popular software, i.e.,
the intra-rater reliability was good and excellent for 80% Posture Pro 8, was evaluated. To do so, 40 badminton
(n = 35) of the computed parameters. players participated in the study, and 33 parameters were
• Inter-rater reliability: A summarized report of the pos- extracted and calculated for each person. The results
tural parameters extracted by the two pairs of raters is showed that the intra-rater reliability was considered good
provided in Table 3 with the relevant ICC. It should be to excellent for nearly all the parameters. However, the
noted that 72% (n = 32) of the parameters exhibit good inter-reliability provided poor results, i.e., only two of the
and excellent inter-rater reliability. measures were considered excellent [24]. Moreover, Hop-
kins et al. evaluated the reliability of a mobile application
designed to measure the standing posture. In their study,
Discussion PostureScreen mobile app for iOS was compared with a
3D scan software. The app had acceptable ICC, and it was
In this work, we proposed a method to analyze posture meas- able to promptly assess posture [25]. Ferreira et al. evalu-
ures based on a mechatronics platform. Using a physical ated and validated a postural assessment software (PAS/
structure and a software toolkit, we were able to accurately SAPO). The study’s sample consisted of 88 images from
measure and judge the patient’s posture. Physiotherapists 22 patients, assessed by five raters. Then, ICC was calcu-
commonly apply qualitative and quantitative visual assess- lated. In this study, the inter-rater reliability was excellent
ment methods for the evaluation of body alignment and pos- and good in 76% of the cases, while intra-reliability was
tural imbalances [20]. excellent and good in 68% of the cases. They concluded
Sardini et al. described a wearable platform for monitor- that PAS/SAPO was accurate as a reliable tool for measur-
ing posture during rehabilitation exercises for the spine in ing posture [26]. Furthermore, Furlanetto et al. validated a
patients with scoliosis. Through a sensor-equipped T-shirt, posture analysis method using a digital image-based pos-
the platform measured the posture of the person using an tural assessment software. They stated that image-based
inductive sensor [21]. The main drawbacks of wearable sen- methods were a valid, simple, low-cost, non-invasive,
sors are their additional weight on the patient, the rigidity of and practical approach that could provide acceptable reli-
the structures that support them, and the shape and size and ability [27]. In the current study, the device demonstrated
other properties that make them invasive for the patient [22]. generally acceptable reliability (more excellent and good
Therefore, applying a non-invasive and invisible platform results) with a good user interface, quick calculations, and
will protect the user’s freedom and improve user satisfaction. a non-invasive approach. Nonetheless, the evaluation of
Our proposed platform includes a non-contact collection of the validity and sensitivity, and the interpretation of these
devices. measures will be vital to confirm the clinical relevance of
Ding et al. presented a new photogrammetry method for the proposed system. A number of papers have focused on
the online assessment of upper-body postures of a person wearable devices for spinal posture analysis [28]. How-
sitting in front of a webcam. Unlike most existing vision- ever, there are some limitations in the measurement of
based methods, the proposed software directly analyzes posture due to the time-consuming process of attaching
single images captured by the webcam. They proposed sensors and the technical problems of sensors in real-
a classification method without explicitly measuring the ity, such as battery limitations and noise sensitivity [29].
variables required for calculating risk scores. The method Therefore, our image-based methodology makes it more
achieved an accuracy of 99.5% for binary classification feasible for the patient and clinicians to be free of limita-
(low vs. high-risk postures) and 88.2% for the classifica- tions by employing an integrated easy-to-use device with
tion of 19 risk levels [23]. While this software is highly the least amount of technical dependency.
13
908 Physical and Engineering Sciences in Medicine (2021) 44:901–910
Anterior Head lateral tilt angle 0–5 0.96 Excellent 0.82 Good
Head inclination angle 0–5 0.93 Excellent 0.66 Fair
Shoulder tilt angle 0–5 0.94 Excellent 0.88 Good
Umbilicus plumb line distance 0–5 mm 0.89 Good 0.55 Fair
Lateral pelvic tilt angle 0–2 0.95 Excellent 0.80 Good
Shoulder on pelvic tilt angle 0–2 0.95 Excellent 0.92 Excellent
Greater trochanter tilt angle 0–2 0.97 Excellent 0.94 Excellent
Tibial tubers sity tilt angle 0–5 0.89 Good 0.69 Fair
Lateral knee angle Not standard 0.92 Excellent 0.90 Excellent
Femor segment length Not standard 0.95 Excellent 0.90 Excellent
Tibia segment length Not standard 0.90 Excellent 0.90 Excellent
Bimalleular distance 0–5 0.88 Good 0.81 Good
Lateral malleolus tilt angle 0–5 0.91 Excellent 0.72 Fair
Asis to medial malleolus distance Not standard 0.86 Good 0.89 Good
Umbilicus to medial malleolus distance Should be similar for right and left sides 0.69 Fair 0.55 Fair
Posterior Scapula tilt angle Not standard range 0.89 Good 0.87 Good
Scapula upward rotation angle Not standard range 0.91 Excellent 0.94 Excellent
Inferior angle midline distance 5 cm 0.77 Good 0.68 Fair
Scapula T2 distance 4 cm 0.93 Excellent 0.92 Excellent
Pelvic tilt angle 0–5 degree 0.96 Excellent 0.91 Excellent
Ankle and feet Not standard range 0.88 Good 0.86 Good
Lateral Crani over tebral angle 38–54 0.97 Excellent 0.80 Good
Head tilt angle 0–5 0.93 Excellent 0.91 Excellent
Head position angle Not standard range: les angle, more FHP 0.68 Fair 0.64 Fair
Upper cervical angle Not standard range: greater angle, more extension 0.91 Excellent 0.93 Excellent
Lower cervical angle Not standard range: greater angle, more extension 0.93 Excellent 0.82 Good
Tragus plumb line distance 0–5 mm 0.91 Excellent 0.88 Excellent
Shoulder protraction angle Not standard range 0.94 Excellent 0.90 Excellent
Thoracic angle Not standard range 0.90 Excellent 0.89 Excellent
Acromion plumb line 0–5 mm 0.88 Good 0.65 Fair
Vertical body angle Not standard range 0.94 Excellent 0.68 Fair
Trunk sagital angle Not standard range 0.94 Excellent 0.90 Excellent
Body sagital angle Not standard range 0.99 Excellent 0.89 Excellent
Sway angle Not standard range 0.93 Excellent 0.90 Excellent
Trunk to lower line angle 0–5 degree 0.89 Good 0.66 Fair
Sagital pelvic tilt angle 0–5 degree 0.88 Good 0.82 Good
Lower limb angle Not standard range 0.94 Excellent 0.91 Excellent
Hip joint angle Not standard range 0.92 Excellent 0.95 Excellent
Greater trochanter plumb line 0–5 mm 0.89 Good 0.80 Good
Lateral knee angle Not standard range 0.91 Excellent 0.66 Fair
Knee to plumb line distance Not standard range 0.98 Excellent 0.91 Excellent
Lateral ankle tilt angle Not standard range 0.97 Excellent 0.89 Excellent
Top Head rotation angle Not standard range 0.89 Good 0.81 Good
Toe out angle Not standard range 0.93 Excellent 0.67 Fair
13
Physical and Engineering Sciences in Medicine (2021) 44:901–910 909
13
910 Physical and Engineering Sciences in Medicine (2021) 44:901–910
25. Hopkins BB, Vehrs PR, Fellingham GW, George JD, Hager R, spinal posture. In: 2006 10th IEEE international symposium on
Ridge ST (2019) Validity and reliability of standing posture wearable computers. IEEE, pp 65–68
measurements using a mobile application. J Manip Physiol Ther 29. Simpson L, Maharaj MM, Mobbs RJ (2019) The role of wear-
42(2):132–140 ables in spinal posture analysis: a systematic review. BMC
26. Ferreira EAG, Duarte M, Maldonado EP, Burke TN, Marques Musculoskelet Disord 20(1):55. https:// d oi. o rg/ 1 0. 1 186/
APJC (2010) Postural assessment software (PAS/SAPO): valida- s12891-019-2430-6
tion and reliabiliy. Clinics 65(7):675–681
27. Furlanetto TS, Candotti CT, Comerlato T, Loss JF (2012) Validat- Publisher's Note Springer Nature remains neutral with regard to
ing a postural evaluation method developed using a Digital Image- jurisdictional claims in published maps and institutional affiliations.
based Postural Assessment (DIPA) software. Comput Methods
Progr Biomed 108(1):203–212. https://doi.org/10.1016/j.cmpb.
2012.03.012
28. Dunne LE, Walsh P, Smyth B, Caulfield B (2006) Design and
evaluation of a wearable optical sensor for monitoring seated
13