Content uploaded by Zengxi Pan
Author content
All content in this area was uploaded by Zengxi Pan on May 24, 2016
Content may be subject to copyright.
A preview of the PDF is not available
Human motion capture sensors and analysis in robotics
Abstract
Purpose
The purpose of this paper is to provide a review of various motion capture technologies and discuss the methods for handling the captured data in applications related to robotics.
Design/methodology/approach
The approach taken in the paper is to compare the features and limitations of motion trackers in common use. After introducing the technology, a summary is given of robotic‐related work undertaken with the sensors and the strengths of different approaches in handling the data are discussed. Each comparison is presented in a table. Results from the author's experimentation with an inertial motion capture system are discussed based on clustering and segmentation techniques.
Findings
The trend in methodology is towards stochastic machine learning techniques such as hidden Markov model or Gaussian mixture model, their extensions in hierarchical forms and non‐linear dimension reduction. The resulting empirical models tend to handle uncertainty well and are suitable for incrementally updating models. The challenges in human‐robot interaction today include expanding upon generalising motions to understand motion planning and decisions and build ultimately context aware systems.
Originality/value
Reviews including descriptions of motion trackers and recent methodologies used in analyzing the data they capture are not very common. Some exist, as has been pointed out in the paper, but this review concentrates more on applications in the robotics field. There is value in regularly surveying the research areas considered in this paper due to the rapid progress in sensors and especially data modeling.
... Skeleton motion data generated using such a system comprise human poses represented via joint angles or positions for each frame [5]. Three-dimensional (3D) skeleton motion data are widely used in several applications, such as human-computer interactions [6], virtual reality [7], robotics [8][9][10], movie production [11], and action recognition [12][13][14][15][16]. ...
... For the natural motion sequences, the bone length loss is needed to keep the bone length constant. The bone length loss is defined as (8) where L b denotes the bone length of clean data, and l b denotes the predicted value of the joint coordinates between the two ends of the bone, calculated with the L2 norm. ...
With recent advances in computer science, there is an increasing need to convert human motion to digital data for human body research. Skeleton motion data comprise human poses represented via joint angles or joint positions for each frame of captured motion. Three-dimensional (3D) skeleton motion data are widely used in various applications, such as virtual reality, robotics, and action recognition. However, they are often noisy and incomplete because of calibration errors, sensor noise, poor sensor resolution, and occlusion due to clothing. Data-driven models have been proposed to denoise and fill incomplete 3D skeleton motion data. However, they ignore the kinematic dependencies between joints and bones, which can act as noise in determining a marker position. Inspired by a directed graph neural network, we propose a novel model to fill and denoise the markers. This model can directly extract spatial information by creating bone data from joint data and temporal information from the long short-term memory layer. In addition, the proposed model can learn the connectivity between joints via an adaptive graph. On evaluation, the proposed model showed good refinement performance for unseen data with a different type of noise level and missing data in the learning process.
... For instance, human-like robots are used for replicating human motion based on one-to-one re-targeting [7]. Motion tracking systems that are generally used for human arm motion tracking can be categorized into optical, inertial, mechanical, magnetic, and acoustic techniques [10]. ...
... Considerable amounts of cameras have been used for resolving occlusion problems. This improves motion accuracy compared to the marker-less system [10]. ...
In human–robot collaboration (HRC), human motion capture can be considered an enabler for switching autonomy between humans and robots to create efficient and safe operations. For this purpose, wearable motion tracking systems such as IMU and lighthouse-based systems have been used to transfer human joint motions into robot controller models. Due to reasons such as global positioning, drift, and occlusion, in some situations, e.g., HRC, both systems have been combined. However, it is still not clear if the motion quality (e.g., smoothness, naturalness, and spatial accuracy) is sufficient when the human operator is in the loop. This article presents a novel approach for measuring human motion quality and accuracy in HRC. The human motion capture has been implemented in a laboratory environment with a repetition of forty-cycle operations. Human motion, specifically of the wrist, is guided by the robot tool center point (TCP), which is predefined for generating circular and square motions. Compared to the robot TCP motion considered baseline, the hand wrist motion deviates up to 3 cm. The approach is valuable for understanding the quality of human motion behaviors and can be scaled up for various applications involving human and robot shared workplaces.
... Capturing human motion and being able to continuously track it, is an important aspect to consider for implementing PbD applications in robotics. In general, the prevalence of tracking technology has increased in recent days due to advances in computing power; with improved tracking algorithms and enhanced post processors which minimize measurement errors [39]. The spatio-temporal information of the measured entity or body part (its continuous deviation from an initial point of reference) is determined by tracking its real-time position and orientation with high sampling rates. ...
... Infrared cameras are placed around the subject to triangulate a marker position on the different body parts using 10 to 12 fixed high-speed and high-resolution cameras. An infrared camera's frequency for capturing high-contrast images varies from 25 Hz to 2 kHz, depending on the number and type of markers [19]. To avoid errors caused by occlusion, most technologies utilize a minimum of three cameras to track the markers [20]. ...
In recent years, numerous studies have been conducted to analyze how humans subconsciously optimize various performance criteria while performing a particular task, which has led to the development of robots that are capable of performing tasks with a similar level of efficiency as humans. The complexity of the human body has led researchers to create a framework for robot motion planning to recreate those motions in robotic systems using various redundancy resolution methods. This study conducts a thorough analysis of the relevant literature to provide a detailed exploration of the different redundancy resolution methodologies used in motion generation for mimicking human motion. The studies are investigated and categorized according to the study methodology and various redundancy resolution methods. An examination of the literature revealed a strong trend toward formulating intrinsic strategies that govern human movement through machine learning and artificial intelligence. Subsequently, the paper critically evaluates the existing approaches and highlights their limitations. It also identifies the potential research areas that hold promise for future investigations.
... Optical motion tracking technologies can be classified based upon their working principle, dividing them into marker-based and markerless [1]. Marker-based motion capture relies on either active infrared (IR) or passive retroreflective markers whose motion is tracked by two or more cameras. ...
Accurate capture finger of movements for biomechanical assessments has typically been achieved within laboratory environments through the use of physical markers attached to a participant’s hands. However, such requirements can narrow the broader adoption of movement tracking for kinematic assessment outside these laboratory settings, such as in the home. Thus, there is the need for markerless hand motion capture techniques that are easy to use and accurate enough to evaluate the complex movements of the human hand. Several recent studies have validated lower-limb kinematics obtained with a marker-free technique, OpenPose. This investigation examines the accuracy of OpenPose, when applied to images from single RGB cameras, against a ‘gold standard’ marker-based optical motion capture system that is commonly used for hand kinematics estimation. Participants completed four single-handed activities with right and left hands, including hand abduction and adduction, radial walking, metacarpophalangeal (MCP) joint flexion, and thumb opposition. The accuracy of finger kinematics was assessed using the root mean square error. Mean total active flexion was compared using the Bland–Altman approach, and the coefficient of determination of linear regression. Results showed good agreement for abduction and adduction and thumb opposition activities. Lower agreement between the two methods was observed for radial walking (mean difference between the methods of 5.03°) and MCP flexion (mean difference of 6.82°) activities, due to occlusion. This investigation demonstrated that OpenPose, applied to videos captured with monocular cameras, can be used for markerless motion capture for finger tracking with an error below 11° and on the order of that which is accepted clinically.
... A robot can have different degrees of freedom and/or different linkage lengths from the human trying to control it. Additionally, different human motion capturing approaches (such as using multiple cameras [3], attaching markers or sensors to the body [4,5,6], using wearable tracker suits [7,8], or using Kinect skeletal trackers [9,10,11,12]) used different coordinate systems and resolutions. These make it difficult to develop a one-to-one mapping between different human input systems and robot structures. ...
Although data-driven motion mapping methods are promising to allow intuitive robot control and teleoperation that generate human-like robot movement, they normally require tedious pair-wise training for each specific human and robot pair. This paper proposes a transferability-based mapping scheme to allow new robot and human input systems to leverage the mapping of existing trained pairs to form a mapping transfer chain, which will reduce the number of new pair-specific mappings that need to be generated. The first part of the mapping schematic is the development of a Synergy Mapping via Dual-Autoencoder (SyDa) method. This method uses the latent features from two autoencoders to extract the common synergy of the two agents. Secondly, a transferability metric is created that approximates how well the mapping between a pair of agents will perform compared to another pair before creating the motion mapping models. Thus, it can guide the formation of an optimal mapping chain for the new human-robot pair. Experiments with human subjects and a Pepper robot demonstrated 1) The SyDa method improves the accuracy and generalizability of the pair mappings, 2) the SyDa method allows for bidirectional mapping that does not prioritize the direction of mapping motion, and 3) the transferability metric measures how compatible two agents are for accurate teleoperation. The combination of the SyDa method and transferability metric creates generalizable and accurate mapping need to create the transfer mapping chain.
Markerless motion capture systems offer the advantage of non-intrusive and on-site motion analysis, but they often suffer from limited accuracy compared to marker-based systems. In this paper, we propose a novel fusion-based approach to enhance the accuracy of on-site motion analysis using OpenPose, a popular markerless motion tracking framework. The proposed method combines data obtained from OpenPose with a marker-based Regions of Interest (ROIs) detection method, aiming to improve the accuracy and reliability of markerless motion capture for on-site applications. Multiple cameras are utilized for 3D reconstruction to refine the joint positions initially provided by OpenPose. The obtained results exhibit a significant improvement in limb length and angle measurements, achieving root mean square errors (RMSE) of less than 32.7mm and 7.61
, respectively, after correction. These findings outperformed the accuracy achieved by OpenPose prior to employing our fusion-based approach.
In recent years, robotic arms have been widely applied in various fields such as industrial automation, logistics and warehousing, healthcare, and services. The transfer of human experience to robots and human motion tracking technology have drawn excessive attentions and become a hot research topic. As a potential solution for automating metalworking processes, robotic arms can learn from human motion experience, thereby improving work efficiency and safety. In healthcare, combining robotic arms with human motion tracking technology can help healthcare workers to provide more effective and safe care for patients. To better and effectively capture the dynamic position and posture of the human arm, a motion capture system with inertial measurement units (IMUs) is employed in this research to measure human arm movements, followed by establishing a human arm kinematic chain model and a seven-axis human arm model to reproduce the captured human arm dynamics. In addition, the placement of the IMUs on the human arm is investigated to analyze the effect on the seven-axis joint angles of the seven-axis human arm model. Installation calibration process is also performed to improve accuracy of capturing the human arm dynamic position and posture for experimental validations. In this study, the motion capture system is adopted in combination with the two established models to improve the accuracy of capturing the dynamic position and posture of the human arm motion.
This article presents a standardized human–robot teleoperation interface (HRTI) evaluation scheme for mobile manipulators. Teleoperation remains the predominant control type for mobile manipulators in open environments, particularly for quadruped manipulators. However, mobile manipulators, especially quadruped manipulators, are relatively novel systems to be implemented in the industry compared to traditional machinery. Consequently, no standardized interface evaluation method has been established for them. The proposed scheme is the first of its kind in evaluating mobile manipulator teleoperation. It comprises a set of robot motion tests, objective measures, subjective measures, and a prediction model to provide a comprehensive evaluation. The motion tests encompass locomotion, manipulation, and a combined test. The duration for each trial is collected as the response variable in the objective measure. Statistical tools, including mean value, standard deviation, and T-test, are utilized to cross-compare between different predictor variables. Based on an extended Fitts' law, the prediction model employs the time and mission difficulty index to forecast system performance in future missions. The subjective measures utilize the NASA-task load index and the system usability scale to assess workload and usability. Finally, the proposed scheme is implemented on a real-world quadruped manipulator with two widely-used HRTIs, the gamepad and the wearable motion capture system.
Purpose ? The purpose of this paper is to achieve reliable estimation of yaw angles by fusing data from low-cost inertial and magnetic sensing. Design/methodology/approach ? In this paper, yaw angle is estimated by fusing inertial and magnetic sensing from a digital compass and a gyroscope, respectively. A Kalman filter estimates the error produced by the gyroscope. Findings ? Drift effect produced by the gyroscope is significantly reduced and, at the same time, the system has the ability to react quickly to orientation changes. The system combines the best of each sensor, the stability of the magnetic sensor and the fast response of the inertial sensor. Research limitations/implications ? The system does not present a stable behavior in the presence of large vibrations. Considerable calibration efforts are needed. Practical implications ? Today, most of human?robot interaction technologies need to have the ability to estimate orientation, especially yaw angle, from small-sized and low-cost sensors. Or
Human-like movement can be efficiently generated by driving a damped inertial plant toward a fixed target with position-based control, then switching to conventional feedback control as the desired endpoint is approached. Key characteristics of human motion, not easily replicated with traditional control architectures, result from tracking a position-based actuation template derived from human trials. Computation of the applied forcing function requires only linear scaling of this template, or displacement-normalized actuation program (DNAP). Simulated ballistic movements generated with the proposed method are shown to be consistent with human subject kinematic trajectories.
This paper investigates two methods of displacement estimation using sampled acceleration and orientation data from a 6 degrees of freedom (DOF) Inertial Measurement Unit (IMU), with the application to sporting training and rehabilitation. Currently, the use of low cost IMUs for this particular application is very impractical due to the accumulation of errors from various sources. Previous studies and projects that have applied IMUs to similar applications have used a lower number of DOF, or have used higher accuracy navigational grade IMUs. Solutions to the acceleration noise accumulation and gyroscope angle error problem are proposed in this paper. A zero velocity update algorithm (ZUPT) is also developed to improve the accuracy of displacement estimation with a low grade IMU. The experimental results from this study demonstrate the feasibility of using an IMU with loose tolerances to determine the displacement. Peak distances of a range of exercises are shown to be measured with accuracies within 5% for the numerical integration methods.
This paper introduces a new approach to 3-D position estimation from
acceleration data, i.e., a 3-D motion tracking system having a small size and
low-cost magnetic and inertial measurement unit (MIMU) composed by both a
digital compass and a gyroscope as interaction technology. A major challenge is
to minimize the error caused by the process of double integration of
accelerations due to motion (these ones have to be separated from the
accelerations due to gravity). Owing to drift error, position estimation cannot
be performed with adequate accuracy for periods longer than few seconds. For
this reason, we propose a method to detect motion stops and only integrate
accelerations in moments of effective hand motion during the demonstration
process. The proposed system is validated and evaluated with experiments
reporting a common daily life pick-and-place task.
In this paper, we present a low-cost IMU-based system, Pedalvatar, which can capture the full-body motion of users in real-time. Unlike the prior approaches using the hip-joint as the root of forward kinematic model, a foot-rooted kinematic model is developed in this work. A state change mechanism has also been investigated to allow dynamically switching the root of kinematic trees between the left and the right foot. Benefitted from this, full-body motions can be well captured in our system as long as there is at least one static foot in the movement. The 'floating' artifact of hip-joint rooted methods has been eliminated in our approach, and more complicated motions such as climbing stairs can be successfully captured in real-time. Comparing to those vision-based systems, this IMU-based system provides more flexibility on capturing outdoor motions that are important for many robotic applications.
This contribution presents a new system for fast and intuitive industrial robot reprogramming. It is based on a luminous marker built with high-intensity LEDs, which are captured by a set of industrial cameras. Using stereoscopy, the marker supplies 6-DoF human wrist tracking with both position and orientation data. This marker can be efficiently attached to any working tool which then provides a way to capture human skills without further intrusion in the tasks. The acquisition technique makes the tracking very robust against lighting conditions so no environment preparation is needed. The robot is automatically programmed from the demonstrated task which delivers complete abstraction of programming concepts. The system is able to perform in real time, and is low-cost starting with a single pair of industrial cameras though more can be used for improved effectiveness and accuracy. The real-time feature means that the robot is ready to perform as soon as the demonstration is over which carries no overhead of reprogramming times. Also, there is no interference with the task itself since the marker is attached to the work tool and the tracking is contactless; the human operator can then perform naturally. The test bed is a real industrial environment: a spray painting application. A prototype has been developed and installed, and is currently in operation. The tests show that the proposed system enables transferring to the machine the human ability of manipulating a spray gun.
This paper focusses on the application of intelligent control techniques (neural networks, fuzzy logic and genetic algorithms) and their hybrid forms (neuro-fuzzy networks, neuro-genetic and fuzzy-genetic algorithms) in the area of humanoid robotic systems. Overall, this survey covers a broad selection of examples that will serve to demonstrate the advantages and disadvantages of the application of intelligent control techniques.