Careful with That! Observation of Human
Movements to Estimate Objects Properties
arXiv:2103.01555v2 [cs.RO] 10 Mar 2021
Linda Lastrico*1,2 , Alessandro Carfı̀1 , Alessia Vignolo3 , Alessandra Sciutti3 ,
Fulvio Mastrogiovanni1 and Francesco Rea2
1
Dipartimento di Informatica, Bioingegneria, Robotica e Ingegneria dei Sistemi
(DIBRIS), Università degli Studi di Genova, Genova, Italy
2
Robotics, Brain and Cognitive Science Department (RBCS), Italian Institute of
Technology, Genova, Italy
linda.lastrico@iit.it
3
Cognitive Architecture for Collaborative Technologies Unit (CONTACT), Italian
Institute of Technology, Genova, Italy
Abstract. Humans are very effective at interpreting subtle properties
of the partner’s movement and use this skill to promote smooth interactions. Therefore, robotic platforms that support human partners in
daily activities should acquire similar abilities. In this work we focused
on the features of human motor actions that communicate insights on
the weight of an object and the carefulness required in its manipulation.
Our final goal is to enable a robot to autonomously infer the degree of
care required in object handling and to discriminate whether the item
is light or heavy, just by observing a human manipulation. This preliminary study represents a promising step towards the implementation of
those abilities on a robot observing the scene with its camera. Indeed,
we succeeded in demonstrating that it is possible to reliably deduct if
the human operator is careful when handling an object, through machine
learning algorithms relying on the stream of visual acquisition from either a robot camera or from a motion capture system. On the other
hand, we observed that the same approach is inadequate to discriminate
between light and heavy objects.
Keywords: Biological motion kinematics · Human motion understanding · Natural communication · Deep learning · Human-robot interaction
1
Introduction and background
In the context of human-robot interaction, a great effort is directed towards the
development of the robot ability to understand implicit signals and subtle cues
that naturally characterize human movements. This comes to have critical importance in situations where robots are used in unconstrained environments, for
instance in manufacturing, helping human operators to lift loads or assisting elderly. In typical human-human interaction a considerable amount of information
is exchanged through non-verbal signals, such as the attitude of an action, its
tempo, the direction of the gaze and the body posture. It has been proved that
2
Lastrico et al.
people are able to correctly estimate the weight of an object, simply observing
another person lifting it [15]. Recent research confirmed that the same information could be transmitted by a humanoid robot controlling the vertical velocity
of its lifting movements [8]. Humans manage to easily manipulate objects they
have never used before: at first, by inferring their properties such as the weight,
the stiffness and the dimensions also from the observation of others manipulating
them; at a later time, using tactile and force feedback to improve the estimation.
Replicating this behaviour in robotic systems is challenging. However, preliminary results have been achieved in estimating objects physical properties, relying
on inference-based vision approaches [13].
The interaction with humanoid robots is particularly critical: driven by their
appearance, humans tend to attribute those robots human-like abilities and, if
their expectations fall short, the interaction may fail [14]. Humans strongly rely
on implicit signals to cooperate; therefore, in this context, to obtain seamless
human-robot collaboration, humanoid robots need to correctly interpret those
implicit signals [3]. Furthermore, if we consider a scenario where the robot acts
as helper or partner in an unconstrained environment, it acquires great importance to endow it with the ability of correctly estimating the characteristics of
the handled objects; as a consequence, the robot can plan a safe and efficient
motion action. In this study, we give particular attention to how a robot could
assess an object features just by seeing it transported by a human partner. Inferring those properties from the human kinematics during the manipulation of
the objects, rather than from their external appearance, grants the ability of
generalizing over previously unseen items.
1.1
Rationale
Suppose to transport a glass full to the brim with water: the effort required
to safely manage it without spilling a drop resembles the challenging scenario
of porting an electronic device that could be damaged. If we want a robot to
perform the same action, the first step would be to give the robot the capability
of recognizing the intrinsic difficulty of the task; if we consider a hand-over task
between a human and a robot, the latter should be aware that it is about to
receive an object that requires a certain degree of carefulness in the handling.
Moreover, an assessment of the weight of the object would allow an efficient lift.
These features could be estimated from the human motion and ideally should
be available before the end of the observed action, to trace the human abilities
and to allow the robot to prepare for the possible handover. Differently from
the weight, the concept of carefulness is not trivial. Previous studies have dealt
with delicate objects, but focused more on robotic manipulation: the difficulty
in the addressed tasks was given from the stiffness or the deformability of the
item; tactile sensors where used for estimating the necessary force to apply a
proper grasp [12,16]. In our study we consider the carefulness necessary to move
an item from a wider perspective. Indeed, not only the object stiffness but also
its fragility, the content about to be spilled, or its sentimental value may lead
a person to perform a particularly careful manipulation. In those real-life cases
Observation of Human Movements to Estimate Objects Properties
3
we would like the robot to successfully estimate the carefulness required just by
observing the human kinematics. As a proof of concept, we recorded some transportation movements involving glasses which differed for weight and carefulness
levels and some kinematic features, derived from the human motion, were used
to train classifier algorithms. These features were obtained for comparison both
from a motion capture system and a robot camera. We hypothesize that, from
the knowledge of kinematic features descriptive of the human movement: (H1)
it is possible to infer if carefulness is required to manipulate an object and (H2)
it is possible to discriminate a lighter object from a heavier one. To validate our
hypothesis we have collected a dataset of human motions while performing a
simple transporting task; then we have trained state-of-the-art classifiers to determine if it is possible to distinguish the carefulness associated with an object
and its weight, exclusively observing the human motion.
2
Experimental setup
The experimental setup used to collect the data consisted of a table, a chair,
two shelves (placed on different sides of the table) facing the volunteer, a scale,
a keyboard with only one functioning key, and four plastic glasses (see Fig. 1).
(a) Lateral view
(b) Top view
Fig. (1) Two views of the experimental setup with the volunteers in two grasp
positions: on the shelf (1a) and on the scale (1b)
Table (1)
Glasses features and abbreviations
Abbreviation
W1C1
W2C1
W1C2
W2C2
Weight (gr) Carefulness level
167
low (no water)
667
low (no water)
167
high (full of water)
667
high (full of water)
4
Lastrico et al.
The four glasses were characterized by two levels of associated carefulness and
two weights, as shown in Table 1. The high level of carefulness was achieved filling
the glass to the brim with water, while for the low level no water was placed in the
glass. The different weights, instead, were obtained by inserting in the glasses a
variable number of coins and screws; for the object with high level of carefulness
the weight of the water was taken into account. Each object was weighted to
guarantee a difference of 500 gr between light an heavy glasses. Glasses were
identical in shape and appearance, and their transparency was chosen so that
participants could clearly see the content of the glass and appropriately plan
their movements . As displayed in Fig. 1, four positions were defined in each
shelf, two on the top and two on the bottom level. These predefined positions
were identified by a letter on a label.
Participants seated at the table and had to perform a structured series of
reaching, lifting and transportation movements of the four glasses. The experiment started with all the four glasses on the shelves, the volunteer with their
arms resting on the table and their right hand in the resting pose, marked with
a blue cross (see Fig. 1b). During the experiment, the volunteers used their right
hand to interact with the objects and their left to press the key of the keyboard.
The experiment was structured as following:
– The volunteer pressed the key of the keyboard and a synthetic voice indicated
the position on the shelf of the object to be transported. The position was
referred to using the corresponding letter.
– The volunteer performed a reaching action toward the specified position and
grasped the glass (see Fig. 1a).
– The volunteer performed a transportation action moving the glass from the
shelf to the scale.
– The volunteer released the glass and returned to the resting pose.
– The volunteer pressed a second time the key and the synthetic voice indicated
a position on the shelf where the glass should be transported. Of course, this
time the selected position on the shelf was empty.
– The volunteer performed a reaching action towards the scale and grasped
the glass (see Fig. 1b).
– The volunteer performed a transportation action moving the glass from the
scale to the final position on the shelf.
– The volunteer released the glass and returned to the resting pose.
The participants repeated this sequence 8 times to familiarize with the task,
while the main experiment consisted of 32 repetitions. A table containing the
shelf initial and final poses for each repetition was previously designed to guarantee a good coverage of all the possible combinations of shelf positions and
glasses. Each volunteer performed exactly the same experiment.
The experiment was conducted thanks to 15 healthy right-handed subjects that
voluntarily agreed to participate into the data collection (7 females, age: 28.6 ±
3.9). All volunteers are members of our organization but none is directly involved
in our research.
Observation of Human Movements to Estimate Objects Properties
2.1
5
Sensors
The data used in this study was collected during the experiments previously
described using a motion capture system from Optotrak, as ground truth, and
one of the cameras of iCub. During the experiments other sensors have been used
to collect data but their analysis is not in the scope of this paper. The humanoid
robot iCub was placed opposite to the table and recorded the scene through its
left camera, with a frame rate of 22 Hz and a resolution of the image of 340 x
240 pixels. The robot was just a passive observer and no interaction with the
participants took place during the experiment. The Optotrak Certus® , NDI,
motion capture (MoCap) system recorded the kinematic of the human motion
through active infrared markers at a frequency of 100 Hz. The markers were
placed on the right hand, wrist and arm. For the following analysis only a subset
of the hand and wrist markers were considered (see Fig. 2). The data coming from
the different sensors was synchronized through the middleware YARP [6] that
gave to each sample a YARP timestamp. By pressing the key on the keyboard
at the end of every trial the data coming from the MoCap were automatically
segmented in different log files and the actual timestamp saved in a separate file.
Successively the timestamps associated with the key pressures have been used
to segment the data recorded by the robot camera.
Fig. (2) Detail of the markers position on the right hand: those circled in red
were interchangeably used to compute the features in each trial
Motion capture system data The data acquired by the motion capture system
consisted in the tridimensional coordinates of each marker with respect to the
reference coordinate frame of the Optotrak. Occlusions limited the MoCap visibility for specific part of the human movement. In our experiment the main
source of occlusion was given by the presence of the shelves, in particular for
the lower right positions. To partially overcome this problem, after a preliminary analysis, we chose to consider for each trial the most visible marker among
a subset of four as representative of the movement. Indeed, during the transportation movements the hand could be assimilated to a rigid body. The four
considered markers were placed respectively on the metacarpophalangeal joints
of the index and of the little finger, on the diaphysis of the third metacarpal and
6
Lastrico et al.
on the smartwatch in correspondence of the radial styloid (see the markers circled in red in Fig. 2 for reference). Two different interpolations, inpaintn [5] and
interp1 of MATLAB ver. R2019b, have been used to reconstruct the data that
are missing because of the occlusions. The data was filtered with a second order
low pass Butterworth filter with a cutoff frequency of 10 Hz. Some trials have
been excluded from the data set because of inconsistencies in the segmentation
among the acquired sensors or because of errors of the subjects into pressing the
key at the right moment, i.e. when their right hand was laying on the table in
the resting position. Overall only 1.25% of the total acquired trials have been
removed. Since our hypothesis is that it is possible to distinguish the features of
the object that is being transported, it was necessary to isolate the transportation movement in every trial. To do so we took advantage of the experiment
design. Indeed each trial presented three clearly identifiable phases: a reaching
action, from the resting pose to the position occupied by the glass (either on the
shelf or on the scale), a transportation movement and finally the departing (see
Fig. 3). Our segmentation assumed that the start and end of the transportation
phase is associated with a pick in the norm velocity of the hand. Therefore, the
segmentation was performed by placing a threshold of 5% on the second peak of
the norm of the velocity, after filtering it with a fourth order filter with a cutoff
frequency of 5 Hz. The resulting data were then down-sampled to obtain the
same frame rate as the camera of the robot.
Fig. (3) Example of the velocity patterns from motion capture (in blue) and
optical flow data (in red). The peaks characterizing the three phases of the trial
(reaching, transportation and departing) are visible
Camera data and optical flow extraction As motion descriptor, from the saved
raw images of the robot camera (see Fig. 4 for an example) we chose to compute the Optical Flow (OF), following an approach already tested [17,18]. In
this method, the optical flow is computed for every time instant using a dense
approach [4], which estimates the apparent motion vector for each pixel of the
image. The magnitude of the optical flow is thresholded to consider only those
parts of the image where the change is significant. A temporal description of the
motion happening in the derived region of interest is then computed averaging
Observation of Human Movements to Estimate Objects Properties
7
the optical flow components. On the velocity extracted, a second order low-pass
Butterworth filter with cutoff frequency of 4 Hz was applied to remove the noise
(see Fig. 3).
(a) View from the iCub per- (b) OF moving towards the (c) OF moving towards the
spective
right of the image
left of the image
Fig. (4) Example of iCub view of the scene and the extracted OF. The colors
codify for the direction of the movement: red is for motion towards the right
part of the image (4b), blue for motion towards the left (4c)
3
Data pre-processing
The same set of motion representations was extracted during a pre-processing
phase from both the motion capture data and the optical flow: the velocity Vi (t),
the curvature Ci (t), the radius of curvature Ri (t) and the angular velocity Ai (t)
[9]. Their analytical expression is stated in Table 2. Such features can be computed for every time instant and by collecting them it is possible to progressively
gather an increasing amount of information about the observed movement. This
would then grant the robot the ability of discriminating online the characteristics of the object handled by the human partner. As shown in [17,18], these data
representations have been successfully used to discriminate online between biological and non-biological motion and to facilitate coordination in human-robot
interaction [10]. In addition, kinematics properties, such as velocity, have been
shown to be relevant in human perception of object weight [1]. Extracting those
features during the pre-processing, instead of directly feeding the classification
algorithms with raw data, allows to better compare the performance achieved
with the two sources of data. Indeed, a precise control over the information used
during the learning process is granted.
3.1
Dataset
As we have detailed before some sequences had to be removed for inconsistencies in the segmentation. This lead to a slightly unbalanced data set, containing
more examples for specific classes. Indeed, class W1C1 had 235 sequences, class
8
Lastrico et al.
Table (2)
Motion features computed from motion capture and optical flow data
Motion feature
Tangential velocity
Analytical expression
Vi (t) = (ui (t), vi (t), ∆t )
Tangential velocity magnitude Vi (t) =
p
ui (t)2 + vi (t)2 + ∆2t
Acceleration
Ai (t) = (ui (t) − ui (t − 1), vi (t) − vi (t − 1), 0)
Curvature
Ci (t) =
kVi (t)×Ai (t)k
kVi (t))k3
Radius of curvature
Ri (t) =
1
Ci (t)
Angular velocity
Ai (t) =
Vi (t)
Ri (t)
W2C1 239, class W1C2 238 and class W2C2 had 236. Although cardinally the
difference is minimum, to preserve the balance of the dataset we decided to fix
the maximum number of sequences for each class to 235 and we have randomly
selected the sequences for W2C1, W1C2 and W2C2. Notice that the four classes
were characterized only by the weight and the carefulness level. Therefore other
variables, such as the initial and final position of the glass and the direction of
the movement, are not considered in the classification.
Due to the characteristics of the glasses, the duration of the transport movement varied consistently among the trials (i.e. the duration of the movement
is consistently longer when the moved glass is full of water, belonging to the
high carefulness class). To obtain sequences with the same number of samples
for each trial, the segmented sequences were re-sampled, using the interp1 function of MATLAB. The number of samples was selected considering the class
associated with the shorter duration of the transport phase, W1C1, and computing the median value among all its trials. The resulting value was 32. Therefore, our dataset was composed of two data structures: one derived from the
MoCap data and the other one from the OF. Both structures had dimensions
940 (trials) × 32 (f rames) × 4 (f eatures).
The re-sampling can be performed only knowing the start and end of the transportation phase. Since in an online scenario this information is not available, a
further attempt was performed exploiting the ability of certain models to handle temporal sequences of different lengths. In this case, instead of re-sampling,
a common zero-padding and masking technique were adopted. Therefore, the
shorter temporal sequences were completed with zero values and those values
were then ignored during the training, while the length of the longest transport movements was preserved. The shape of the data structures after the zero
padding was: 940 (trials) × 132 (f rames) × 4 (f eatures).
Observation of Human Movements to Estimate Objects Properties
4
9
Classifiers
As introduced in Sect. 1.1, the goal of the classification is to discriminate between the two possible features of the transported glasses: (H1) the carefulness
level associated with the object and (H2) the weight. Therefore, we decided
to approach the problem using two binary classifiers, one for each feature, implemented in Python using Keras libraries [2]. As mentioned in Sect. 3.1 two
models were tested: the first one relied on re-sampled features, while the second
one used the original data with variable lengths.
4.1
Convolutional, Long-Short-Term-Memory and Deep Neural
Network
Previous literature suggests that the combined use of Convolutional Neural Network (CNN), Long-Short Term Memory (LSTM) and Deep Neural Networks
(DNN) is a good solution for classifying time dependent data, such as speech
or motion kinematics [11,7]. Therefore, our first model was inspired by [11] and
consisted of two time distributed 1-D convolutional layers (that took as input 4
subsequences of 8 frames each), a max pooling and flatten layers, a 100 neurons
LSTM, a 100 neurons Dense layer and a 2 neurons output layer with a sigmoidal
activation function. A Leave-One-Out approach was adopted, to test the ability
of the model to generalize over different participants. Thus, for each one of the
15 folds, the data from 14 subjects were used as training set and the data of the
fifteenth participant as test set. The 20% of the data for each training set was
kept for validation, and early stopping was implemented according to the validation loss function (with a patience parameter set to 5 epochs): this allowed to
obtain good accuracy without incurring in overfitting. The batch size was fixed
to 16. The model was fit with ADAM optimization algorithm and categorical
cross-entropy as loss function. With respect to the model described in [11] some
regularizers were added to avoid overfitting and make the network less sensitive to specific neurons weights. A L1-L2 kernel regularization was added to the
two 1D convolutional layers (l1 = 0.001, l2 = 0.002) and a L2 kernel regularizer
(l2 = 0.001) was added to the fully connected DNN layer; moreover, 0.5 dropouts
were introduced.
4.2
Long-Short-Term-Memory and Deep Neural Network
The second model was implemented to test the possibility of generalizing over
temporal sequences of variable length. To implement such an approach the data
were padded with zeroes, as mentioned in the previous Section. Since the required masking layer was not supported by the Keras implementation of the
CNN layer, we decided to opt for a simpler model: a 64 neurons LSTM, followed
by a 32 neurons dense layer and a 2 neurons output layer with a sigmoidal activation function; also in this case L1-L2 regularization and 0.5% dropout were
used to avoid overfitting. The optimization algorithm, the loss function and the
validation approach with early stopping were the same as before. By using this
10
Lastrico et al.
model the possibility of learning independently from the length of the temporal
sequence was granted. This represents a further step towards the implementation of the same algorithm on the robot; indeed, no previous knowledge on the
duration of the movement would be required to perform the classification, since
the model is trained on variable temporal sequences.
5
Results
Results for the classifiers performance are presented for both the weight and
the carefulness features and for both the considered source of data: the motion
capture and the optical flow from the robot camera.
5.1
Carefulness level
The performances in the classification of the carefulness level with the model
presented in Sect. 4.1 are reported in Table 3.
Table (3) Model accuracy (%, mean and standard deviation) on carefulness
level classification with the CNN-LSTM-DNN model. In brackets are the results
when volunteer 8 was included in the data set
Motion capture
Optical flow
Training 92.15(92.00) ± 2.14(3.42) 94.03(92.18) ± 1.05(1.00)
91.68(90.97) ± 5.00(11.12) 90.54(89.43) ± 6.56(7.59)
Test
(a) MoCap accuracy
(b) OF accuracy
Fig. (5) Accuracy in the carefulness classification with CNN-LSTM-DNN
model for the validation set for each fold. Accuracy from motion capture (5a)
and from optical flow features (5b)
Observation of Human Movements to Estimate Objects Properties
11
When performing the Leave-One-Out cross validation we noticed that the classification accuracy associated to volunteer 8 was significantly lower than the average (MoCap test all: 90.97 ± 11.12 vol8: 51.62; OF test all: 89.43 ± 7.59 vol8:
77.42). Examining the experiment videos we noticed that the volunteer 8 was
particularly careful even when handling the glasses not containing water. Our
impression was confirmed after computing the median duration of the not careful
movements among the subjects. The duration for volunteer 8 (2.04±0.18 seconds,
median and median absolute deviation) differed significantly from the ones of
the other participants, as for the rest of the group the median duration resulted
1.47 ± 0.15 seconds (Kruskal-Wallis test: χ2 (14, N = 480) = 136.8, p < .01). In
Table 3 we have reported in brackets the results when including this subject in
the dataset. As can be observed, when the participant was included the accuracy
variance on the test increased significantly for each the sensing modalities.
Figure 5 shows the trend of the accuracy over the epochs for the validation set
of each one of the folds. Comparing the graphs for the two sources of data ((a)
motion capture, (b) optical flow) it can be noticed how the first one reaches an
accuracy above the 80% in less than 10 epochs, while, using the features from the
optical flow, more training is necessary to reach the same level of accuracy (over
20 epochs). Furthermore, the accuracy trend of the motion capture features is
more stable.
Similarly, the carefulness classification performance with the model presented
in Sect. 4.2, fed with the original temporal sequences of variable lengths, is shown
in Table 4. As before, the variability in the test accuracy reduced when volunteer
8 is excluded from the dataset, and the overall accuracy improved for both the
sensing modalities. With this model, compared to the values in Table 3, the
accuracy achieved with the MoCap data is higher, while the one of the OF
slightly reduced.
Table (4) Model accuracy (%, mean and standard deviation) on carefulness
level classification for simpler LSTM-DNN model. In brackets the results considering volunteer 8
Motion capture
Optical flow
Training 96.57(94.32) ± 1.19(1.77) 92.10(90.39) ± 4.58(2.56)
95.17(92.66) ± 5.56(8.49) 88.38(86.50) ± 8.68(10.75)
Test
5.2
Weight
In Table 5 are shown the results for the classification of the weight achieved with
re-sampled data on the first implemented model. In this case, volunteer 8 did
not present any peculiarity and therefore it was included in the dataset. As we
can observe in Table 5, the accuracy with the motion capture data is above 60%
and is higher than the one obtained from the optical flow.
12
Lastrico et al.
Table (5) Model accuracy (%, mean and standard deviation) on weight classification with the CNN-LSTM-DNN model, fed with re-sampled data
Training
Test
Motion capture Optical flow
64.10 ± 2.34
55.24 ± 2.37
61.83 ± 7.16
54.47 ± 4.29
Finally, Table 6 reports the accuracy for the weight classification with the LSTMDNN model, fed with the original temporal sequences of different lengths. In this
case the performance was comparable between the data from the two sensing
modalities.
Table (6) Model accuracy (%, mean and standard deviation) on weight level
classification for the second model, LSTM-DNN
Motion capture Optical flow
Training
54.95 ± 2.66
55.30 ± 1.95
Test
54.75 ± 5.27
53.29 ± 3.59
We have noticed that, despite adopting the same approach, the accuracy on the
weight classification is not as satisfying as the one achieved for the carefulness. A
possible explanation of these results could be related to the different effect that
weight may have on different transport movements. Possibly the weight influence
varies if the transportation is from top to bottom or vice-versa. Furthermore,
the presence of water in some of the glasses may have led the subjects to focus
mainly on the carefulness feature, unconsciously overlooking the weight difference. Therefore, we add two specifications of the second hypothesis: (H2.1) the
influence of the weight during transportation is dependent on the trajectory of
the motion; (H2.2) when an object is associated with an high level of carefulness, the weight has a limited influence on the transportation movement. Both
hypotheses were tested with the first model, which gave better results for the
weight classification. Concerning the first hypothesis, we reduced the variability
in the movements and tried to discriminate the weight in the subset of transport
movements from the scale towards the shelves (MoCap: Tr : 68.90 ± 2.68 Test:
63.42 ± 8.96; OF: Tr : 59.10 ± 4.27 Test: 55.17 ± 6.24); there is a slight improvement for both the data sources compared to the values in Table 5. Notice that
the trajectories still have a discrete amount of variability since the position to
reach on the shelf could be left or right, high or low. The second hypothesis
was investigated by testing the weight discrimination within the subset of objects which required the same carefulness level: low (MoCap: Tr : 64.49 ± 5.24
Test: 61.93 ± 6.86; OF: Tr : 62.52 ± 3.53 Test: 56.84 ± 6.77) or high (MoCap:
Tr : 62.72 ± 3.65 Test: 59.03 ± 8.73; OF: Tr : 57.92 ± 1.31 Test: 53.48 ± 7.63).
Observation of Human Movements to Estimate Objects Properties
13
For both the tests the results are inconclusive, since the classification accuracies have not changed much respect to the ones reported in Table 5. It should
be noted though that the dimension of the dataset used to validate hypotheses
(H2.1) and (H2.2) halved, which has an impact on the statistical relevance of
the results.
6
Discussion
Regarding the carefulness feature, as reported in Table 3 the first classifier is able
to correctly discriminate if the transportation of the object requires carefulness or
not, independently from the sensing modality used. Considering the performance
on the data coming from the two sources, no significant difference is detected
between them. Therefore, not only using an accurate system such as the motion
capture, that integrates sensory inputs from different locations to estimate the
position in space of the target, but also using the camera of the robot (single point
of view), it is possible to extract features to discriminate between careful and not
careful motions. Figure 5 shows an insight on how the learning process advanced
for the two data sources. Even though the final performances are comparable,
it can be appreciated how the model trained with the features from the motion
capture converges quicker to an accuracy value above the 80%.
The approach adopted with the second classifier is more general, in the sense
that data are not re-sampled to have the same dimension but the variability in
their duration is taken into account. Even though this model is simpler, with just
one LSTM and one dense layer, the performance on the carefulness classification
considering the MoCap data increased (see Table 4 for reference). Although the
accuracy using the optical flow is slightly lower, we consider this as a promising
step towards the implementation of the same algorithm on the robot.
Concerning the weight, the accuracy achieved for both the sensing modalities and for both the models is lower than the one obtained for the carefulness
(see Tables 5 and 6 for reference). To explain this outcome in Sect. 5.2 we have
formalized two additional hypotheses. (H2.1) was inspired by [8], where it has
been proposed that the vertical component of the velocity during the manipulation of an object is perceived by humans as informative about its weight.
Since the trials in our dataset explored a variety of directions and elevations,
this introduced a great variability in the vertical component of the velocity. Instead, concerning (H2.2), we have supposed that the greatest challenge for the
volunteers during the experiment is to safely handle the glasses full of water;
the difference in weight between the objects was not remarkable in comparison
with the stark contrast given by the presence (or absence) of water. As mentioned in Sect. 5.2 the first classifier was tested against these hypotheses, but
no significant improvements in the accuracy have been achieved. Given the results of our experiment we can not validate hypothesis (H2). However, since we
have explored only a subset of the possible kinematic features we can not argue
against this hypothesis either. A possibility for future works is to focus on the
14
Lastrico et al.
vertical component of the velocity. Furthermore, (H2.1) and (H2.2) should be
explored on reasonably extended datasets to obtain more reliable results.
7
Conclusions
As human-robot interactions are becoming increasingly frequent, it is crucial
that robots gain certain abilities, such as the correct interpretation of implicit
signals associated with the human movement. In this study we focused on two
fundamental implicit signals commonly communicated in human movements: the
impact of the weight and the carefulness required in the object manipulation
(e.g. transport of fragile, heavy and unstable objects). Our hypotheses aimed to
demonstrate that it is possible to discriminate between lighter and heavier items
(H2) and to infer the carefulness required by human operator in manipulating
objects (H1). We proved that it is feasible to reliably discriminate when the
human operator recruits motor commands of careful manipulation during the
transportation of an object. Indeed, it is reliable to estimate extreme carefulness
from two different typologies of sensory acquisition: from motion tracking system
and from the single view point of the robot‘s camera observing the movement.
On the other hand, the proposed algorithms show lower accuracy when applied
to weight classification, and these results does not allow us to validate our second
hypothesis. The estimation of the weight from human motion should be subject
of further studies, exploring other classification strategies or kinematic features
subset (e.g. extraction of the vertical components of the velocity during manipulation). This study firmly supports the research in human-robot interaction,
especially in the direction of addressing complex situations in realistic settings
(e.g.: industrial environment, construction site, home care assistance, etc.). In
these specific scenarios the robot can autonomously leverage on insights inferred
from implicit signals, such as the carefulness required to move a object, in order
to facilitate the cooperation with the human partner.
Acknowledgement
This paper is supported by CHIST-ERA, (2014-2020) project InDex (Robot
In-hand Dexterous manipulation).
References
1. Bingham, G.: Kinematic form and scaling: further investigations on the visual
perception of lifted weight. Journal of experimental psychology. Human perception
and performance 13(2), 155—177 (1987)
2. Chollet, F., et al.: Keras. https://keras.io (2015)
3. Dragan, A.D., Lee, K.C.T., Srinivasa, S.S.: Legibility and predictability of robot
motion. In: Proceedings of the 8th ACM/IEEE International Conference on
Human-Robot Interaction, pp. 301–308. Tokyo, Japan (2013)
Observation of Human Movements to Estimate Objects Properties
15
4. Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In:
In: Proceedings of the 13th Scandinavian Conference on Image Analysis, LNCS
2749, pp. 363–370. Gothenburg, Sweden (2003)
5. Garcia, D.: Robust smoothing of gridded data in one and higher dimensions with
missing values. Computational Statistics and Data Analysis 54, 1167–1178 (2010)
6. Metta, G., Fitzpatrick, P., Natale, L.: Yarp: Yet another robot platform. International Journal of Advanced Robotic Systems 3 (2006)
7. Neverova, N., Wolf, C., Lacey, G., Fridman, L., Chandra, D., Barbello, B., Taylor,
G.: Learning human identity from motion patterns. IEEE Access 4, 1810–1820
(2016)
8. Palinko, O., Sciutti, A., Patane, L., Rea, F., Nori, F., Sandini, G.: Communicative
lifting actions in human-humanoid interaction. In: Proceedings of the 14th IEEERAS International Conference on Humanoid Robots, pp. 1116–1121. Madrid, Spain
(2015)
9. Press, C.: Action observation and robotic agents: Learning and anthropomorphism.
Neuroscience & Biobehavioral Reviews 35(6), 1410 – 1418 (2011)
10. Rea, F., Vignolo, A., Sciutti, A., Noceti, N.: Human motion understanding for
selecting action timing in collaborative human-robot interaction. Frontiers in
Robotics and AI 6, 58 (2019)
11. Sainath, T.N., Vinyals, O., Senior, A., Sak, H.: Convolutional, long short-term
memory, fully connected deep neural networks. In: Proceedings of the 2015 IEEE
ICASSP, pp. 4580–4584. Brisbane, Australia (2015)
12. Sanchez, J., Corrales, J.A., Bouzgarrou, B.C., Mezouar, Y.: Robotic manipulation
and sensing of deformable objects in domestic and industrial applications: a survey.
The International Journal of Robotics Research 37(7), 688–716 (2018)
13. Sanchez-Matilla, R., Chatzilygeroudis, K., Modas, A., Duarte, N.F., Xompero, A.,
Frossard, P., Billard, A., Cavallaro, A.: Benchmark for human-to-robot handovers
of unseen containers with unknown filling. IEEE Robotics and Automation Letters
5(2), 1642–1649 (2020)
14. Sandini, G., Sciutti, A.: Humane robots—from robots with a humanoid body to
robots with an anthropomorphic mind. ACM Transactions on Human-Robot Interaction 7, 1–4 (2018)
15. Sciutti, A., Patane, L., Nori, F., Sandini, G.: Understanding object weight from
human and humanoid lifting actions. Autonomous Mental Development, IEEE
Transactions 6, 80–92 (2014)
16. Su, Z., Hausman, K., Chebotar, Y., Molchanov, A., Loeb, G.E., Sukhatme, G.S.,
Schaal, S.: Force estimation and slip detection/classification for grip control using a biomimetic tactile sensor. In: Proceedings of 15th IEEE-RAS International
Conference on Humanoid Robots, pp. 297–303. Seoul, Korea (2015)
17. Vignolo, A., Noceti, N., Rea, F., Sciutti, A., Odone, F., Sandini, G.: Detecting
biological motion for human–robot interaction: A link between perception and
action. Frontiers in Robotics and AI 4, 14 (2017)
18. Vignolo, A., Rea, F., Noceti, N., Sciutti, A., Odone, F., Sandini, G.: Biological
movement detector enhances the attentive skills of humanoid robot icub. In: Proceedings of the 16th IEEE-RAS International Conference on Humanoid Robots.
Cancun, Mexico (2016)