Abstract
Robots will eventually make part of our daily lives, helping us at home, taking care of the elderly, and collaborating at work. In such Human-Robot Collaboration (HRC) scenarios, achieving abnormal movement identification can effectively deal with unexpected anomalies such as human collisions, external perturbations, and unexpected changes in the environment. To this end, Long-short Term Memory (LSTMs) based prediction methods are widely proposed for abnormal identification, which typically has an implicit requirement of fixed-length input signals such that the identification thresholds are calculated from the prediction-error sequences with the same length. However, in robotics, this is rarely the case, generalization in HRC is a desirable characteristic that indicates the recorded executions would have different lengths for a specific movement. To address this problem, we first extend the concept of stacked LSTMs to predict anomalies by admitting the input multivariate time series of varying lengths. Subsequently, prediction errors with different lengths are modeled using a probabilistic model for tackling the temporal uncertainty. Consequently, dynamic threshold representation is learned from the trained probabilistic model for abnormal movement identification. A self-designed robot manipulation task consisting of six individual movements is used to evaluate the proposed approach and compared it to baselines. Experimental results indicate that the proposed method with an average anomaly identification accuracy of 94% outperforms the baselines.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Park, D., Hoshi, Y., Kemp, C.C.: A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder. IEEE Robot. Autom. Lett. 3(3), 1544–1551 (2018)
Heo, Y.J., Kim, D., Lee, W., Kim, H., Park, J., Chung, W.K.: Collision detection for industrial collaborative robots: a deep learning approach. IEEE Robot. Autom. Lett. 4(2), 740–746 (2019)
Husein, A.M., Arsyal, M., Sinaga, S., Syahputa, H.: Generative adversarial networks time series models to forecast medicine daily sales in hospital. SinkrOn 3(2), 112–118 (2019)
Cao, P., Gan, Y., Dai, X.: Model-based sensorless robot collision detection under model uncertainties with a fast dynamics identification. Int. J. Adv. Robot. Syst. 16(3), 1729881419853713 (2019)
Birjandi, S.A.B., Kühn, J., Haddadin, S.: Observer-extended direct method for collision monitoring in robot manipulators using proprioception and imu sensing. IEEE Robot. Autom. Lett. 5(2), 954–961 (2020)
Kaur, U., Khan, U., Chauhan, N.R., Mukherjee, S.: Collision detection and inverse dynamics control of kuka lbr iiwa robot. Int. J. Mechatron. Autom. 8(1), 9–21 (2021)
Zhang, Y., Zhu, W., Rosendo, A.: Qr code-based self-calibration for a fault-tolerant industrial robot arm. IEEE Access 7, 73349–73356 (2019)
Haddadin, S., De Luca, A., Albu-Schäffer, A.: Robot collisions: a survey on detection, isolation, and identification. IEEE Trans. Robot. 33(6), 1292–1312 (2017)
Zanchettin, A.M., Ceriani, N.M., Rocco, P., Ding, H., Matthias, B.: Safety in human-robot collaborative manufacturing environments: Metrics and control. IEEE Trans. Autom. Sci. Eng. 13(2), 882–893 (2015)
Zacharaki, A., Kostavelis, I., Gasteratos, A., Dokas, I.: Safety bounds in human robot interaction: A survey. Safe. Sci. 127, 104667 (2020)
Buda, T.S., Caglayan, B., Assem, H.: Deepad: A generic framework based on deep learning for time series anomaly detection. In: Pacific-Asia conference on knowledge discovery and data mining, pp 577–588. Springer (2018)
Ijspeert, A.J., Nakanishi, J., Hoffmann, H., Pastor, P., Schaal, S.: Dynamical movement primitives: learning attractor models for motor behaviors. Neural Comput. 25(2), 328–373 (2013)
Calinon, S.: Learning from demonstration (programming by demonstration), Encyclopedia of Robotics (2009)
Huang, Y., Rozo, L., Silvério, J., Caldwell, D.G.: Kernelized movement primitives. Int. J. Robot. Res. 38(7), 833–852 (2019)
Munir, M., Siddiqui, S.A., Dengel, A., Ahmed, S.: Deepant: a deep learning approach for unsupervised anomaly detection in time series. IEEE Access 7, 1991–2005 (2018)
Di Lello, E., De Laet, T., Bruyninckx, H.: Hierarchical Dirichlet Process Hidden Markov Models for Abnormality Detection in Robotic Assembly. In: Workshop on Bayesian Nonparametric Models (BNPM) for Reliable Planning and Decision-Making under Uncertainty, NIPS, vol. 2012 (2012)
Di Lello, E., Klotzbücher, M., De Laet, T., Bruyninckx, H.: Bayesian time-series models for continuous fault detection and recognition in industrial robotic tasks. In: 2013 IEEE/RSJ international conference on intelligent robots and systems. IEEE, pp 5827–5833 (2013)
Nguyen, H., Tran, K.P., Thomassey, S., Hamad, M.: Forecasting and anomaly detection approaches using lstm and lstm autoencoder techniques with the applications in supply chain management. Int. J. Inform. Manag. 57, 102282 (2021)
Taylor, A., Leblanc, S., Japkowicz, N.. In: 2016 IEEE international conference on data science and advanced analytics (DSAA), pp 130–139. IEEE (2016)
Ergen, T., Kozat, S.S.: Unsupervised anomaly detection with lstm neural networks. IEEE Trans. Neural Netw. Learn. Syst. 31(8), 3127–3141 (2019)
Shewalkar, A., Nyavanandi, D., Ludwig, S.A.: Performance evaluation of deep neural networks applied to speech recognition: rnn, lstm and gru. J. Artif. Intell. Soft Comput. Res. 9(4), 235–245 (2019)
Sak, H., Senior, A., Beaufays, F.: Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In: Fifteenth annual conference of the international speech communication association (2014)
Wang, R., Nie, K., Wang, T., Yang, Y., Long, B.: Deep learning for anomaly detection. In: Proceedings of the 13th international conference on web search and data mining, pp 894–896 (2020)
An, J., Cho, S.: Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE 2(1), 1–18 (2015)
Li, D., Chen, D., Jin, B., Shi, L., Goh, J., Ng, S.-K.: Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In: International conference on artificial neural networks, pp 703–716. Springer (2019)
Shao, H., Soong, B.-H.: Traffic Flow Prediction with Long Short-Term Memory Networks (Lstms). In: 2016 IEEE Region 10 Conference (TENCON), pp 2986–2989. IEEE (2016)
Gangopadhyay, T., Y. Tan, S., Huang, G., Sarkar, S.: Temporal attention and stacked lstms for multivariate time series prediction (2018)
Hundman, K., Constantinou, V., Laporte, C., Colwell, I., Soderstrom, T.: Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 387–395. ACM (2018)
Bontemps, L., McDermott, J., Le-Khac, N.-A., et al.: Collective anomaly detection based on long short-term memory recurrent neural networks. In: International conference on future data and security engineering, pp 141–152. Springer (2016)
Malhotra, P., Vig, L., Shroff, G., Agarwal, P.: Long short term memory networks for anomaly detection in time series. In: Proceedings, Presses universitaires de Louvain, p 89 (2015)
Chauhan, S., Vig, L. IEEE, DSAA (2015)
Wu, H., Guan, Y., Rojas, J.: Analysis of multimodal bayesian nonparametric autoregressive hidden markov models for process monitoring in robotic contact tasks. Int. J. Adv. Robot. Syst. 16(2), 1729881419834840 (2019)
Wu, H., Guan, Y., Rojas, J.: A latent state-based multimodal execution monitor with anomaly detection and classification for robot introspection. Appl. Sci. 9(6), 1072 (2019)
Ralaivola, L., D’Alché-Buc, F.: Time series filtering, smoothing and learning using the kernel kalman filter. In: Proceedings. 2005 IEEE international joint conference on neural networks, 2005, vol. 3, pp 1449–1454. IEEE (2005)
Maeda, G., Ewerton, M., Neumann, G., Lioutikov, R., Peters, J.: Phase estimation for fast action recognition and trajectory generation in human–robot collaboration. Int. J. Robot. Res. 36(13-14), 1579–1594 (2017)
Maaten, L.v.d., Hinton, G.: Visualizing data using t-sne. J. Machine Learn. Res. 9(Nov), 2579–2605 (2008)
Acknowledgments
We thank Shuangda Duan and Dong Liu for their assistance on system development, and Prof. Juan Rojas for his revision on English writing throughout this paper.
Funding
This work is supported by Guangdong Province Key Areas R&D Program (Grant No. 2019B090919002), Basic and Applied Basic Research Project of Guangzhou (Grant No. 202002030237), GDAS’ Project of Thousand doctors(post-doctors) Introduction (2020GDASYL-20200103128), Foshan Key Technology Research Project(Grant No. 1920001001148), Guangdong Province International Cooperation Project of Science and Technology (Grant No. 2019A050510040), National Science Foundation of China (Grant No. 61950410758).
Author information
Authors and Affiliations
Contributions
This paper has six authors. Authors made most of the contributions on conceptualization, development of theory, validation, verification of the analytical methods, discussion of results, and contributed to the final manuscript. Individual contributions follow: the draft writing, methodology, and development of theory finished by Hongmin Wu; the code development, experimentation, and verification of the analytical methods did by Wu Yan and Zhihao Xu; the project administration and discussion of results did by Shuai Li and Taobo Cheng; Xuefeng Zhou provided the formal analysis and investigation of this study.
Corresponding author
Ethics declarations
Ethics approval
This study did not require ethics approval as the information consists of naturalistic observations regarding choice. Choices remained anonymous. There is not any identifier information that would allow attribution of private information to an individual.
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Consent to participate
Informed consent was obtained from all participants in the study.
Consent to publish
Participant consented to the submission of this article to the journal.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wu, H., Yan, W., Xu, Z. et al. Multimodal Prediction-Based Robot Abnormal Movement Identification Under Variable Time-length Experiences. J Intell Robot Syst 104, 8 (2022). https://doi.org/10.1007/s10846-021-01496-x
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-021-01496-x