Abstract
In order to satisfy the robotic personalized service requirements that can select exclusive items to perform inference and planning according to different service individuals, the service robots need to have the ability to independently obtain the ownership relationship between humans and their carrying items. In this work, we present a novel semantic learning strategy for item ownership. Firstly, a human-carrying-items detection network based on human posture estimation and object detection model is used. Then, the transferred convolutional neural network is used to extract the characteristics of the objects and the back-end classifier to recognize the object instance. At the same time, the face detection and recognition model are used to identify the service individual. Finally, on the basis of the former two, the active learning of ownership items is completed. The experimental results show that the proposed ownership semantic learning strategy can determine the ownership relationship of private goods accurately and efficiently. The solution of this problem can improve the intelligence level of robot life service.
Similar content being viewed by others
References
H. Wu, X. J. Wu, Q. Ma, G. H. Tian. Cloud robot: Semantic map building for intelligent service task. Applied Intelligence, vol.49, no. 2, pp. 319–334, 2019. DOI: https://doi.org/10.1007/s10489-018-1277-0.
W. He, Z. J. Li, C. L. P. Chen. A survey of Human-centered Intelligent robots: Issues and challenges. IEEE/CAA Journal of Automatica Sinica, vol.4, no.4, pp. 602–609, 2017. DOI: https://doi.org/10.1109/JAS.2017.7510604.
Y. Yang, F. Qiu, H. Li, L. Zhang, M. L. Wang, M. Y. Fu. Large-scale 3D semantic mapping using stereo vision. International Journal of Automation and Computing, vol.15, no. 2, pp. 194–206, 2018. DOI: https://doi.org/10.1007/s11633-018-1118-y.
E. Daglarh, S. F. Daglarh, G. O. Giinel, H. Kose. Improving human-robot interaction based on joint attention. Applied Intelligence, vol.47, no. 1, pp. 62–82, 2017. DOI: https://doi.org/10.1007/sl0489-016-0876-x.
T. M. Wang, Y. Tao, H. Liu. Current researches and future development trend of intelligent robot: A review. International Journal of Automation and Computing, vol.15, no. 5, pp. 525–546, 2018. DOI: https://doi.org/10.1007/s11633-018-1115-1.
J. Rivera-Rubio, S. Idrees, I. Alexiou, L. Hadjilucas, A. A. Bharath. Small hand-held object recognition test (SHORT). In Proceedings of IEEE Winter Conference on Applications of Computer Vision, IEEE, Steamboat Springs, USA, pp. 524–531, 2014. DOI: https://doi.org/10.1109/WACV.2014.6836057.
J. Rivera-Rubio, S. Idrees, I. Alexiou, L. Hadjilucas, A. A. Bharath. A dataset for hand-held object recognition. In Proceedings of IEEE International Conference on Image Processing, IEEE, Paris, France, pp. 5881–5885, 2014. DOI: https://doi.org/10.1109/ICIP.2014.7026188.
J. W. Hsieh, J. C. Cheng, L. C. Chen, C. H. Chuang, D. Y. Chen. Handheld object detection and its related event analysis using ratio histogram and mixture of HMMs. Journal of Visual Communication and Image Representation, vol. 25, no. 6, pp. 1399–1415, 2014. DOI: https://doi.org/10.1016/j.jvcir.2014.05.009.
X. Lv, S. Q. Jiang, L. Herranz, S. Wang. RGB-D handheld object recognition based on heterogeneous feature fusion. Journal of Computer Science and Technology, vol. 30, no.2, pp.340–352, 2015. DOI: https://doi.org/10.1007/s11390-015-1527-0.
X. Li, S. Q. Jiang, X. Lv, C. P. Chen. Learning to recognize hand-held objects from scratch. In Proceedings of the 17th Pacific-Rim Conference on Multimedia, Springer, Xi’an, China, pp. 527–539, 2016. DOI: https://doi.org/10.1007/978-3-319-48896-7_52.
K. Yamaguchi, M. Hadi Kiapour, L. E. Ortiz, T. L. Berg. Parsing clothing in fashion photographs. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Providence, USA, pp. 3570–3577, 2012. DOI: https://doi.org/10.1109/CVPR.2012.6248101.
X. D. Liang, C. Y. Xu, X. H. Shen, J. C. Yang, S. Liu, J. H. Tang, L. Lin, S. C. Yan. Human parsing with contextualized convolutional neural network. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.39, no. 1, pp. 115–127, 2017. DOI: https://doi.org/10.1109/TPAMI.2016.2537339.
K. Gong, X. D. Liang, D. Y. Zhang, X. H. Shen, L. Lin. Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 6757–6765, 2017. DOI: https://doi.org/10.1109/CVPR.2017.715.
X. J. Chen, R. Mottaghi, X. B. Liu, S. Fidler, R. Urtasun, A. Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 1979–1986, 2014. DOI: https://doi.org/10.1109/CVPR.2014.254.
J. S. Li, J. Zhao, Y. C. Wei, C. Y. Lang, Y. D. Li, T. Sim, S. C. Yan, J. S. Feng. Multiple-human parsing in the wild, Available: https://arxiv.org/abs/1705.07206, March 2018.
Z. Cao, T. Simon, S. E. Wei, Y. Sheikh. Realtime multi-person 2D pose estimation using part affinity fields. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 1302–1310, 2017. DOI: https://doi.org/10.1109/CVPR.2017.143.
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, A. C. Berg. SSD: Single shot MultiBox detector. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 21–37, 2016. DOI: https://doi.org/10.1007/978-3-319-46448-02.
P. Felzenszwalb, D. McAllester, D. Ramanan. A discrim-inatively trained, multiscale, deformable part model. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Anchorage, USA, 2008. DOI: https://doi.org/10.1109/CVPR.2008.4587597.
R. Girshick, J. Donahue, T. Darrell, J. Malik. Rich feature hierarchies for accurate object detection and semantic seg-mentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 580–587, 2014. DOI: https://doi.org/10.1109/CVPR.2014.81.
R. Girshick. Fast R-CNN. In Proceedings of 2015 IEEE International Conference on Computer Vision, IEEE, Santiago, Chile, pp. 1440–1448, 2015. DOI: https://doi.org/10.1109/ICCV.2015.169.
S. Q. Ren, K. M. He, R. Girshick, J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, MIT Press, Montreal, Canada, pp. 91–99, 2015.
J. Redmon, S. Divvala, R. Girshick, A. Farhadi. You only look once: Unified, real-time object detection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 779–788, 2016. DOI: https://doi.org/10.1109/CVPR.2016.91.
K. Simonyan, A. Zisserman. Very deep convolutional net-works for large-scale image recognition, Available: https://arxiv.org/abs/1409.1556, April 2015.
T. Y. Lin, P. Goyal, R. Girshick, K. M. He, P. Dollar. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, published on-line. DOI: https://doi.org/10.1109/TPAMI.2018.2858826.
L. Pishchulin, A. Jain, M. Andriluka, T. Thormahlen, B. Schiele. Articulated people detection and pose estimation: Reshaping the future. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Providence, USA, pp. 3178–3185, 2012. DOI: https://doi.org/10.1109/CV-PR.2012.6248052.
G. Gkioxari, B. Hariharan, R. Girshick, J. Malik. Using k-poselets for detecting people and localizing their keypoints. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 3582–3589, 2014. DOI: https://doi.org/10.1109/CVPR.2014.458.
L. Pishchulin, E. Insafutdinov, S. Y. Tang, B. Andres, M. Andriluka, P. Gehler, B. Schiele. DeepCut: Joint subset partition and labeling for multi person pose estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp.4929–1937, 2016. DOI: https://doi.org/10.1109/CVPR.2016.533.
E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, B. Schiele. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 34–50, 2016. DOI: https://doi.org/10.1007/978-3-319-46466-43.
S. Yang, P. Luo, C. C. Loy, X. O. Tang. WIDER FACE: A face detection benchmark. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 5525–5533, 2016. DOI: https://doi.org/10.1109/CV-PR.2016.596.
Z. Liu, P. Luo, X. G. Wang, X. O. Tang. Deep learning face attributes in the wild. In Proceedings of IEEE Inter-national Conference on Computer Vision, IEEE, Santiago, Chile, pp. 3730–3738, 2015. DOI: https://doi.org/10.1109/ICCV.2015.425.
Acknowledgements
This work was supported by the Joint Funds of National Natural Science Foundation of China (Nos. U1813215 and 2018YFB1307101), National Natural Science Foundation of China (Nos. 61603213, 61773239, 61973187, 61973192 and 91748115), Shandong Provincial Natural Science Foundation, China (No.ZR2017MF014), Jinan Technology project (No. 20150219) and Taishan Scholars Programme of Shandong Province.
Author information
Authors and Affiliations
Corresponding author
Additional information
Hao Wu received the M.Sc. and Ph.D. degrees in control theory and control engineering from Shandong University, China in 1997 and 2011, respectively. Currently, she is an associate professor at Shandong University, China. Her research mainly focuses on the navigation of service robot
Her research interests include robotics, control theory and automatic control system.
Zhao-Wei Chen received the B.Sc. degree in automation from Shandong University, China in 2017. He is currently a master student at Shandong University, China.
His research interests include deep learning and computer vision.
Guo-Hui Tian received the M.Sc. degree in industry automation from Shandong University, China in 1993, and received the Ph. D. degree in automatic control theory and application from Northeast University, China in 1997. He has a postdoctoral research at Engineering Department of Tokyo University, Japan from 2003 to 2005. Currently, he is a professor and a doctoral tutor at Shandong University, China. His research mainly focuses on service robot and smart space.
His research interests include robotics, feedback control systems, and control theory.
Qing Ma received the M.Sc. and Ph.D. degrees in control theory and control engineering from Shandong University, China in 2002 and 2012, respectively. Currently, he is a lecturer at Shandong University, China.
His research interests include machine learning and control theory.
Meng-Lin Jiao received the B.Sc. degree in automation from Shandong University, China in 2018. He is a master student in control science and engineering at Shandong University, China.
His research interests include machine learning and navigation of service robot.
Rights and permissions
About this article
Cite this article
Wu, H., Chen, ZW., Tian, GH. et al. Item Ownership Relationship Semantic Learning Strategy for Personalized Service Robot. Int. J. Autom. Comput. 17, 390–402 (2020). https://doi.org/10.1007/s11633-019-1206-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11633-019-1206-7