Natural Language-Based Automatic Programming for Industrial Robots
Abstract
Cyber-Physical System (CPS), which is a part of Industry 4.0, suggests that the physical systems such as robots can be controlled by automation systems to minimize human workload. With the rise of automatic programming systems, programmers are no longer required to have a thorough understanding of the code. As a result of the interaction between the robot and the human, automatic programming systems generate a robot program. Many processing and recognition technologies for human-computer interaction interface are required to make the robot realize a more natural interaction. It is necessary to consider disruptive technologies in order to provide innovation and enable us to change the way we program. In this paper, we proposed approach to do automatic programming for industrial robots with natural language. To begin, we use a multi-attention mechanism to measure the matching probability of natural language instructions to objects in the environment. Then, using a modular programming method, we generate code for robots and combine the prediction results. We extend the existing dataset for evaluation to make it more suitable for describing the actual manufacturing environment, taking into account position, attribute, and constraints. The experimental results show that the model in this paper has a 20% higher recognition rate than other existing methods for accurately locating the object position, and the similarity between code written by experienced engineers and code generated by our more reached 80%.
References
[1]
Ferreira P, Doltsinis S, and Lohse N Symbiotic assembly systems - a new paradigm Procedia CIRP 2014 17 26-31
[2]
Pramanik, P.K.D., Mukherjee, B., Pal, S., et al. Ubiquitous manufacturing in the age of industry 4.0: a state-of-the-art primer. In: A Roadmap to Industry 4.0: Smart Production, Sharp Business and Sustainable Development, pp. 73–112. Springer, Cham (2020)
[3]
Chang V, Ramachandran M, and Muñoz VM Modern Industrial IoT, Big Data and Supply Chain: Proceedings of the IIoTBDSC 2020 2021
[4]
Krüger J, Lien TK, and Verl A Cooperation of human and machines in assembly lines[J] CIRP Ann. 2009 58 2 628-646
[5]
Pan Z, Polden J, Larkin N, et al. Recent Progress on Programming Methods for Industrial Robots[C] 2010 VDE
[6]
IFR presents World Robotics Report (2020). https://ifr.org/ifr-press-releases/news/record-2.7-million-robots-work-in-factories-around-the-globe
[7]
Pires JN, Nilsson K, and Petersen HG From the guest editors-industrial robotics applications and industry-academia cooperation in Europe[J] IEEE Robot. Autom. Mag. 2005 12 3 5-6
[8]
Biggs G, Macdonald B.: A survey of robot programming systems. Proc. Austral. Conf. Robot. Autom. 1 (2003)
[9]
Ni P, Li Y, Li G, et al. A hybrid Siamese neural network for natural language inference in cyber-physical systems[J] ACM Trans. Internet Technol. 2021 21 2 1-25
[10]
Natural language communication with robots. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2016)
[11]
Shridhar, M., Hsu, D.: Interactive visual grounding of referring expressions for human-robot interaction[J]. arXiv preprint arXiv:1806.03831 (2018)
[12]
Jia, Y., She, L., Cheng, Y., et al.: Program robots manufacturing tasks by natural language instructions. In: 2016 IEEE International Conference on Automation Science and Engineering (CASE). IEEE (2016)
[13]
Jia, Y., Xi, N., Chai, J.Y., et al.: Perceptive feedback for natural language control of robotic operations. In: 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2014)
[14]
Impacts and investigations of disruptive technologies for Industry 4.0. Technol. Forecast. Soc. Change. 174, 121232 (2021)
[15]
Presenting cloud business performance for manufacturing organizations. Inf. Syst. Front., 22(1), 59–75 (2020)
[16]
Freund E and Luedemann-Ravit B A System to automate the Generation of Program Variants for Industrial Robot Applications[C] 2002 IEEE
[17]
Hopler R and Otter M A Versatile C++ Toolbox for Model Based, Real Time Control Systems of Robotic Manipulators[C] 2001 IEEE
[18]
Blank D, Kumar D, Meeden L, et al. The pyro toolkit for AI and robotics[J] AI Mag. 2006 27 1 39
[19]
Ferrari, M., Ferrari, G.: Building robots with lego mindstorms. Elsevier (2001)
[20]
Bischoff, R., Kazi, A., Seyfarth, M.: The MORPHA style guide for icon-based programming[C]. IEEE (2002)
[21]
Maloney J, Nagle M, and Mönig JGP A General Purpose Blocks-Based Language[C] 2017
[22]
Fuhlbrigge, T.A., Rossano G., Zhang H., et al.: Method and apparatus for developing a metadata-infused software program for controlling a robot. U.S. Patent No. 7,643,907 (2010)
[23]
Argall BD, Chernova S, Veloso M, et al. A survey of robot learning from demonstration[J] Robot. Auton. Syst. 2009 57 5 469-483
[24]
Breazeal C and Scassellati B Robots that imitate humans[J] Trends Cogn. Sci. 2002 6 11 481-487
[25]
Billard, A., Calinon, S., Dillmann, R., et al.: Survey: robot programming by demonstration[J]. Handb. Robot. 59(BOOK_CHAP), (2008)
[26]
Bisk Y, Shih KJ, Choi Y, et al. Learning Interpretable Spatial Operations in a Rich 3d Blocks World[C] 2018
[27]
Paul R, Arkin J, Roy N, et al. Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators. In: Robotics: Science and Systems XII (2016)
[28]
Luong, M., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation[J]. arXiv preprint arXiv:1508.04025. (2015)
[29]
Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Playing atari with deep reinforcement learning[J]. arXiv preprint arXiv:1312.5602. (2013)
[30]
Williams RJ Simple statistical gradient-following algorithms for connectionist reinforcement learning[J] Mach. Learn. 1992 8 3–4 229-256
[31]
Reinforcement learning for mapping instructions to actions. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. (2009)
[32]
Misra, D., Langford, J., Artzi, Y. Mapping instructions and visual observations to actions with reinforcement learning[J]. arXiv preprint arXiv:1704.08795. (2017)
[33]
Environment-driven lexicon induction for high-level instructions. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, vol. 1: Long Papers. (2015)
[34]
Johnson J, Hariharan B, Van Der Maaten L, et al. Inferring and executing programs for visual reasoning. In: Proceedings of the IEEE international conference on computer vision. (2017)
[35]
Learning to interpret natural language commands through human-robot dialog. In: Twenty-Fourth International Joint Conference on Artificial Intelligence. (2015)
[36]
Thomason J, Sinapov J, Svetlik M, et al. Learning Multi-Modal Grounded Linguistic Semantics by Playing" I Spy".[C] 2016
[37]
Thomason J, Padmakumar A, Sinapov J, et al. Opportunistic active learning for grounding natural language descriptions. In: Conference on Robot Learning. PMLR (2017)
[38]
Weintrop D, Afzal A, Salac J, et al. Evaluating CoBlox: a Comparative Study of Robotics Programming Environments for Adult Novices[C] 2018
[39]
Weintrop D, Shepherd DC, Francis P, et al. Blockly goes to Work: Block-Based Programming for Industrial Robots[C] 2017 IEEE
[40]
Wu C, Liu J, Wang X, et al. Object-difference attention: A simple relational attention for visual question answering. In: Proceedings of the 26th ACM international conference on Multimedia. (2018)
[41]
Wu C, Liu J, Wang X, et al. Differential Networks for Visual Question Answering[C] 2019
[42]
Lee, C., Gottschlich, J., Roth, D.: Toward Code Generation: A Survey and Lessons from Semantic Parsing. Arxiv (2021)
[43]
Wang L, Blunsom P, Grefenstette E, Hermann KM, Kočiský T, Wang F, and Senior A Latent Predictor Networks for Code Generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 2016 Berlin, Germany Association for Computational Linguistics 599-609
[44]
Yin, Pengcheng, Neubig, Graham: TRANX: A Transitionbased Neural Abstract Syntax Parser for Semantic Parsing and Code Generation. In EMNLP. (2018)
[45]
Quirk, Chris, Mooney, Raymond, Galley, Michel: "Language to code: Learning semantic parsers for if-this-then-that recipes." Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). (2015)
[46]
Sun, Zeyu, et al.: A grammar-based structural cnn decoder for code generation. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33. no. 01. (2019)
[47]
Sun, Zeyu, et al.: "Treegen: A tree-based transformer architecture for code generation." Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34. no. 05. (2020)
[48]
Schuster M and Paliwal KK Bidirectional recurrent neural networks[J] IEEE Trans. Signal Process. 1997 45 11 2673-2681
[49]
Faster r-cnn: Towards real-time object detection with region proposal networks. Adv. Neural Inform. Process. Sys. 28 (2015)
[50]
Kingma, D.P., Ba, J. Adam: A method for stochastic optimization[J]. arXiv preprint arXiv:1412.6980. (2014)
[51]
Robertson SE and Spärck JK Simple, Proven Approaches to Text Retrieval[R] 1994 University of Cambridge, Computer Laboratory
[52]
Ducasse S, Rieger M, and Demeyer S A Language Independent Approach for Detecting Duplicated Code[C] 1999 IEEE
[53]
Kamiya T, Kusumoto S, and Inoue K CCFinder: a multilinguistic token-based code clone detection system for large scale source code[J] IEEE Trans. Softw. Eng. 2002 28 7 654-670
[54]
Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J] Proc. IEEE 1998 86 11 2278-2324
[55]
Tan H, Bansal M. Source-target inference models for spatial instruction understanding. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32. no. 1. (2018)
Recommendations
Natural language in introductory programming: an experimental study
ITiCSE '11: Proceedings of the 16th annual joint conference on Innovation and technology in computer science educationAlthough characterized as being "high level", classical programming languages such as Pascal and C have a grammar that is very different from natural language. In this research field, two main streams are noteworthy, one of them is characterized by an ...
Programming language, natural language? Supporting the diverse computational activities of novice programmers
Given the current focus on teaching computational concepts to all from an early age, combined with the growing trend to empower end users to become producers of technology rather than mere consumers, we consider the issue of computational notation. ...
Evaluating Robot Behavior in Response to Natural Language
HRI '18: Companion of the 2018 ACM/IEEE International Conference on Human-Robot InteractionHuman-robot teaming can be improved if a robot»s actions meet human users» expectations. The goal of this research is to determine what variations of robot actions in response to natural language match human judges» expectations in a series of tasks. We ...
Comments
Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Nature B.V. 2022.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 01 September 2022
Accepted: 27 June 2022
Received: 23 December 2021
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in