research-article

How do engineers perceive difficulties in engineering of machine-learning systems?: questionnaire survey

Authors:

Fuyuki Ishikawa,

Nobukazu YoshiokaAuthors Info & Claims

CESSER-IP '19: Proceedings of the Joint 7th International Workshop on Conducting Empirical Studies in Industry and 6th International Workshop on Software Engineering Research and Industrial Practice

Pages 2 - 9

https://doi.org/10.1109/CESSER-IP.2019.00009

Published: 27 May 2019 Publication History

Abstract

There is increasing interest in machine learning (ML) techniques and their applications in recent years. Although there has been intensive support by frameworks and libraries for the implementation of ML-based systems, investigation into engineering disciplines and methods is still at the early phase. The most pressing issue in this field is identifying the essential challenges for the software engineering research community as engineering of ML-based systems requires novel approaches due to the essentially different nature of ML-based systems. In this paper, we analyze the results of a questionnaire administered to 278 people who have worked on ML-based systems in practice, clarify the essential difficulties and their causes as perceived by practitioners, and suggest potential research directions.

References

[1]

D. Gunning, "Explainable artificial intelligence (XAI)," IJCAI 2016 Workshop on Deep Learning for Artificial Intelligence (DLAI), July 2016.

[2]

I. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," in International Conference on Learning Representations (ICLR), May 2015.

[3]

D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, and M. Young, "Machine learning: The high interest credit card of technical debt," in NIPSS 2014 Workshop on Software Engineering for Machine Learning (SE4ML), December 2014.

[4]

M. Zinkevich, "Rules for reliable machine learning: Best practices for ML engineering," NIPS 2016 Workshop on Reliable Machine Learning in the Wild, December 2017.

[5]

E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley, "What's your ML test score? a rubric for ML production systems," NIPS 2016 Workshop on Reliable Machine Learning in the Wild, December 2017.

[6]

K. Pei, Y. Cao, J. Yang, and S. Jana, "Deepxplore: Automated whitebox testing of deep learning systems," in The 26th Symposium on Operating Systems Principles (SOSP 2017), October 2017, pp. 1--18.

Digital Library

[7]

Y. Tian, K. Pei, S. Jana, and B. Ray, "DeepTest: automated testing of deep-neural-network-driven autonomous cars," in The 40th International Conference on Software Engineering (ICSE 2018), May 2018, pp. 303--314.

Digital Library

[8]

X. Huang, M. Kwiatkowska, S. Wang, and M. Wu, "Safety verification of deep neural networks," in The 29th International Conference on Computer Aided Verification (CAV 2017), July 2017, pp. 3--29.

[9]

M. Mirman, T. Gehr, and M. Vechev, "Differentiable abstract interpretation for provably robust neural networks," in The 35th International Conference on Machine Learning (ICML 2018), vol. 80, July 2018, pp. 3578--3586.

[10]

T. Dreossi, A. Donzé, and S. A. Seshia, "Compositional falsification of cyber-physical systems with machine learning components," in The 9th NASA Formal Methods Symposium (NFM 2017), May 2017, pp. 357--372.

[11]

A. Dwarakanath, M. Ahuja, S. Sikand, R. M. Rao, R. P. J. C. Bose, N. Dubash, and S. Podder, "Identifying implementation bugs in machine learning based image classifiers using metamorphic testing," in The 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2018), July 2018, pp. 118--120.

Digital Library

[12]

M. T. Ribeiro, S. Singh, and C. Guestrin, ""why should i trust you?": Explaining the predictions of any classifier," in The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2016), August 2016, pp. 1135--1144.

Digital Library

[13]

N. Polyzotis, S. Roy, S. E. Whang, and M. Zinkevich, "Data management challenges in production machine learning," in The 2017 ACM International Conference on Management of Data (SIGMOD 2017), May 2017, pp. 1723--1726.

Digital Library

[14]

Q. Liu, P. Li, W. Zhao, W. Cai, S. Yu, and V. C. M. Leung, "A survey on security threats and defensive techniques of machine learning: A data driven view," IEEE Access, vol. 6, pp. 12 103--12 117, February 2018.

[15]

S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cortés, "A survey on metamorphic testing," IEEE Transactions on Software Engineering, vol. 42, no. 9, pp. 805--824, September 2016.

[16]

E. J. Weyuker, "On testing non-testable programs," The Computer Journal, vol. 25, no. 4, pp. 465--470, November 1982.

Cited By

Arteaga Garcia ENicolaci Pimentel JFeng ZGerosa MSteinmacher ISarma ARoychoudhury APaiva AAbreu RStorey M(2024)How to Support ML End-User Programmers through a Conversational AgentProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3608130(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3608130
Alsamman ASchmitz AWimmer M(2023)Towards an Organically Growing Hate Speech Dataset in Hate Speech Detection Systems in a Smart Mobility ApplicationProceedings of the 24th Annual International Conference on Digital Government Research10.1145/3598469.3598473(36-43)Online publication date: 11-Jul-2023
https://dl.acm.org/doi/10.1145/3598469.3598473
Andersen JMaalej W(2023)Design Patterns for Machine Learning-Based Systems With Humans in the LoopIEEE Software10.1109/MS.2023.334025641:4(151-159)Online publication date: 8-Dec-2023
https://dl.acm.org/doi/10.1109/MS.2023.3340256
Show More Cited By

Recommendations

Machine Learning and Software Engineering
ICTAI '02: Proceedings of the 14th IEEE International Conference on Tools with Artificial Intelligence

Machine learning deals with the issue of how to build programs that improve their performance at some task through experience. Machine learning algorithms have proven to be of great practical value in a variety of application domains. Not surprisingly, ...
Automating Software Engineering with Machine Learning
ISEC '22: Proceedings of the 15th Innovations in Software Engineering Conference

Software plays a crucial role in our everyday lives. The scarcity of skilled software engineers has become a bottleneck in delivering better software at scale. Can we automate software engineering to help improve developer productivity and software ...
Machine Learning and Software Engineering

Machine learning deals with the issue of how to build programs that improve their performance at some task through experience. Machine learning algorithms have proven to be of great practical value in a variety of application domains. They are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CESSER-IP '19: Proceedings of the Joint 7th International Workshop on Conducting Empirical Studies in Industry and 6th International Workshop on Software Engineering Research and Industrial Practice

May 2019

57 pages

Conference Chairs:
Marcus Ciolkowski
QAware GmbH, Germany
,
Dusica Marijan
Simula Research Laboratory, Norway
,
Matthias Galster
University of Canterbury, New Zealand
,
Weiyi Shang
Concordia University, Canada
,
Andreas Jedlitschka
Fraunhofer IESE, Germany
,
Rakesh Shukla
Indrashil University, India
,
Kanchana Padmanabhan
Rubikloud Technologies, Canada

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

IEEE Press

Publication History

Published: 27 May 2019

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '19

Sponsor:

SIGSOFT
IEEE-CS

ICSE '19: 41st International Conference on Software Engineering

May 27, 2019

Quebec, Montreal, Canada

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
207
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Arteaga Garcia ENicolaci Pimentel JFeng ZGerosa MSteinmacher ISarma ARoychoudhury APaiva AAbreu RStorey M(2024)How to Support ML End-User Programmers through a Conversational AgentProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3608130(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3608130
Alsamman ASchmitz AWimmer M(2023)Towards an Organically Growing Hate Speech Dataset in Hate Speech Detection Systems in a Smart Mobility ApplicationProceedings of the 24th Annual International Conference on Digital Government Research10.1145/3598469.3598473(36-43)Online publication date: 11-Jul-2023
https://dl.acm.org/doi/10.1145/3598469.3598473
Andersen JMaalej W(2023)Design Patterns for Machine Learning-Based Systems With Humans in the LoopIEEE Software10.1109/MS.2023.334025641:4(151-159)Online publication date: 8-Dec-2023
https://dl.acm.org/doi/10.1109/MS.2023.3340256
Maalej WPham YChazette L(2023)Tailoring Requirements Engineering for Responsible AIComputer10.1109/MC.2023.324318256:4(18-27)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/MC.2023.3243182
Martínez-Fernández SBogner JFranch XOriol MSiebert JTrendowicz AVollmer AWagner S(2022)Software Engineering for AI-Based Systems: A SurveyACM Transactions on Software Engineering and Methodology10.1145/348704331:2(1-59)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1145/3487043
Anisetti MArdagna CDamiani EPanero PChbeir RManolopoulos YDamiani EBenslimane DBellatreche LMorzy T(2020)A Methodology for Non-Functional Property Evaluation of Machine Learning ModelsProceedings of the 12th International Conference on Management of Digital EcoSystems10.1145/3415958.3433101(38-45)Online publication date: 2-Nov-2020
https://dl.acm.org/doi/10.1145/3415958.3433101
Serban Avan der Blom KHoos HVisser J(2020)Adoption and Effects of Software Engineering Best Practices in Machine LearningProceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3382494.3410681(1-12)Online publication date: 5-Oct-2020
https://dl.acm.org/doi/10.1145/3382494.3410681
Marijan DShang WShukla R(2020)Implications of Resurgence in Artificial Intelligence for Research Collaborations in Software EngineeringACM SIGSOFT Software Engineering Notes10.1145/3356773.335681344:3(68-70)Online publication date: 22-Oct-2020
https://dl.acm.org/doi/10.1145/3356773.3356813

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten