research-article

Building Domain-Specific Machine Learning Workflows: A Conceptual Framework for the State of the Practice

Authors:

Bentley James Oakes,

Michalis Famelis,

Houari SahraouiAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology, Volume 33, Issue 4

Article No.: 91, Pages 1 - 50

https://doi.org/10.1145/3638243

Published: 17 April 2024 Publication History

Abstract

Domain experts are increasingly employing machine learning to solve their domain-specific problems. This article presents to software engineering researchers the six key challenges that a domain expert faces in addressing their problem with a computational workflow, and the underlying executable implementation. These challenges arise out of our conceptual framework which presents the “route” of transformations that a domain expert may choose to take while developing their solution.

To ground our conceptual framework in the state of the practice, this article discusses a selection of available textual and graphical workflow systems and their support for the transformations described in our framework. Example studies from the literature in various domains are also examined to highlight the tools used by the domain experts as well as a classification of the domain specificity and machine learning usage of their problem, workflow, and implementation.

The state of the practice informs our discussion of the six key challenges, where we identify which challenges and transformations are not sufficiently addressed by available tools. We also suggest possible research directions for software engineering researchers to increase the automation of these tools and disseminate best-practice techniques between software engineering and various scientific domains.

References

[1]

Aderibigbe Israel Adekitan, Jeremiah Abolade, and Olamilekan Shobayo. 2019. Data mining approach for predicting the daily Internet data traffic of a smart university. Journal of Big Data 6, 1 (2019), 1–23.

[2]

Azza E. Ahmed, Joshua M. Allen, Tajesvi Bhat, Prakruthi Burra, Christina E. Fliege, Steven N. Hart, Jacob R. Heldenbrand, Matthew E. Hudson, Dave Deandre Istanto, Michael T. Kalmbach, et al. 2021. Design considerations for workflow management systems use in production genomics research and the clinic. Scientific Reports 11, 1 (2021), 1–18.

[3]

Lissette Almonte, Iván Cantador, Esther Guerra, and Juan de Lara. 2020. Towards automating the construction of recommender systems for low-code development platforms. In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings. 1–10.

Digital Library

[4]

Ethem Alpaydin. 2020. Introduction to Machine Learning. MIT Press, Cambridge, MA.

[5]

Peter Amstutz, Maxim Mikheev, Michael R. Crusoe, Nebojša Tijanić, Samuel Lampa, et al. 2021. Existing workflow systems. Common Workflow Language Wiki, GitHub. Retrieved January 6, 2022 from https://github.com/common-workflow-language/common-workflow-language/wiki

[6]

Sanjay Aneja, Enoch Chang, and Antonio Omuro. 2019. Applications of artificial intelligence in neuro-oncology. Current Opinion in Neurology 32, 6 (2019), 850–856.

[7]

Syeeda Nilofer Banoo. 2020. Flow-Based Programming for Machine Learning. Master’s Thesis. Technical University of Munich.

[8]

Kyle D. Bemis, April Harry, Livia S. Eberlin, Christina Ferreira, Stephanie M. van de Ven, Parag Mallick, Mark Stolowitz, and Olga Vitek. 2015. Cardinal: An R package for statistical analysis of mass spectrometry-based imaging experiments. Bioinformatics 31, 14 (2015), 2418–2420.

[9]

Meriem Ben Chaaben. 2023. Few-Shot Prompt Learning for Automating Model Completion. Master’s Thesis. Université de Montréal.

[10]

Oussama Ben Sghaier, Jean-Sebastien Boudrias, and Houari Sahraoui. 2023. Toward optimal psychological functioning in AI-driven software engineering tasks: The SEWELL-CARE Assessment Framework. arXiv:2311.07410 (2023).

[11]

Michael R. Berthold, Nicolas Cebron, Fabian Dill, Thomas R. Gabriel, Tobias Kötter, Thorsten Meinl, Peter Ohl, Christoph Sieb, Kilian Thiel, and Bernd Wiswedel. 2008. KNIME: The Konstanz information miner. In Data Analysis, Machine Learning and Applications, Christine Preisach, Hans Burkhardt, Lars Schmidt-Thieme, and Reinhold Decker (Eds.). Springer, Berlin, Germany, 319–326.

[12]

Matthias Biehl, Jad El-Khoury, Frédéric Loiret, and Martin Törngren. 2014. On the modeling and generation of service-oriented tool chains. Software & Systems Modeling 13 (2014), 461–480.

Digital Library

[13]

Sumon Biswas, Mohammad Wardat, and Hridesh Rajan. 2022. The art and practice of data science pipelines: A comprehensive study of data science pipelines in theory, in-the-small, and in-the-large. In Proceedings of the 44th International Conference on Software Engineering. 2091–2103.

Digital Library

[14]

Alexander C. Bock and Ulrich Frank. 2021. In search of the essence of low-code: An exploratory study of seven development platforms. In Proceedings of the 2021 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C ’21). IEEE, Los Alamitos, CA, 57–66.

[15]

Andrea Borghesi, Federico Baldo, and Michela Milano. 2020. Improving deep learning models via constraint-based domain knowledge: A brief survey. arXiv preprint arXiv:2005.10691 (2020).

[16]

John Brooke. 1996. SUS: A “quick and dirty” usability scale. Usability Evaluation in Industry 189, 3 (1996), 189–194.

[17]

Adrian Bumann and Robin Teigland. 2021. The challenges of knowledge combination in ML-based crowdsourcing—The ODF Killer Shrimp Challenge using ML and Kaggle. In Proceedings of the 54th Hawaii International Conference on System Sciences. 4930.

[18]

Jordi Cabot. 2020. Positioning of the low-code movement within the field of model-driven engineering. In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings. 1–3.

Digital Library

[19]

Yuzhe Cai, Shaoguang Mao, Wenshan Wu, Zehua Wang, Yaobo Liang, Tao Ge, Chenfei Wu, Wang You, Ting Song, Yan Xia, et al. 2023. Low-code LLM: Visual programming over LLMs. arXiv preprint arXiv:2304.08103 (2023).

[20]

Lucas A. M. C. Carvalho, Regina Wang, Yolanda Gil, and Daniel Garijo. 2017. NiW: Converting notebooks into workflows to capture dataflow and provenance. In Proceedings of the K-CAP Workshops.

[21]

Marina Celestine, Nachiket A. Nadkarni, Clément M. Garin, Salma Bougacha, and Marc Dhenain. 2020. Sammba-MRI: A library for processing SmAll-MaMmal BrAin MRI data in Python. Frontiers in Neuroinformatics 14 (2020), 24.

[22]

Moharram Challenger, Ken Vanherpen, Joachim Denil, and Hans Vangheluwe. 2020. FTG+PM: Describing engineering processes in multi-paradigm modelling. In Foundations of Multi-Paradigm Modelling for Cyber-Physical Systems. Springer, Cham, Switzerland, 259–271.

[23]

Michele Chinosi and Alberto Trombetta. 2012. BPMN: An introduction to the standard. Computer Standards & Interfaces 34, 1 (2012), 124–134.

Digital Library

[24]

Benoit Combemale, Jorg Kienzle, Gunter Mussbacher, Hyacinth Ali, Daniel Amyot, Mojtaba Bagherzadeh, Edouard Batot, Nelly Bencomo, Benjamin Benni, Jean-Michel Bruel, et al. 2020. A hitchhiker’s guide to model-driven engineering for data-centric systems. IEEE Software 38, 4 (2020), 71–84.

Digital Library

[25]

OpenJS Foundation and Contributors. n.d. Node-RED. Retrieved December 26, 2023 from https://nodered.org

[26]

Michael R. Crusoe, Sanne Abeln, Alexandru Iosup, Peter Amstutz, John Chilton, Nebojša Tijanić, Hervé Ménager, Stian Soiland-Reyes, Bogdan Gavrilovic, and Carole Goble. 2021. Methods included: Standardizing computational reuse and portability with the common workflow language. arXiv preprint arXiv:2105.07028 (2021).

[27]

Rafael Ferreira da Silva, Henri Casanova, Kyle Chard, Ilkay Altintas, Rosa M. Badia, Bartosz Balis, Tainã Coleman, Frederik Coppens, Frank Di Natale, Bjoern Enders, et al. 2021. A community roadmap for scientific workflows research and development. In Proceedings of the 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS ’21). IEEE, Los Alamitos, CA, 81–90.

[28]

Giordano d’Aloisio, Antinisca Di Marco, and Giovanni Stilo. 2022. Modeling quality and machine learning pipelines through extended feature models. arXiv preprint arXiv:2207.07528 (2022).

[29]

István Dávid, Hans Vangheluwe, and Yentl Van Tendeloo. 2018. Translating engineering workflow models to DEVS for performance evaluation. In Proceedings of the 2018 Winter Simulation Conference (WSC ’18). IEEE, Los Alamitos, CA, 616–627.

Digital Library

[30]

Ewa Deelman, Anirban Mandal, Ming Jiang, and Rizos Sakellariou. 2019. The role of machine learning in scientific workflows. International Journal of High Performance Computing Applications 33, 6 (2019), 1128–1139.

Digital Library

[31]

Janez Demšar, Tomaž Curk, Aleš Erjavec, Črt Gorup, Tomaž Hočevar, Mitar Milutinovič, Martin Možina, Matija Polajnar, Marko Toplak, Anže Starič, et al. 2013. Orange: Data mining toolbox in Python. Journal of Machine Learning Research 14, 1 (2013), 2349–2353.

Digital Library

[32]

Janez Demšar and Blaz Zupan. 2005. From Experimental Machine Learning to Interactive Data Mining. White Paper. Faculty of Computer and Information Science, University of Ljubljana. https://www.ailab.si/orange

[33]

Claudio Di Sipio, Juri Di Rocco, Davide Di Ruscio, and Phuong T. Nguyen. 2023. MORGAN: A modeling recommender system based on graph kernel. Software and Systems Modeling 22 (2023), 1427–1449.

[34]

Paolo Di Tommaso, Maria Chatzou, Evan W. Floden, Pablo Prieto Barja, Emilio Palumbo, and Cedric Notredame. 2017. Nextflow enables reproducible computational workflows. Nature Biotechnology 35, 4 (2017), 316–319.

[35]

Juan Sebastian Beleno Diaz and Claudia Bauzer Medeiros. 2017. WorkflowHunt: Combining keyword and semantic search in scientific workflow repositories. In Proceedings of the 2017 IEEE 13th International Conference on e-Science (e-Science ’17). IEEE, Los Alamitos, CA, 138–147.

[36]

William Digan, Aurélie Névéol, Antoine Neuraz, Maxime Wack, David Baudoin, Anita Burgun, and Bastien Rance. 2021. Can reproducibility be improved in clinical natural language processing? A study of 7 clinical NLP suites. Journal of the American Medical Informatics Association 28, 3 (2021), 504–515.

[37]

Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. 2020. Benchmarking materials property prediction methods: The Matbench test set and Automatminer reference algorithm. npj Computational Materials 6, 1 (2020), 1–10.

[38]

Vito D’Orazio, James Honaker, Raman Prasady, and Michael Shoemate. 2019. Modeling and forecasting armed conflict: AutoML with human-guided machine learning. In Proceedings of the 2019 IEEE International Conference on Big Data (Big Data ’19). IEEE, Los Alamitos, CA, 4714–4723.

[39]

Oscar Esteban, Rastko Ciric, Karolina Finc, Ross W. Blair, Christopher J. Markiewicz, Craig A. Moodie, James D. Kent, Mathias Goncalves, Elizabeth DuPre, Daniel E. P. Gomez, et al. 2020. Analysis of task-based functional MRI data preprocessed with fMRIPrep. Nature Protocols 15, 7 (2020), 2186–2202.

[40]

Oscar Esteban, Christopher J. Markiewicz, Ross W. Blair, Craig A. Moodie, A. Ilkay Isik, Asier Erramuzpe, James D. Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, et al. 2019. fMRIPrep: A robust preprocessing pipeline for functional MRI. Nature Methods 16, 1 (2019), 111–116.

[41]

Riley Evans, Samantha Frohlich, and Meng Wang. 2022. CircuitFlow: A domain specific language for dataflow programming. In Proceedings of the International Symposium on Practical Aspects of Declarative Languages. 79–98.

Digital Library

[42]

Philip A. Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso, and Sven Nahnsen. 2020. The nf-core framework for community-curated bioinformatics pipelines. Nature Biotechnology 38, 3 (2020), 276–278.

[43]

Javier Fabra, María José Ibáñez, Pedro Alvarez, and Joaquin Ezpeleta. 2018. Behavioral analysis of scientific workflows with semantic information. IEEE Access 6 (2018), 66030–66046.

[44]

Michalis Famelis and Marsha Chechik. 2019. Managing design-time uncertainty. Software & Systems Modeling 18, 2 (2019), 1249–1284.

Digital Library

[45]

Yuanrui Fan, Xin Xia, David Lo, and Ahmed E. Hassan. 2018. Chaff from the wheat: Characterizing and determining valid bug reports. IEEE Transactions on Software Engineering 46, 5 (2018), 495–525.

[46]

Melanie Christine Föll, Lennart Moritz, Thomas Wollmann, Maren Nicole Stillger, Niklas Vockert, Martin Werner, Peter Bronsert, Karl Rohr, Björn Andreas Grüning, and Oliver Schilling. 2019. Accessible and reproducible mass spectrometry imaging data analysis in Galaxy. GigaScience 8, 12 (2019), giz143.

[47]

Melanie Christine Föll, Veronika Volkmann, Kathrin Enderle-Ammour, Konrad Wilhelm, Dan Guo, Olga Vitek, Peter Götz Christian Bronsert, and Oliver Schilling. 2022. Moving translational mass spectrometry imaging towards transparent and reproducible data analyses: A case study of an urothelial cancer cohort analyzed in the Galaxy framework. Clinical Proteomics 19, 1 (2022), 8.

[48]

Grigori Fursin. 2021. Collective knowledge: Organizing research projects as a database of reusable components and portable workflows with common interfaces. Philosophical Transactions of the Royal Society A 379, 2197 (2021), 20200211.

[49]

Daniel Garijo, Yolanda Gil, and Oscar Corcho. 2017. Abstract, link, publish, exploit: An end to end framework for workflow sharing. Future Generation Computer Systems 75 (2017), 271–283.

[50]

Holger Giese, Tihamér Levendovszky, and Hans Vangheluwe. 2006. Summary of the workshop on multi-paradigm modeling: Concepts and tools. In Proceedings of the International Conference on Model Driven Engineering Languages and Systems. 252–262.

[51]

Yolanda Gil, James Honaker, Shikhar Gupta, Yibo Ma, Vito D’Orazio, Daniel Garijo, Shruti Gadewar, Qifan Yang, and Neda Jahanshad. 2019. Towards human-guided machine learning. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 614–624.

Digital Library

[52]

Yolanda Gil, Varun Ratnakar, and Christian Fritz. 2010. Assisting scientists with complex data analysis tasks through semantic workflows. In Proceedings of the 2010 AAAI Fall Symposium Series.

[53]

Yolanda Gil, Varun Ratnakar, Jihie Kim, Pedro Gonzalez-Calero, Paul Groth, Joshua Moody, and Ewa Deelman. 2010. Wings: Intelligent workflow-based design of computational experiments. IEEE Intelligent Systems 26, 1 (2010), 62–72.

Digital Library

[54]

Primož Godec, Matjaž Pančur, Nejc Ilenič, Andrej Čopar, Martin Stražar, Aleš Erjavec, Ajda Pretnar, Janez Demšar, Anže Starič, Marko Toplak, et al. 2019. Democratized image analytics by visual programming through integration of deep models and small-scale machine learning. Nature Communications 10, 1 (2019), 1–7.

[55]

Lucian José Gonçales, Kleinner Farias, and Bruno C. da Silva. 2021. Measuring the cognitive load of software developers: An extended systematic mapping study. Information and Software Technology 136 (2021), 106563.

Digital Library

[56]

Dmitry Gordeev and Philipp Singer. 2020. From Football Newbies to NFL (Data) Champions: A Winner’s Interview with The Zoo. (Jan. 2020). Retrieved December 26, 2023 from https://medium.com/kaggle-blog/from-football-newbies-to-nfl-data-champions-a-winners-interview-with-the-zoo-391793168714

[57]

Krzysztof Gorgolewski, Christopher D. Burns, Cindee Madison, Dav Clark, Yaroslav O. Halchenko, Michael L. Waskom, and Satrajit S. Ghosh. 2011. Nipype: A flexible, lightweight and extensible neuroimaging data processing framework in Python. Frontiers in Neuroinformatics 5 (2011), 13.

[58]

Qiang Gu, Anup Kumar, Simon Bray, Allison Creason, Alireza Khanteymoori, Vahid Jalili, Björn Grüning, and Jeremy Goecks. 2021. Galaxy-ML: An accessible, reproducible, and scalable machine learning toolkit for biomedicine. PLOS Computational Biology 17, 6 (2021), e1009014.

[59]

D. Hasterok, Matthew Gard, C. M. B. Bishop, and David Kelsey. 2019. Chemical identification of metamorphic protoliths using machine learning methods. Computers & Geosciences 132 (2019), 56–68.

[60]

Xin He, Kaiyong Zhao, and Xiaowen Chu. 2021. AutoML: A survey of the state-of-the-art. Knowledge-Based Systems 212 (2021), 106622.

[61]

Kexin Huang, Cao Xiao, Lucas M. Glass, Cathy W. Critchlow, Greg Gibson, and Jimeng Sun. 2021. Machine learning applications for therapeutic tasks with genomics data. Patterns 2, 10 (2021), 100328. DOI:

[62]

Felicien Ihirwe, Davide Di Ruscio, Silvia Mazzini, Pierluigi Pierini, and Alfonso Pierantonio. 2020. Low-code engineering for Internet of Things: A state of research. In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings. 1–8.

Digital Library

[63]

Peter Ivie and Douglas Thain. 2018. Reproducibility in scientific computing. ACM Computing Surveys 51, 3 (2018), 1–36.

Digital Library

[64]

Kevin Maik Jablonka, Daniele Ongari, Seyed Mohamad Moosavi, and Berend Smit. 2020. Big-data science in porous materials: Materials genomics and machine learning. Chemical Reviews 120, 16 (2020), 8066–8129.

[65]

Anubhav Jain, Shyue Ping Ong, Wei Chen, Bharat Medasani, Xiaohui Qu, Michael Kocher, Miriam Brafman, Guido Petretto, Gian-Marco Rignanese, Geoffroy Hautier, et al. 2015. FireWorks: A dynamic workflow system designed for high-throughput applications. Concurrency and Computation: Practice and Experience 27, 17 (2015), 5037–5059.

Digital Library

[66]

Vahid Jalili, Enis Afgan, Qiang Gu, Dave Clements, Daniel Blankenberg, Jeremy Goecks, James Taylor, and Anton Nekrutenko. 2020. The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2020 update. Nucleic Acids Research 48, W1 (2020), W395–W402.

[67]

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 2021. Highly accurate protein structure prediction with AlphaFold. Nature 596, 7873 (2021), 583–589.

[68]

Juha Kärnä, Juha-Pekka Tolvanen, and Steven Kelly. 2009. Evaluating the use of domain-specific modeling in practice. In Proceedings of the Object-Oriented Programming, Systems, Languages and Applications Workshop on Domain-Specific Modeling.

[69]

Vedran Kasalica and Anna-Lena Lamprecht. 2020. APE: A command-line tool and API for automated workflow composition. In Proceedings of the International Conference on Computational Science. 464–476.

Digital Library

[70]

Athanassios M. Kintsakis, Fotis E. Psomopoulos, Andreas L. Symeonidis, and Pericles A. Mitkas. 2017. Hermes: Seamless delivery of containerized bioinformatics workflows in hybrid cloud (HTC) environments. SoftwareX 6 (2017), 217–224.

[71]

Panu Kortelainen. 2021. Manage Your Workflows: A Classification Framework and Technology Review of Workflow Management Systems. Ph.D. Dissertation. Tampere University.

[72]

Philip Kortum, Claudia Ziegler Acemyan, and Frederick L. Oswald. 2021. Is it time to go positive? Assessing the positively worded System Usability Scale (SUS). Human Factors 63, 6 (2021), 987–998.

[73]

Maxat Kulmanov, Fatima Zohra Smaili, Xin Gao, and Robert Hoehndorf. 2021. Semantic similarity and machine learning with ontologies. Briefings in Bioinformatics 22, 4 (2021), 1–18.

[74]

Anup Kumar, Helena Rasche, Björn Grüning, and Rolf Backofen. 2021. Tool recommender system in Galaxy using deep learning. GigaScience 10, 1 (2021), giaa152.

[75]

Deborah Lafuente, Brenda Cohen, Guillermo Fiorini, Agustín Alejo García, Mauro Bringas, Ezequiel Morzan, and Diego Onna. 2021. A gentle introduction to machine learning for chemists: An undergraduate workshop using Python notebooks for visualization, data processing, analysis, and modeling. Journal of Chemical Education 98, 9 (2021), 2892–2898.

[76]

Samuel Lampa, Jonathan Alvarsson, and Ola Spjuth. 2016. Towards agile large-scale predictive modelling in drug discovery with flow-based programming design principles. Journal of Cheminformatics 8, 1 (2016), 1–12.

[77]

Samuel Lampa, Martin Dahlö, Jonathan Alvarsson, and Ola Spjuth. 2019. SciPipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines. GigaScience 8, 5 (2019), giz044.

[78]

Anna-Lena Lamprecht, Magnus Palmblad, Jon Ison, Veit Schwämmle, Mohammad Sadnan Al Manir, Ilkay Altintas, Christopher J. O. Baker, Ammar Ben Hadj Amor, Salvador Capella-Gutierrez, Paulos Charonyktakis, et al. 2021. Perspectives on automated composition of workflows in the life sciences. F1000Research 10 (2021), 897.

[79]

Bettina Laugwitz, Theo Held, and Martin Schrepp. 2008. Construction and evaluation of a user experience questionnaire. In HCI and Usability for Education and Work. Lecture Notes in Computer Science, Vol. 5298. Springer, 63–76.

[80]

Rodger Lea. n.d. Node-RED Programming Guide. Retrieved December 26, 2023 from http://noderedguide.com/

[81]

Doris Jung-Lin Lee and Stephen Macke. 2020. A human-in-the-loop perspective on AutoML: Milestones and the road ahead. IEEE Data Engineering Bulletin 2020 (2020), 1–12.

[82]

Kin Wai Lee and Renee Ka Yin Chin. 2020. The effectiveness of data augmentation for melanoma skin cancer prediction using convolutional neural networks. In Proceedings of the 2020 IEEE 2nd International Conference on Artificial Intelligence in Engineering and Technology (IICAIET ’20). IEEE, Los Alamitos, CA, 1–6.

[83]

Fabian Lehmann, David Frantz, Sören Becker, Ulf Leser, and Patrick Hostert. 2021. FORCE on Nextflow: Scalable analysis of earth observation data on commodity clusters. In Proceedings of the International Workshop on Complex Data Challenges in Earth Observation.

[84]

Jeremy Leipzig. 2017. A review of bioinformatic pipeline frameworks. Briefings in Bioinformatics 18, 3 (2017), 530–536.

[85]

Hugo López-Fernández, Osvaldo Graña-Castro, Alba Nogueira-Rodríguez, Miguel Reboiro-Jato, and Daniel Glez-Peña. 2021. Compi: A framework for portable and reproducible pipelines. PeerJ Computer Science 7 (2021), e593.

[86]

I. Scott MacKenzie. 1992. Fitts’ law as a research and design tool in human-computer interaction. Human-Computer Interaction 7, 1 (1992), 91–139.

Digital Library

[87]

Vivien Marx. 2020. When computational pipelines go ‘clank.’ Nature Methods 17, 7 (2020), 659–662.

[88]

Kiran Mathew, Joseph H. Montoya, Alireza Faghaninia, Shyam Dwarakanath, Muratahan Aykol, Hanmei Tang, Iek-heng Chu, Tess Smidt, Brandon Bocklund, Matthew Horton, et al. 2017. Atomate: A high-level interface to generate, execute, and analyze computational materials science workflows. Computational Materials Science 139 (2017), 140–152.

[89]

Russell P. McIver. 2015. A Knowledge-Based Approach to Scientific Workflow Composition. Ph.D. Dissertation. Cardiff University.

[90]

Arsenty D. Melnikov, Yuri P. Tsentalovich, and Vadim V. Yanshole. 2019. Deep learning for the precise peak detection in high-resolution LC–MS data. Analytical Chemistry 92, 1 (2019), 588–592.

[91]

Marçal Mora-Cantallops, Salvador Sánchez-Alonso, Elena García-Barriocanal, and Miguel-Angel Sicilia. 2021. Traceability for trustworthy AI: A review of models and tools. Big Data and Cognitive Computing 5, 2 (2021), 20.

[92]

J. Paul Morrison. 1994. Flow-based programming. In Proceedings of the 1st International Workshop on Software Engineering for Parallel and Distributed Systems. 25–29.

[93]

Marzieh Mousavian, Jianhua Chen, Zachary Traylor, and Steven Greening. 2021. Depression detection from sMRI and rs-fMRI images using machine learning. Journal of Intelligent Information Systems 57, 2 (2021), 395–418.

Digital Library

[94]

Gunter Mussbacher, Benoit Combemale, Silvia Abrahão, Nelly Bencomo, Loli Burgueño, Gregor Engels, Jörg Kienzle, Thomas Kühn, Sébastien Mosser, Houari Sahraoui, et al. 2020. Towards an assessment grid for intelligent modeling assistance. In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings. 1–10.

Digital Library

[95]

Gunter Mussbacher, Benoit Combemale, Jörg Kienzle, Silvia Abrahão, Hyacinth Ali, Nelly Bencomo, Márton Búr, Loli Burgueño, Gregor Engels, Pierre Jeanjean, et al. 2020. Opportunities in intelligent modeling assistance. Software and Systems Modeling 19, 5 (2020), 1045–1053.

Digital Library

[96]

Franz Nachbar, Wilhelm Stolz, Tanja Merkle, Armand B. Cognetta, Thomas Vogt, Michael Landthaler, Peter Bilek, Otto Braun-Falco, and Gerd Plewig. 1994. TheABCD rule of dermatoscopy: High prospective value in the diagnosis of doubtful melanocytic skin lesions. Journal of the American Academy of Dermatology 30, 4 (1994), 551–559.

[97]

Farid Nakhle and Antoine L. Harfouche. 2021. Ready, steady, Go AI: A practical tutorial on fundamentals of artificial intelligence and its applications in phenomics image analysis. Patterns 2, 9 (2021), 100323. DOI:

[98]

Soroosh Nalchigar. 2020. From Business Goals to Analytics and Machine Learning Solutions: A Conceptual Modeling Framework. Ph.D. Dissertation. University of Toronto, Canada.

[99]

Alba Nogueira-Rodríguez, Hugo López-Fernández, Osvaldo Graña-Castro, Miguel Reboiro-Jato, and Daniel Glez-Peña. 2020. Compi Hub: A public repository for sharing and discovering Compi pipelines. In Proceedings of the International Conference on Practical Applications of Computational Biology and Bioinformatics. 51–59.

[100]

Azita Nouri, Philip E. Davis, Pradeep Subedi, and Manish Parashar. 2021. Exploring the role of machine learning in scientific workflows: Opportunities and challenges. arXiv preprint arXiv:2110.13999 (2021).

[101]

Bentley James Oakes, Romain Franceschini, Simon Van Mierlo, and Hans Vangheluwe. 2019. The computational notebook paradigm for multi-paradigm modeling. In Proceedings of the 2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C ’19). 449–454. DOI:

Digital Library

[102]

Orange. n.d. Data Mining Fruitful & Fun Orange:Orange Website. Retrieved December 26, 2023 from https://orange.biolab.si/

[103]

Perrine Paul-Gilloteaux, Sébastien Tosi, Jean-Karim Hériché, Alban Gaignard, Hervé Ménager, Raphaël Marée, Volker Baecker, Anna Klemm, Matúš Kalaš, Chong Zhang, et al. 2021. Bioimage analysis workflows: Community resources to navigate through a complex ecosystem. F1000Research 10 (2021), 320.

[104]

Russell A. Poldrack, Krzysztof J. Gorgolewski, and Gael Varoquaux. 2018. Computational and informatics advances for reproducible data analysis in neuroimaging. arXiv preprint arXiv:1809.10024 (2018).

[105]

Luigi Quaranta, Fabio Calefato, and Filippo Lanubile. 2021. KGTorrent: A dataset of Python Jupyter notebooks from Kaggle. In Proceedings of the 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR ’21). IEEE, Los Alamitos, CA, 550–554.

[106]

Taylor Reiter, Phillip T. Brooks, Luiz Irber, Shannon E. K. Joslin, Charles M. Reid, Camille Scott, C. Titus Brown, and N. Tessa Pierce-Ward. 2021. Streamlining data-intensive biology with workflow systems. GigaScience 10, 1 (2021), giaa140.

[107]

Philipp Ruf, Manav Madan, Christoph Reich, and Djaffar Ould-Abdeslam. 2021. Demystifying MLOps and presenting a recipe for the selection of open-source tools. Applied Sciences 11, 19 (2021), 8861.

[108]

Adam Rule, Amanda Birmingham, Cristal Zuniga, Ilkay Altintas, Shih-Cheng Huang, Rob Knight, Niema Moshiri, Mai H. Nguyen, Sara Brin Rosenthal, Fernando Pérez, et al. 2019. Ten simple rules for writing and sharing computational analyses in Jupyter notebooks. PLOS Computational Biology 15, 7 (2019), e1007007.

[109]

Vinícius W. Salazar, João Vitor Ferreira Cavalcante, Daniel de Oliveira, Fabiano Thompson, and Marta Mattoso. 2021. BioProv—A provenance library for bioinformatics workflows. Journal of Open Source Software 6, 67 (2021), 3622.

[110]

Khodakaram Salimifard and Mike Wright. 2001. Petri net-based modelling of workflow systems: An overview. European Journal of Operational Research 134, 3 (2001), 664–676.

[111]

Aécio Santos, Sonia Castelo, Cristian Felix, Jorge Piazentin Ono, Bowen Yu, Sungsoo Ray Hong, Cláudio T. Silva, Enrico Bertini, and Juliana Freire. 2019. Visus: An interactive system for automatic machine learning model building and curation. In Proceedings of the Workshop on Human-in-the-Loop Data Analytics. 1–7.

Digital Library

[112]

Jacob Scharcanski and M. Emre Celebi. 2013. Computer Vision Techniques for the Diagnosis of Skin Cancer. Springer.

Digital Library

[113]

S. Sendall and W. Kozaczynski. 2003. Model transformation: The heart and soul of model-driven software development. IEEE Software 20, 5 (2003), 42–45. DOI:

Digital Library

[114]

Zohreh Sharafi, Yu Huang, Kevin Leach, and Westley Weimer. 2021. Toward an objective measure of developers’ cognitive activities. ACM Transactions on Software Engineering and Methodology 30, 3 (2021), 1–40.

Digital Library

[115]

Rina Singh, Jeffrey A. Graves, Valentine Anantharaj, and Sreenivas R. Sukumar. 2019. Evaluating scientific workflow engines for data and compute intensive discoveries. In Proceedings of the 2019 IEEE International Conference on Big Data (Big Data ’19). IEEE, Los Alamitos, CA, 4553–4560.

[116]

Saúl Solorio-Fernández, J. Ariel Carrasco-Ochoa, and José Fco. Martínez-Trinidad. 2020. A review of unsupervised feature selection methods. Artificial Intelligence Review 53, 2 (2020), 907–948.

Digital Library

[117]

Patricia Centeno Soto, Nour Ramzy, Felix Ocker, and Birgit Vogel-Heuser. 2021. An ontology-based approach for preprocessing in machine learning. In Proceedings of the 2021 IEEE 25th International Conference on Intelligent Engineering Systems (INES ’21). IEEE, Los Alamitos, CA, 133–138.

[118]

Christopher Sutton, Luca M. Ghiringhelli, Takenori Yamamoto, Yury Lysogorskiy, Lars Blumenthal, Thomas Hammerschmidt, Jacek R. Golebiowski, Xiangyue Liu, Angelo Ziletti, and Matthias Scheffler. 2019. Crowd-sourcing materials-science challenges with the NOMAD 2018 Kaggle competition. npj Computational Materials 5, 1 (2019), 1–11.

[119]

Christoph Tauchert, Peter Buxmann, and Jannis Lambinus. 2020. Crowdsourcing data science: A qualitative analysis of organizations’ usage of Kaggle competitions. In Proceedings of the 53rd Hawaii International Conference on System Sciences.

[120]

Mehmet Tekman, Bérénice Batut, Alexander Ostrovsky, Christophe Antoniewski, Dave Clements, Fidel Ramirez, Graham J. Etherington, Hans-Rudolf Hotz, Jelle Scholtalbers, Jonathan R. Manning, et al. 2020. A single-cell RNA-seq training and analysis suite using the Galaxy framework. Gigascience 9, 10 (2020), giaa102.

[121]

Guillaume Theaud, Jean-Christophe Houde, Arnaud Boré, François Rheault, Felix Morency, and Maxime Descoteaux. 2020. TractoFlow: A robust, efficient and reproducible diffusion MRI pipeline leveraging Nextflow & Singularity. NeuroImage 218 (2020), 116889.

[122]

Curtis Thompson. 2020. [Killer Shrimp] 2nd Place Solution. Retrieved December 26, 2023 from https://www.kaggle.com/cwthompson/killer-shrimp-2nd-place-solution

[123]

Marko Toplak, Stuart T. Read, Christophe Sandt, and Ferenc Borondics. 2021. Quasar: Easy machine learning for biospectroscopy. Cells 10, 9 (2021), 2300.

[124]

Wil M. P. Van Der Aalst and Arthur H. M. Ter Hofstede. 2005. YAWL: Yet another workflow language. Information Systems 30, 4 (2005), 245–275.

Digital Library

[125]

Yves Vandenbrouck, David Christiany, Florence Combes, Valentin Loux, and Virginie Brun. 2019. Bioinformatics tools and workflow to select blood biomarkers for early cancer diagnosis: An application to pancreatic cancer. Proteomics 19, 21-22 (2019), 1800489.

[126]

Jessica Velasco, Cherry Pascion, Jean Wilmar Alberio, Jonathan Apuang, John Stephen Cruz, Mark Angelo Gomez, Benjamin Molina Jr., Lyndon Tuala, August Thio-ac, and Romeo Jorda Jr. 2019. A smartphone-based skin disease classification using MobileNet CNN. arXiv preprint arXiv:1911.07929 (2019).

[127]

Markus Voelter, Bernd Kolb, Klaus Birken, Federico Tomassetti, Patrick Alff, Laurent Wiart, Andreas Wortmann, and Arne Nordmann. 2019. Using language workbenches and domain-specific languages for safety-critical software development. Software & Systems Modeling 18, 4 (2019), 2507–2530.

Digital Library

[128]

Anthony Yu-Tung Wang, Ryan J. Murdock, Steven K. Kauwe, Anton O. Oliynyk, Aleksander Gurlo, Jakoah Brgoch, Kristin A. Persson, and Taylor D. Sparks. 2020. Machine learning for materials scientists: An introductory guide toward best practices. Chemistry of Materials 32, 12 (2020), 4954–4965. DOI:

[129]

Dakuo Wang, Q. Vera Liao, Yunfeng Zhang, Udayan Khurana, Horst Samulowitz, Soya Park, Michael Muller, and Lisa Amini. 2021. How much automation does a data scientist want? arXiv preprint arXiv:2101.03970 (2021).

[130]

Yiping Wen, Junjie Hou, Zhen Yuan, and Dong Zhou. 2020. Heterogeneous information network-based scientific workflow recommendation for complex applications. Complexity 2020 (2020), 4129063.

Digital Library

[131]

Mark D. Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, et al. 2016. The FAIR guiding principles for scientific data management and stewardship. Scientific Data 3, 1 (2016), 1–9.

[132]

Julia K. Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk, Teresa Deinlein, Rainer Hofmann-Wellenhof, Luc Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, et al. 2019. Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition. JAMA Dermatology 155, 10 (2019), 1135–1141.

[133]

Laura Wratten, Andreas Wilm, and Jonathan Göke. 2021. Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers. Nature Methods 18, 10 (2021), 1161–1168.

[134]

Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. 2023. A survey on large language models for recommendation. arXiv preprint arXiv:2305.19860 (2023).

[135]

Yinhao Wu, Bin Chen, An Zeng, Dan Pan, Ruixuan Wang, and Shen Zhao. 2022. Skin cancer classification with deep learning: A systematic review. Frontiers in Oncology 12 (2022), 893972.

[136]

Moe Thandar Wynn, H. M. W. Verbeek, Wil M. P. van der Aalst, Arthur H. M. ter Hofstede, and David Edmond. 2009. Business process verification—Finally a reality! Business Process Management Journal 15, 1 (2009), 74–92.

[137]

Yuanshun Yao, Zhujun Xiao, Bolun Wang, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. 2017. Complexity vs. performance: Empirical analysis of machine learning as a service. In Proceedings of the 2017 Internet Measurement Conference. 384–397.

Digital Library

[138]

Chaoning Zhang, Philipp Benz, Dawit Mureja Argaw, Seokju Lee, Junsik Kim, Francois Rameau, Jean-Charles Bazin, and In So Kweon. 2021. ResNet or DenseNet? Introducing dense shortcuts to ResNet. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3550–3559.

[139]

Guoqing Zhou, Ben Nebgen, Nicholas Lubbers, Walter Malone, Anders M. N. Niklasson, and Sergei Tretiak. 2020. Graphics processing unit-accelerated semiempirical Born Oppenheimer molecular dynamics using PyTorch. Journal of Chemical Theory and Computation 16, 8 (2020), 4951–4962.

Cited By

Clear TCajander ÅClear AMcDermott RDaniels MDivitini MForshaw MHumble NKasinidou MKleanthous SKultur CParvini GPolash MZhu TMonga MLonati VBarendsen ESheard JPaterson J(2025)AI Integration in the IT Professional Workplace: A Scoping Review and Interview Study with Implications for Education and Professional Competencies2024 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3689187.3709607(34-67)Online publication date: 22-Jan-2025
https://dl.acm.org/doi/10.1145/3689187.3709607
Versino STurchi TMalizia A(2024)Leveraging Visual Languages to Foster User Participation in Designing Trustworthy Machine Learning Systems: A Comparative Study2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00013(24-32)Online publication date: 2-Sep-2024
https://doi.org/10.1109/VL/HCC60511.2024.00013
Jiang HShi SZhang SZheng JLi Q(2024)SLInterpreter: An Exploratory and Iterative Human-AI Collaborative System for GNN-Based Synthetic Lethal PredictionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345632531:1(919-929)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456325

Index Terms

Building Domain-Specific Machine Learning Workflows: A Conceptual Framework for the State of the Practice
1. Computing methodologies
  1. Artificial intelligence
    1. Knowledge representation and reasoning
  2. Machine learning

Recommendations

The role of machine learning in scientific workflows

Machine learning (ML) is being applied in a number of everyday contexts from image recognition, to natural language processing, to autonomous vehicles, to product recommendation. In the science realm, ML is being used for medical diagnosis, new materials ...
Scientific workflows

This article commences the special section on scientific workflows. A brief history is given of workflow approaches in general. More detail is given of scientific workflows, including sample applications. Challenges and research issues are then ...
A Dataflow-Based Scientific Workflow Composition Framework

Scientific workflow has recently become an enabling technology to automate and speed up the scientific discovery process. Although several scientific workflow management systems (SWFMSs) have been developed, a formal scientific workflow composition ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 33, Issue 4

May 2024

940 pages

EISSN:1557-7392

DOI:10.1145/3613665

Editor:
Mauro Pezzè
USI Università della Svizzera italiana and SIT Schaffhausen Institute of Technology, Switzerland

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2024

Online AM: 21 December 2023

Accepted: 02 December 2023

Revised: 24 November 2023

Received: 06 April 2022

Published in TOSEM Volume 33, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
529
Total Downloads

Downloads (Last 12 months)461
Downloads (Last 6 weeks)36

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Clear TCajander ÅClear AMcDermott RDaniels MDivitini MForshaw MHumble NKasinidou MKleanthous SKultur CParvini GPolash MZhu TMonga MLonati VBarendsen ESheard JPaterson J(2025)AI Integration in the IT Professional Workplace: A Scoping Review and Interview Study with Implications for Education and Professional Competencies2024 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3689187.3709607(34-67)Online publication date: 22-Jan-2025
https://dl.acm.org/doi/10.1145/3689187.3709607
Versino STurchi TMalizia A(2024)Leveraging Visual Languages to Foster User Participation in Designing Trustworthy Machine Learning Systems: A Comparative Study2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00013(24-32)Online publication date: 2-Sep-2024
https://doi.org/10.1109/VL/HCC60511.2024.00013
Jiang HShi SZhang SZheng JLi Q(2024)SLInterpreter: An Exploratory and Iterative Human-AI Collaborative System for GNN-Based Synthetic Lethal PredictionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345632531:1(919-929)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456325

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents