research-article

Open access

The Environmental Cost of Engineering Machine Learning-Enabled Systems: A Mapping Study

Authors:

Kouider Chadli,

Goetz Botterweck,

Takfarinas SaberAuthors Info & Claims

EuroMLSys '24: Proceedings of the 4th Workshop on Machine Learning and Systems

Pages 200 - 207

https://doi.org/10.1145/3642970.3655828

Published: 22 April 2024 Publication History

Abstract

The integration of Machine Learning (ML) across public and industrial sectors has become widespread, posing unique challenges in comparison to conventional software development methods throughout the lifecycle of ML-Enabled Systems. Particularly, with the rising importance of ML platforms in software operations and the computational power associated with their frequent training, testing, and retraining, there is a growing concern about the sustainability of DevOps practices in the context of Al-enabled software. Despite the increasing interest in this domain, a comprehensive overview that offers a holistic perspective on research related to sustainable AI is currently lacking. This paper addresses this gap by presenting a Systematic Mapping Study that thoroughly examines techniques, tools, and lessons learned to assess and promote environmental sustainability in MLOps practices for ML-Enabled Systems.

References

[1]

Bilge Acun, Benjamin Lee, Fiodar Kazhamiaka, Kiwan Maeng, Udit Gupta, Manoj Chakkaravarthy, David Brooks, and Carole-Jean Wu. 2023. Carbon explorer: A holistic framework for designing carbon aware datacenters. In ACM. 118--132.

[2]

Phyllis Ang, Bhuwan Dhingra, and Lisa Wu Wills. 2022. Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models. arXiv preprint arXiv:2204.07288 (2022).

[3]

Lasse F Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. 2020. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. arXiv preprint arXiv:2007.03051 (2020).

[4]

Alexis J Bañon Gomis, Manuel Guillén Parra, W Michael Hoffman, and Robert E McNulty. 2011. Rethinking the concept of sustainability. Business and Society Review 116, 2 (2011), 171--191.

[5]

Christoph Becker, Ruzanna Chitchyan, Leticia Duboc, Steve Easterbrook, Martin Mahaux, Birgit Penzenstadler, Guillermo Rodriguez-Navas, Camille Salinesi, Norbert Seyff, Colin Venters, et al. 2014. The Karlskrona manifesto for sustainability design. arXiv preprint arXiv:1410.6968 (2014).

[6]

Alexander EI Brownlee, Jason Adair, Saemundur O Haraldsson, and John Jabbo. 2021. Exploring the accuracy-energy trade-off in machine learning. In GI. IEEE, 11--18.

[7]

Semen Andreevich Budennyy, Vladimir Dmitrievich Lazarev, Nikita Nikolaevich Zakharenko, Aleksei N Korovin, OA Plosskaya, Denis Valer'evich Dimitrov, VS Akhripkin, IV Pavlov, Ivan Valer'evich Oseledets, Ivan Segundovich Barsola, et al. 2022. Eco2ai: carbon emissions tracking of machine learning models as the first step towards sustainable ai. In Doklady Mathematics, Vol. 106. Springer, S118-S128.

[8]

Qingqing Cao, Aruna Balasubramanian, and Niranjan Balasubramanian. 2020. Towards accurate and reliable energy measurement of NLP models. arXiv preprint arXiv:2010.05248 (2020).

[9]

Qingqing Cao, Yash Kumar Lal, Harsh Trivedi, Aruna Balasubramanian, and Niranjan Balasubramanian. 2021. IrEne: Interpretable energy prediction for transformers. arXiv preprint arXiv:2106.01199 (2021).

[10]

Joel Castaño, Silverio Martínez-Fernández, Xavier Franch, and Justus Bogner. 2023. Exploring the Carbon Footprint of Hugging Face's ML Models: A Repository Mining Study. arXiv preprint arXiv:2305.11164 (2023).

[11]

Dagoberto Castellanos-Nieves and Luis García-Forte. 2023. Improving Automated Machine-Learning Systems through Green AI. Applied Sciences 13, 20 (2023), 11583.

[12]

Vivek Chavan, Paul Koch, Marian Schlüter, and Clemens Briese. 2023. Towards realistic evaluation of industrial continual learning scenarios with an emphasis on energy consumption and computational footprint. In IEEE/CVF. 11506--11518.

[13]

Lucas Høyberg Puvis de Chavannes, Mads Guldborg Kjeldgaard Kongsbak, Timmie Rantzau, and Leon Derczynski. 2021. Hyperparameter power impact in transformer language model training. In Proceedings of workshop on simple and efficient natural language processing. 96--118.

[14]

Santiago del Rey, Silverio Martínez-Fernández, Luís Cruz, and Xavier Franch. 2023. Do DL models and training environments have an impact on energy consumption?. In SEAA. IEEE, 150--158.

[15]

Radosvet Desislavov, Fernando Martínez-Plumed, and José Hernández-Orallo. 2023. Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning. Sustainable Computing: Informatics and Systems 38 (2023), 100857.

[16]

Jesse Dodge, Taylor Prewitt, Remi Tachet des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexandra Sasha Luccioni, Noah A Smith, Nicole DeCario, and Will Buchanan. 2022. Measuring the carbon intensity of AI in cloud instances. In Proceedings of ACM Conference on Fairness, Accountability, and Transparency. 1877--1894.

Digital Library

[17]

Tamar Eilam, Pedro Bello-Maldonado, Bishwaranjan Bhattacharjee, Carlos Costa, Eun Kyung Lee, and Asser Tantawi. 2023. Towards a Methodology and Framework for AI Sustainability Metrics. In Proceedings of Workshop on Sustainable Computer Systems. 1--7.

Digital Library

[18]

Brad Everman, Trevor Villwock, Dayuan Chen, Noe Soto, Oliver Zhang, and Ziliang Zong. 2023. Evaluating the Carbon Impact of Large Language Models at the Inference Stage. In IPCCC. IEEE, 150--157.

[19]

Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Parteek Sharma, Fan Chen, and Lei Jiang. 2023. LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models. arXiv preprint arXiv:2309.14393 (2023).

[20]

Ahmed Bahaa Farid, Yehia Mostafa Helmy, and Mahmoud Mohamed Bahloul. 2017. Enhancing Lean Software Development by using Devops Practices. International Journal of Advanced Computer Science and Applications 8, 7 (2017).

[21]

Nathan C Frey, Dan Zhao, Simon Axelrod, Michael Jones, David Bestor, Vijay Gadepally, Rafael Gómez-Bombarelli, and Siddharth Samsi. 2022. Energy-aware neural architecture selection and hyperparameter optimization. In IPDPSW. IEEE, 732--741.

[22]

Anshul Gandhi, Dongyoon Lee, Zhenhua Liu, Shuai Mu, Erez Zadok, Kanad Ghose, Kartik Gopalan, Yu David Liu, Syed Rafiul Hussain, and Patrick Mcdaniel. 2023. Metrics for sustainability in data centers. ACM SIGENERGY Energy Informatics Review 3, 3 (2023), 40--46.

Digital Library

[23]

Eva García-Martín, Crefeda Faviola Rodrigues, Graham Riley, and Håkan Grahn. 2019. Estimation of energy consumption in machine learning. J. Parallel and Distrib. Comput. 134 (2019), 75--88.

Digital Library

[24]

Stefanos Georgiou, Maria Kechagia, Tushar Sharma, Federica Sarro, and Ying Zou. 2022. Green ai: Do deep learning frameworks have different costs?. In Proceedings of International Conference on Software Engineering. 1082--1094.

Digital Library

[25]

N. Gift and A. Deza. 2021. Practical MLOps: Operationalizing Machine Learning Models. O'Reilly.

[26]

Md Yousuf Harun and Christopher Kanan. 2023. Overcoming the Stability Gap in Continual Learning. arXiv preprint arXiv:2306.01904 (2023).

[27]

Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. 2020. Towards the systematic reporting of the energy and carbon footprints of machine learning. The Journal of Machine Learning Research 21, 1 (2020), 10039--10081.

Digital Library

[28]

Chip Huyen. 2022. Designing machine learning systems. "O'Reilly Media, Inc.".

[29]

Claire Jean-Quartier, Katharina Bein, Lukas Hejny, Edith Hofer, Andreas Holzinger, and Fleur Jeanquartier. 2023. The Cost of Understanding---XAI Algorithms towards Sustainable ML in the View of Computational Cost. Computation 11, 5 (2023), 92.

[30]

Sorin Liviu Jurj, Flavius Opritoiu, and Mircea Vladutiu. 2020. Environmentally-friendly metrics for evaluating the performance of deep learning models and systems. In ICONIP. Springer, 232--244.

[31]

Lynn H Kaack, Priya L Donti, Emma Strubell, George Kamiya, Felix Creutzig, and David Rolnick. 2022. Aligning artificial intelligence with climate change mitigation. Nature Climate Change 12, 6 (2022), 518--527.

[32]

Minsu Kim, Walid Saad, Mohammad Mozaffari, and Merouane Debbah. 2022. On the tradeoff between energy, precision, and accuracy in federated quantized neural networks. In IEEE International Conference on Communications. IEEE, 2194--2199.

[33]

Ask Berstad Kolltveit and Jingyue Li. 2022. Operationalizing machine learning models: a systematic literature review. In Proceedings of Workshop on Software Engineering for Responsible AI. 1--8.

Digital Library

[34]

Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 (2019).

[35]

Loic Lannelongue, Jason Grealey, and Michael Inouye. 2021. Green algorithms: quantifying the carbon footprint of computation. Advanced science 8, 12 (2021), 2100707.

[36]

Leonardo Leite, Carla Rocha, Fabio Kon, Dejan Milojicic, and Paulo Meirelles. 2019. A survey of DevOps concepts and challenges. ACM Computing Surveys (CSUR) 52, 6 (2019), 1--35.

Digital Library

[37]

Baolin Li, Rohan Basu Roy, Daniel Wang, Siddharth Samsi, Vijay Gadepally, and Devesh Tiwari. 2023. Toward Sustainable HPC: Carbon Footprint Estimation and Environmental Implications of HPC Systems. In Proceedings of the International D27 for High Performance Computing, Networking, Storage and Analysis. 1--15.

Digital Library

[38]

Baolin Li, Siddharth Samsi, Vijay Gadepally, and Devesh Tiwari. 2023. Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1--15.

Digital Library

[39]

Alexandra Sasha Luccioni, Yacine Jernite, and Emma Strubell. 2023. Power Hungry Processing: Watts Driving the Cost of AI Deployment? arXiv preprint arXiv:2311.16863 (2023).

[40]

Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2023. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research 24, 253 (2023), 1--15.

[41]

Jens Malmodin and Dag Lundén. 2018. The energy and carbon footprint of the global ICT and E&M sectors 2010-2015. Sustainability 10, 9 (2018), 3027.

[42]

Joseph McDonald, Baolin Li, Nathan Frey, Devesh Tiwari, Vijay Gadepally, and Siddharth Samsi. 2022. Great power, great responsibility: Recommendations for reducing energy for training language models. arXiv preprint arXiv:2205.09646 (2022).

[43]

Sudipto Mondal, Fashat Bin Faruk, Dibosh Rajbongshi, Mohammad Masum Khondhoker Efaz, and Md Motaharul Islam. 2023. GEECO: Green Data Centers for Energy Optimization and Carbon Footprint Reduction. Sustainability 15, 21 (2023), 15249.

[44]

Gianluca Moro, Luca Ragazzi, and Lorenzo Valgimigli. 2023. Carburacy: summarization models tuning and comparison in eco-sustainable regimes with a novel carbon-aware accuracy. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 14417--14425.

Digital Library

[45]

Nadia Nahar, Shurui Zhou, Grace Lewis, and Christian Kästner. 2022. Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process. In Proceedings of International Conference on Software Engineering (ICSE). Association for Computing Machinery, 413--425.

Digital Library

[46]

David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350 (2021).

[47]

Birgit Penzenstadler and Henning Femmer. 2013. A generic model for sustainability with process-and product-specific instances. In Proceedings of workshop on Green in/by software engineering. 3--8.

Digital Library

[48]

Ana Radovanović, Ross Koningstein, Ian Schneider, Bokan Chen, Alexandre Duarte, Binz Roy, Diyue Xiao, Maya Haridasan, Patrick Hung, Nick Care, et al. 2022. Carbon-aware computing for datacenters. IEEE Transactions on Power Systems 38, 2 (2022), 1270--1280.

[49]

Ankita Raturi, Birgit Penzenstadler, Bill Tomlinson, and Debra Richardson. 2014. Developing a sustainability non-functional requirements framework. In Proceedings of International Workshop on Green and Sustainable Software. 1--8.

Digital Library

[50]

Philipp Ruf, Manav Madan, Christoph Reich, and Djaffar Ould-Abdeslam. 2021. Demystifying mlops and presenting a recipe for the selection of open-source tools. Applied Sciences 11, 19 (2021), 8861.

[51]

Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. 2023. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference. In HPEC. IEEE, 1--9.

[52]

Stefano Savazzi, Vittorio Rampa, Sanaz Kianoush, and Mehdi Bennis. 2022. An energy and carbon footprint analysis of distributed and federated learning. IEEE Transactions on Green Communications and Networking 7, 1 (2022), 248--264.

[53]

Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. Green AI. Commun. ACM 63, 12 (11 2020), 54--63.

[54]

David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. Advances in neural information processing systems 28 (2015).

[55]

Raghavendra Selvan, Nikhil Bhagwat, Lasse F Wolff Anthony, Benjamin Kanding, and Erik B Dam. 2022. Carbon footprint of selecting and training deep learning models for medical image analysis. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 506--516.

Digital Library

[56]

Lucas Cardoso Silva, Fernando Rezende Zagatti, Bruno Silva Sette, Lucas Nildaimon dos Santos Silva, Daniel Lucrédio, Diego Furtado Silva, and Helena de Medeiros Caseli. 2020. Benchmarking machine learning solutions in production. In ICMLA. IEEE, 626--633.

[57]

Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.02243 (2019).

[58]

Ayesha Tabassam. 2023. MLOps: A Step Forward to Enterprise Machine Learning. ArXiv abs/2305.19298 (2023).

[59]

Neil C Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F Manso. 2020. The computational limits of deep learning. arXiv preprint arXiv:2007.05558 (2020).

[60]

Tanja Tornede, Alexander Tornede, Jonas Hanselle, Felix Mohr, Marcel Wever, and Eyke Hűllermeier. 2023. Towards green automated machine learning: Status quo and future directions. Journal of Artificial Intelligence Research 77 (2023), 427--457.

Digital Library

[61]

Roberto Verdecchia, Luís Cruz, June Sallou, Michelle Lin, James Wickenden, and Estelle Hotellier. 2022. Data-centric green ai an exploratory empirical study. In ICT4S. IEEE, 35--45.

[62]

Roberto Verdecchia, June Sallou, and Luís Cruz. 2023. A systematic review of Green AI. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery (2023), e1507.

[63]

Xiaorong Wang, Clara Na, Emma Strubell, Sorelle Friedler, and Sasha Luccioni. 2023. Energy and Carbon Considerations of Fine-Tuning BERT. arXiv preprint arXiv:2311.10267 (2023).

[64]

Dustin Wright, Christian Igel, Gabrielle Samuel, and Raghavendra Selvan. 2023. Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AI. arXiv preprint arXiv:2309.02065 (2023).

[65]

Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. 2022. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems 4 (2022), 795--813.

[66]

Yinlena Xu, Silverio Martínez-Fernández, Matias Martinez, and Xavier Franch. 2023. Energy Efficiency of Training Neural Network Architectures: An Empirical Study. arXiv preprint arXiv:2302.00967 (2023).

[67]

Tim Yarally, Luis Cruz, Daniel Feitosa, June Sallou, and Arie Van Deursen. 2023. Uncovering energy-efficient practices in deep learning training: Preliminary steps towards green ai. In CAIN. IEEE, 25--36.

[68]

Jie You, Jae-Won Chung, and Mosharaf Chowdhury. 2023. Zeus: Understanding and Optimizing {GPU} Energy Consumption of {DNN} Training. In NSDI. 119--139.

[69]

Ashkan Yousefpour, Shen Guo, Ashish Shenoy, Sayan Ghosh, Pierre Stock, Kiwan Maeng, Schalk-Willem Krüger, Michael Rabbat, Carole-Jean Wu, and Ilya Mironov. 2023. Green Federated Learning. arXiv preprint arXiv:2303.14604 (2023).

Cited By

Deroy OBacciu DBahrami BDella Santina CHauert S(2024)Shared Awareness Across Domain‐Specific Artificial Intelligence: An Alternative to Domain‐General Intelligence and Artificial ConsciousnessAdvanced Intelligent Systems10.1002/aisy.2023007406:10Online publication date: 17-Jul-2024
https://doi.org/10.1002/aisy.202300740

Recommendations

Gray clustering assessment of environmental cost-benefit study in enterprise—A case of Haier
Applied Mathematics Related to Nonlinear Problems

The Haier’s environmental invest have a positive impact on Economic and environmental performance. The pollution and waste created by enterprises has increasingly become a serious problem as it leads to deterioration of ecological environment and climate ...
Research on Coal Environmental Cost Optimization Strategy
ICEET '09: Proceedings of the 2009 International Conference on Energy and Environment Technology - Volume 01

With an increase in the scale of society and the economy, coal environmental problems are becoming more serious and the environmental cost is increasing twofold. Environmental problems cause many social issues and economic losses for enterprises. The ...
Toward sustainable software engineering (NIER track)
ICSE '11: Proceedings of the 33rd International Conference on Software Engineering

Current software engineering practices have significant effects on the environment. Examples include e-waste from computers made obsolete due to software upgrades, and changes in the power demands of new versions of software. Sustainable software ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

EuroMLSys '24: Proceedings of the 4th Workshop on Machine Learning and Systems

April 2024

218 pages

ISBN:9798400705410

DOI:10.1145/3642970

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 April 2024

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Lero

Conference

EuroSys '24

Sponsor:

SIGOPS

EuroSys '24: Nineteenth European Conference on Computer Systems

April 22, 2024

Athens, Greece

Acceptance Rates

Overall Acceptance Rate 18 of 26 submissions, 69%

Upcoming Conference

EuroSys '25

Sponsor:
sigops

Twentieth European Conference on Computer Systems

March 30 - April 3, 2025

Rotterdam , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
511
Total Downloads

Downloads (Last 12 months)511
Downloads (Last 6 weeks)75

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Deroy OBacciu DBahrami BDella Santina CHauert S(2024)Shared Awareness Across Domain‐Specific Artificial Intelligence: An Alternative to Domain‐General Intelligence and Artificial ConsciousnessAdvanced Intelligent Systems10.1002/aisy.2023007406:10Online publication date: 17-Jul-2024
https://doi.org/10.1002/aisy.202300740

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents