research-article

AutoAblation: Automated Parallel Ablation Studies for Deep Learning

Authors:

Sina Sheikholeslami,

Moritz Meister,

Amir H. Payberah,

Vladimir Vlassov,

Jim DowlingAuthors Info & Claims

EuroMLSys '21: Proceedings of the 1st Workshop on Machine Learning and Systems

Pages 55 - 61

https://doi.org/10.1145/3437984.3458834

Published: 26 April 2021 Publication History

Abstract

Ablation studies provide insights into the relative contribution of different architectural and regularization components to machine learning models' performance. In this paper, we introduce AutoAblation, a new framework for the design and parallel execution of ablation experiments. AutoAblation provides a declarative approach to defining ablation experiments on model architectures and training datasets, and enables the parallel execution of ablation trials. This reduces the execution time and allows more comprehensive experiments by exploiting larger amounts of computational resources. We show that AutoAblation can provide near-linear scalability by performing an ablation study on the modules of the Inception-v3 network trained on the TenGeoPSAR dataset.

References

[1]

M. Abadi et al. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265--283.

Digital Library

[2]

D. Berthelot et al. 2019. MixMatch: A Holistic Approach to Semi-Supervised Learning. arXiv preprint arXiv:1905.02249 (2019).

[3]

N. Carlson et al. 2009. Psychology: the Science of Behavior. Pearson.

[4]

B. Chambers and M. Zaharia. 2018. Spark: The Definitive Guide: Big Data Processing Made Simple. O'Reilly Media, Inc.

[5]

F. Chollet et al. 2015. Keras.

[6]

J. Deng et al. 2009. Imagenet: A Large-Scale Hierarchical Image Database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248--255.

[7]

A. Erdem et al. 2019. Leave One Feature Out Importance. https://github.com/aerdem4/lofo-importance.

[8]

W. A. Falcon et al. 2019. PyTorch Lightning. GitHub. https://github.com/PyTorchLightning/pytorch-lightning 3 (2019).

[9]

R. Girshick et al. 2014. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 580--587.

Digital Library

[10]

M. Hessel et al. 2018. Rainbow: Combining improvements in deep reinforcement learning. In 33 AAAI Conference on Artificial Intelligence.

[11]

E. Horvitz et al. 2003. Learning and reasoning about interruption. In Proceedings of the 5th International Conference on Multimodal Interfaces. ACM, 20--27.

Digital Library

[12]

Y. LeCun. 1998. The MNIST Database of Handwritten Digits. http://yann.lecun.com/exdb/mnist/.

[13]

Z. C. Lipton and J. Steinhardt. 2018. Troubling trends in machine learning scholarship. arXiv preprint arXiv:1807.03341 (2018).

[14]

S. M. Lundberg and SI. Lee. 2017. A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems 30 (2017), 4765--4774.

[15]

M. Meister et al. 2020. Maggy: Scalable Asynchronous Parallel Hyperparameter Search. In Workshop on Distributed Machine Learning. 28--33.

Digital Library

[16]

M. Meister et al. 2020. Towards Distribution Transparency for Supervised ML With Oblivious Training Functions. In Workshop on MLOps Systems.

[17]

R. Meyes et al. 2019. Ablation Studies in Artificial Neural Networks. arXiv preprint arXiv:1901.08644 (2019).

[18]

P. Moritz et al. 2018. Ray: A Distributed Framework for Emerging AI Applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 561--577.

[19]

T. O'Malley et al. 2019. Keras Tuner. https://github.com/keras-team/keras-tuner.

[20]

A. Paszke et al. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32 (2019), 8026--8037.

[21]

M. T. Ribeiro et al. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1135--1144.

Digital Library

[22]

M. Richardson et al. 2006. Beyond PageRank: Machine Learning for Static Ranking. In Proceedings of the 15th International Conference on World Wide Web. ACM, 707--715.

Digital Library

[23]

T. Sellam et al. 2019. DeepBase: Deep Inspection of Neural Networks. In Proceedings of the 2019 International Conference on Management of Data. 1117--1134.

Digital Library

[24]

S. Sheikholeslami. 2019. Ablation Programming for Machine Learning. Master's thesis.

[25]

C. Szegedy et al. 2016. Rethinking the Inception Architecture for Computer Vision. In IEEE Conference on Computer Vision and Pattern Recognition. 2818--2826.

[26]

C. Wang et al. 2019. A labelled ocean SAR imagery dataset of ten geophysical phenomena from Sentinel-1 wave mode. Geoscience Data Journal 6, 2 (2019), 105--115.

[27]

J. Wexler et al. 2019. The What-If Tool: Interactive Probing of Machine Learning Models. arXiv preprint arXiv:1907.04135 (2019).

[28]

L. Yang et al. 2017. Open Sourcing TensorFlowOnSpark: Distributed Deep Learning on Big-Data Clusters.

[29]

M. Zaharia et al. 2010. Spark: Cluster Computing with Working Sets. HotCloud 10, 10--10 (2010), 95.

Cited By

Skurowski PMyszor DPaszkuta MMoroń TCyran K(2024)Energy Demand in AR Applications—A Reverse Ablation Study of the HoloLens 2 DeviceEnergies10.3390/en1703055317:3(553)Online publication date: 23-Jan-2024
https://doi.org/10.3390/en17030553
Yoon JLee KOh JKim HJeong J(2024)Insights and Considerations in Development and Performance Evaluation of Generative Adversarial Networks (GANs): What Radiologists Need to KnowDiagnostics10.3390/diagnostics1416175614:16(1756)Online publication date: 13-Aug-2024
https://doi.org/10.3390/diagnostics14161756
Wanjau SWambugu GOirere AMuketha G(2024)Discriminative spatial-temporal feature learning for modeling network intrusion detection systemsJournal of Computer Security10.3233/JCS-22003132:1(1-30)Online publication date: 2-Feb-2024
https://dl.acm.org/doi/10.3233/JCS-220031
Show More Cited By

Index Terms

AutoAblation: Automated Parallel Ablation Studies for Deep Learning
1. Computing methodologies
  1. Machine learning
  2. Modeling and simulation
    1. Model development and analysis

Recommendations

A Deep Learning-based Classification Algorithm for the Origin of Premature Ventricular Contractions
ICBBB '24: Proceedings of the 2024 14th International Conference on Bioscience, Biochemistry and Bioinformatics

Catheter ablation is an effective and safe method for treating outflow tract ventricular arrhythmias. Preoperative preliminary localization of abnormal excitation origin is of great value in designing ablation strategies and improving the efficiency of ...
Automated detection of arrhythmias using different intervals of tachycardia ECG segments with convolutional neural network

Classification of normal and tachycardia arrhythmias ECG segments.Two and five seconds ECG segments are considered.Convolutional neural network is employed.QRS detection is not performed.Accuracy of 92.50% and 94.9% obtained for two and five seconds ...
A clinical study on Atrial Fibrillation, Premature Ventricular Contraction, and Premature Atrial Contraction screening based on an ECG deep learning model
Abstract
It is still a challenge to develop an electrocardiography (ECG) interpreter based on ECG basic characteristics because of the uncertainty of ECG delineation. Based on the clinical investigation in this study, ECG devices generated ...
Highlights
- Develop a deep learning model and algorithms to improve the precisions of clinically-used ECG machine interpretations on AF, PVC, and PAC.

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

EuroMLSys '21: Proceedings of the 1st Workshop on Machine Learning and Systems

April 2021

130 pages

ISBN:9781450382984

DOI:10.1145/3437984

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 April 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

H2020 European Research Council

Conference

EuroSys '21

Sponsor:

SIGOPS

EuroSys '21: Sixteenth European Conference on Computer Systems

April 26, 2021

Online, United Kingdom

Acceptance Rates

EuroMLSys '21 Paper Acceptance Rate 18 of 26 submissions, 69%;

Overall Acceptance Rate 18 of 26 submissions, 69%

Upcoming Conference

EuroSys '25

Sponsor:
sigops

Twentieth European Conference on Computer Systems

March 30 - April 3, 2025

Rotterdam , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
577
Total Downloads

Downloads (Last 12 months)203
Downloads (Last 6 weeks)24

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Skurowski PMyszor DPaszkuta MMoroń TCyran K(2024)Energy Demand in AR Applications—A Reverse Ablation Study of the HoloLens 2 DeviceEnergies10.3390/en1703055317:3(553)Online publication date: 23-Jan-2024
https://doi.org/10.3390/en17030553
Yoon JLee KOh JKim HJeong J(2024)Insights and Considerations in Development and Performance Evaluation of Generative Adversarial Networks (GANs): What Radiologists Need to KnowDiagnostics10.3390/diagnostics1416175614:16(1756)Online publication date: 13-Aug-2024
https://doi.org/10.3390/diagnostics14161756
Wanjau SWambugu GOirere AMuketha G(2024)Discriminative spatial-temporal feature learning for modeling network intrusion detection systemsJournal of Computer Security10.3233/JCS-22003132:1(1-30)Online publication date: 2-Feb-2024
https://dl.acm.org/doi/10.3233/JCS-220031
Defilippo AVeltri PLió PGuzzi P(2024)Leveraging graph neural networks for supporting automatic triage of patientsScientific Reports10.1038/s41598-024-63376-214:1Online publication date: 31-May-2024
https://doi.org/10.1038/s41598-024-63376-2
Deshpande KHolzweber JThalhuber SHämmerle AMayrhofer MPichler AFilho P(2024)Manufacturing Line Ablation, an approach to perform reliable early predictionProcedia Computer Science10.1016/j.procs.2024.01.075232:C(752-765)Online publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.procs.2024.01.075
Yang ZEmmert-Streib F(2024)Optimal performance of Binary Relevance CNN in targeted multi-label text classificationKnowledge-Based Systems10.1016/j.knosys.2023.111286284:COnline publication date: 17-Apr-2024
https://dl.acm.org/doi/10.1016/j.knosys.2023.111286
Altarabichi MNowaczyk SPashami SSheikholharam Mashhadi PHandl J(2024)Rolling the dice for better deep learning performanceInformation Sciences: an International Journal10.1016/j.ins.2024.120500667:COnline publication date: 1-May-2024
https://dl.acm.org/doi/10.1016/j.ins.2024.120500
Chowdhury SDurrani NAli A(2024)What do end-to-end speech models learn about speaker, language and channel information? A layer-wise and neuron-level analysisComputer Speech and Language10.1016/j.csl.2023.10153983:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.csl.2023.101539
Grushetskaya YSips MSchachtschneider RSaberioon MMahan A(2024)HPExplorer: XAI Method to Explore the Relationship Between Hyperparameters and Model PerformanceMachine Learning and Knowledge Discovery in Databases. Applied Data Science Track10.1007/978-3-031-70378-2_20(319-334)Online publication date: 22-Aug-2024
https://doi.org/10.1007/978-3-031-70378-2_20
Abela BMasek MAbu-Khalaf JSuter DGupta A(2024)An Exploration of Diabetic Foot Osteomyelitis X-ray Data for Deep Learning ApplicationsArtificial Intelligence in Medicine10.1007/978-3-031-66535-6_4(30-39)Online publication date: 25-Jul-2024
https://doi.org/10.1007/978-3-031-66535-6_4
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents