research-article

Open access

Composable Workflow for Accelerating Neural Architecture Search Using In Situ Analytics for Protein Classification

Authors:

Georgia Channing,

Ariel Rorabaugh,

Osamu Miyashita,

Silvina Caino-Lores,

Catherine Schuman,

Michela TauferAuthors Info & Claims

ICPP '23: Proceedings of the 52nd International Conference on Parallel Processing

Page 1

https://doi.org/10.1145/3605573.3605636

Published: 13 September 2023 Publication History

All formats PDF

Abstract

Neural architecture search (NAS), which automates the design of neural network (NN) architectures for scientific datasets, requires significant computational resources and time — often on the order of days or weeks of GPU hours and training time. We design the Analytics for Neural Network (A4NN) workflow, a composable workflow that significantly reduces the time and resources required to design accurate and efficient NN architectures. We introduce a parametric fitness prediction strategy and distribute training across multiple accelerators to decrease the aggregated NN training time. A4NN rigorously record neural architecture histories, model states, and metadata to reproduce the search for near-optimal NNs. We demonstrate A4NN’s ability to reduce training time and resource consumption on a dataset generated by an X-ray Free Electron Laser (XFEL) experiment simulation. When deploying A4NN, we decrease training time by up to 37% and epochs required by up to 38%.

Supplemental Material

PDF File

Artifact Descriptor

Download
362.95 KB

References

[1]

Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. 2017. Accelerating Neural Architecture Search Using Performance Prediction. In Proceedings of the NIPS Workshop on Meta-Learning. arxiv:1705.10823 [cs.LG]

[2]

Prasanna Balaprakash, Romain Egele, Misha Salim, Stefan Wild, Venkatram Vishwanath, Fangfang Xia, Tom Brettin, and Rick Stevens. 2019. Scalable reinforcement-learning-based neural architecture search for cancer deep learning research. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis.

Digital Library

[3]

Han Cai, Ligeng Zhu, and Song Han. 2018. Proxylessnas: Direct Neural Architecture Search on Target Task and Hardware. arXiv preprint arXiv:1812.00332 (2018).

[4]

Ryan Chard, Zhuozhao Li, Kyle Chard, Logan Ward, Yadu Babuji, Anna Woodard, Steven Tuecke, Ben Blaiszik, Michael Franklin, and Ian Foster. 2019. DLHub: Model and Data Serving for Science. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IPDPS). 283–292.

[5]

Shanyu Chen, Zhipeng He, Xinyin Han, Xiaoyu He, Ruilin Li, Haidong Zhu, Dan Zhao, Chuangchuang Dai, Yu Zhang, Zhonghua Lu, Xuebin Chi, and Beifang Niu. 2019. How Big Data and High-performance Computing Drive Brain Science. Genomics, Proteomics & Bioinformatics 17, 4 (2019), 381–392. Big Data in Brain Science.

[6]

Anshul Choudhary, John F. Lindner, Elliott G. Holliday, Scott T. Miller, Sudeshna Sinha, and William L. Ditto. 2020. Physics-enhanced Neural Networks Learn Order and Chaos. Phys. Rev. E 101 (Jun 2020), 062207. Issue 6. https://link.aps.org/doi/10.1103/PhysRevE.101.062207

[7]

Juan-Pablo Correa-Baena, Kedar Hippalgaonkar, Jeroen van Duren, Shaffiq Jaffer, Vijay R. Chandrasekhar, Vladan Stevanovic, Cyrus Wadia, Supratik Guha, and Tonio Buonassisi. 2018. Accelerating Materials Development via Automation, Machine Learning, and High-Performance Computing. Joule 2, 8 (2018), 1410–1420.

[8]

Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015. Speeding up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves. In Proceedings of the 24th International Conference on Artificial Intelligence (Buenos Aires, Argentina) (IJCAI’15). AAAI Press, 3460–3468.

Digital Library

[9]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, and Sylvain Gelly. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv preprint arXiv:2010.11929 (2020).

[10]

Francisco Erivaldo Fernandes Junior and Gary G. Yen. 2019. Particle Swarm Optimization of Deep Neural Networks Architectures for Image Classification. Swarm and Evolutionary Computation 49 (2019), 62–74.

[11]

A. Gertych, Z. Swiderska-Chadaj, and Z Ma. 2019. Convolutional Neural Networks Can Accurately Distinguish Four Histologic Growth Patterns of Lung Adenocarcinoma in Digital Slides. Scientific Reports 9, 1483 (2019).

[12]

Jie Hou, Badri Adhikari, and Cheng Jianlin. 2018. DeepSF: Deep Convolutional Neural Network for Mapping Protein Sequences to Folds. Bioinformatics 34:8 (2018), 1295–1303.

[13]

Haifeng Jin, Qingquan Song, and Xia Hu. 2019. Auto-keras: An efficient neural architecture search system. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2019).

Digital Library

[14]

Travis Johnston, Steven R. Young, David Hughes, Robert M. Patton, and Devin White. 2017. Optimizing Convolutional Neural Networks for Cloud Detection. In Proceedings of the Machine Learning on HPC Environments (MLHPC) (Denver, CO, USA). Article 4, 9 pages.

Digital Library

[15]

A. Kamilaris and F. Prenafeta-Boldú. 2018. A Review of the Use of Convolutional Neural Networks in Agriculture. Journal of Agricultural Science 156:3 (2018), 312–322.

[16]

Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabás Póczos, and Eric P. Xing. 2018. Neural Architecture Search with Bayesian Optimisation and Optimal Transport. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. NeurIPS, 2020–2029.

Digital Library

[17]

M F Kasim, D Watson-Parris, L Deaconu, S Oliver, P Hatfield, D H Froula, G Gregori, M Jarvis, S Khatiwala, J Korenaga, and et al.2021. Building high accuracy emulators for scientific simulations with deep neural architecture search. Machine Learning: Science and Technology 3, 1 (2021), 015013.

[18]

Ariel Keller Rorabaugh, Silvina Caíno-Lores, Travis Johnston, and Michela Taufer. 2022. Building High-Throughput Neural Architecture Search Workflows via a Decoupled Fitness Prediction Engine. IEEE Transactions on Parallel and Distributed Systems 33, 11 (2022), 2913–2926.

[19]

Ariel Keller Rorabaugh, Silvina Caíno-Lores, Michael R. Wyatt II, Travis Johnston, and Michela Taufer. 2021. Architecture Descriptions and High Frequency Accuracy and Loss Data of Random Neural Networks Trained on Image Datasets.

[20]

Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. 2017. Learning Curve Prediction with Bayesian Neural Networks. In International Conference on Learning Representations. https://openreview.net/forum?id=S11KBYclx

[21]

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. arxiv:1603.06560 [cs.LG]

[22]

Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. 2018. Tune: A Research Platform for Distributed Model Selection and Training. arXiv preprint arXiv:1807.05118 (2018).

[23]

Chaoyue Liu and Mikhail Belkin. 2018. Accelerating SGD with Momentum for Over-parameterized Learning. arXiv preprint arXiv:1810.13395 (2018).

[24]

Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. 2018. Progressive Neural Architecture Search. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[25]

Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. 2018. Hierarchical Representations for Efficient Architecture Search. In Proceedings of the International Conference on Learning Representations.

[26]

Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. 2019. NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm. arxiv:1810.03522 [cs.CV]

[27]

Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2018. Neural Architecture Optimization. In Advances in Neural Information Processing Systems 31. NeurIPS.

[28]

Zhiping Mao, Ameya D. Jagtap, and George Em Karniadakis. 2020. Physics-informed Neural Networks for High-speed Flows. Computer Methods in Applied Mechanics and Engineering 360 (2020), 112789. https://www.sciencedirect.com/science/article/pii/S0045782519306814

[29]

Paula Olaya, Silvina Caino-Lores, Vanessa Lama, Ria Patel, Ariel Keller Rorabaugh, Osamu Miyashita, Florence Tama, and Michela Taufer. 2022. Identifying structural properties of proteins from X-ray free electron laser diffraction patterns. 2022 IEEE 18th International Conference on e-Science (e-Science) (2022).

[30]

Gyunam Park and Minseok Song. 2020. Predicting performances in business processes using deep neural networks. Decision Support Systems 129 (2020), 113191.

Digital Library

[31]

Ria Patel, Ariel Keller Rorabaugh, Paula Olaya, Silvina Caino-Lores, Georgia Channing, Catherine Schuman, Osamu Miyashita, Florence Tama, and Michela Taufer. 2022. A methodology to generate efficient neural networks for classification of scientific datasets. 2022 IEEE 18th International Conference on e-Science (e-Science) (2022).

[32]

Robert M Patton, J Travis Johnston, Steven R Young, Catherine D Schuman, Don D March, Thomas E Potok, Derek C Rose, Seung-Hwan Lim, Thomas P Karnowski, and Maxim A Ziatdinov. 2018. 167-PFlops Deep Learning for Electron Microscopy: From Learning Physics to Atomic Manipulation. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC). 638–648.

Digital Library

[33]

Robert M. Patton, J. Travis Johnston, Steven R. Young, Catherine D. Schuman, Thomas E. Potok, Derek C. Rose, Seung-Hwan Lim, Junghoon Chae, Le Hou, Shahira Abousamra, Dimitris Samaras, and Joel Saltz. 2019. Exascale Deep Learning to Accelerate Cancer Research. In 2019 IEEE International Conference on Big Data (Big Data). 1488–1496.

[34]

Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. 2018. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 (2018).

[35]

Rahul Ramesh and Pratik Chaudhari. 2022. Model Zoo: A Growing "Brain" That Learns Continually. arxiv:2106.03027 [cs.LG]

[36]

Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. 2017. Large-scale Evolution of Image Classifiers. In Proceedings of the 34th International Conference on Machine Learning - Volume 70. ICML, 2902–2911.

[37]

Binxin Ru, Clare Lyle, Lisa Schut, Miroslav Fil, Mark van der Wilk, and Yarin Gal. 2021. Speedy Performance Estimation for Neural Architecture Search. https://arxiv.org/abs/2006.04492. arxiv:2006.04492 [stat.ML]

[38]

Max Schwarzer, Bryce Rogan, Yadong Ruan, Zhengming Song, Diana Y. Lee, Allon G. Percus, Viet T. Chau, Bryan A. Moore, Esteban Rougier, Hari S. Viswanathan, and Gowri Srinivasan. 2019. Learning to Fail: Predicting Fracture Evolution in Brittle material models using recurrent graph convolutional neural networks. Computational Materials Science 162 (2019), 322–332. https://www.sciencedirect.com/science/article/pii/S0927025619301223

[39]

Jonathan Shlomi, Peter Battaglia, and Jean-Roch Vlimant. 2021. Graph Neural Networks in Particle Physics. Machine Learning: Science and Technology 2, 2 (Jan 2021), 021001. https://doi.org/10.1088/2632-2153/abbf9a

[40]

Tiberiu Stan, Zachary T. Thompson, and Peter W. Voorhees. 2020. Optimizing Convolutional Neural Networks to Perform Semantic Segmentation on Large Materials Imaging Datasets: X-ray Tomography and Serial Sectioning. Materials Characterization 160 (2020), 110119. https://www.sciencedirect.com/science/article/pii/S1044580319304930

[41]

Yanan Sun, Bing Xue, Mengjie Zhang, and Gary G. Yen. 2020. Evolving deep convolutional neural networks for Image Classification. IEEE Transactions on Evolutionary Computation 24, 2 (2020), 394–407.

[42]

Tom Viering and Marco Loog. 2021. The Shape of Learning Curves: a Review. arXiv preprint arXiv:2103.10948 (2021).

[43]

Guangyu Robert Yang and Xiao-Jing Wang. 2020. Artificial Neural Networks for Neuroscientists: A Primer. Neuron 107, 6 (2020), 1048–1070.

[44]

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019. Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes. arXiv preprint arXiv:1904.00962 (2019).

[45]

Steven R. Young, Derek C. Rose, Travis Johnston, William T. Heller, Thomas P. Karnowski, Thomas E. Potok, Robert M. Patton, Gabriel Perdue, and Jonathan Miller. 2017. Evolving Deep Networks using HPC. Proceedings of the Machine Learning on HPC Environments (2017).

Digital Library

[46]

Xiaolong Zheng, Peng Zheng, Liang Zheng, Yang Zhang, and Rui-Zhi Zhang. 2020. Multi-channel Convolutional Neural Networks for Materials Properties Prediction. Computational Materials Science 173 (2020), 109436. https://www.sciencedirect.com/science/article/pii/S0927025619307359

[47]

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning transferable architectures for Scalable Image Recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018).

Cited By

An SChanning GSchuman CTaufer M(2023)VINARCH: A Visual Analytics Interactive Tool for Neural Network Archaeology2023 IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops)10.1109/CLUSTERWorkshops61457.2023.00020(50-51)Online publication date: 31-Oct-2023
https://doi.org/10.1109/CLUSTERWorkshops61457.2023.00020

Index Terms

Composable Workflow for Accelerating Neural Architecture Search Using In Situ Analytics for Protein Classification
1. Applied computing
  1. Physical sciences and engineering
    1. Chemistry
    2. Physics
2. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification
    2. Machine learning approaches
      1. Bio-inspired approaches
        Genetic algorithms
      2. Neural networks

Recommendations

Neural Architecture Search Applied to Hybrid Morphological Neural Networks
Intelligent Systems
Abstract
This work addresses a way to train morphological neural network differentially using backpropagation. The proposed algorithm can also learn whether to use erosion or dilation, based on the data being processed. Finally, we apply architecture ...
Fine-Grained Software Vulnerability Detection via Neural Architecture Search
Database Systems for Advanced Applications
Abstract
Vulnerability detection methods based on the deep learning have achieved remarkable performance improvements compared to traditional methods. Current deep learning-based detectors mostly use a single RNN or its variants (i.e., LSTM or GRU) to ...
Graph neural architecture prediction
Abstract
Graph neural networks (GNNs) have shown their superiority in the modeling of graph data. Recently, increasing attention has been paid to automatic graph neural architecture search, aiming to overcome the shortcomings of manually constructing GNN ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICPP '23: Proceedings of the 52nd International Conference on Parallel Processing

August 2023

858 pages

ISBN:9798400708435

DOI:10.1145/3605573

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

ICPP 2023

ICPP 2023: 52nd International Conference on Parallel Processing

August 7 - 10, 2023

UT, Salt Lake City, USA

Acceptance Rates

Overall Acceptance Rate 91 of 313 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
334
Total Downloads

Downloads (Last 12 months)258
Downloads (Last 6 weeks)29

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

An SChanning GSchuman CTaufer M(2023)VINARCH: A Visual Analytics Interactive Tool for Neural Network Archaeology2023 IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops)10.1109/CLUSTERWorkshops61457.2023.00020(50-51)Online publication date: 31-Oct-2023
https://doi.org/10.1109/CLUSTERWorkshops61457.2023.00020

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents