Computer Science > Distributed, Parallel, and Cluster Computing
[Submitted on 17 Aug 2020 (v1), revised 27 Oct 2020 (this version, v6), latest version 15 Mar 2021 (v7)]
Title:AIPerf: Automated machine learning as an AI-HPC benchmark
View PDFAbstract:The plethora of complex artificial intelligence (AI) algorithms and available high performance computing (HPC) power stimulates the expeditious development of AI components with heterogeneous designs. Consequently, the need for cross-stack performance benchmarking of AI-HPC systems emerges rapidly. The current HPC benchmarks can not reflect AI computing power without representative workloads and the current AI benchmarks have fixed problem size therefore limited scalability. To address these issues, we propose an end-to-end benchmark suite utilizing automated machine learning (AutoML) that represents real AI scenarios. More importantly, AutoML is auto-adaptive to various scales of machines with an extreme computational cost therefore a desired workload. We implement the algorithms in a highly parallel and flexible way to ensure the efficiency and optimization potential on diverse systems with customizable configurations. The major metric to quantify the performance is floating-point operations per second (FLOPS) that is measured in an analytical and systematic approach. We verify the benchmark's stability at discrete timestamps and the linear scalability on various numbers of machines equipped with up to 400 AI accelerators. With flexible workload size as well as single metric measurement, our benchmark can scale from small clusters to large AI-HPC and rank them easily. The source code, specifications and detailed procedures are publicly accessible on GitHub.
Submission history
From: Zhixiang Ren [view email][v1] Mon, 17 Aug 2020 08:06:43 UTC (785 KB)
[v2] Wed, 19 Aug 2020 08:53:50 UTC (785 KB)
[v3] Wed, 26 Aug 2020 07:29:24 UTC (785 KB)
[v4] Thu, 8 Oct 2020 13:22:19 UTC (524 KB)
[v5] Mon, 12 Oct 2020 13:50:56 UTC (808 KB)
[v6] Tue, 27 Oct 2020 10:02:34 UTC (530 KB)
[v7] Mon, 15 Mar 2021 02:25:55 UTC (920 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.