Cited By
View all- Mienye IJere N(2024)A Survey of Decision Trees: Concepts, Algorithms, and ApplicationsIEEE Access10.1109/ACCESS.2024.341683812(86716-86727)Online publication date: 2024
Rigorous Evaluation of Computer Processors with Statistical Model Checking
Pages 1242 - 1254
Abstract
Experiments with computer processors must account for the inherent variability in executions. Prior work has shown that real systems exhibit variability, and random effects must be injected into simulators to account for it. Thus, we can run multiple executions of a given benchmark and generate a distribution of results. Prior work uses standard statistical techniques that are not suitable. While the result distributions may take any forms that are unknown a priori, many works naively assume they are Gaussian, which can be far from the truth. To allow rigorous evaluation for arbitrary result distributions, we introduce statistical model checking (SMC) to the world of computer architecture. SMC is a statistical technique that is used in research communities that depend heavily on statistical guarantees. SMC provides a rigorous mathematical methodology that employs experimental sampling for probabilistic evaluation of properties of interest, such that one can determine with a desired confidence whether a property (e.g., System X is 1.1x faster than System Y) is true or not. SMC alone is not enough for computer architects to draw conclusions based on their data. We create an end-to-end framework called SMC for Processor Analysis (SPA) which utilizes SMC techniques to provide insightful conclusions given experimental data.
References
[1]
Gul Agha and Karl Palmskog. 2018. A survey of statistical model checking. ACM Transactions on Modeling and Computer Simulation 28, 1 (2018), 6:1–6:39.
[2]
A. R. Alameldeen, C. J. Mauer, M. Xu, P. J. Harper, M. M. K. Martin, and D. J. Sorin. 2002. Evaluating Non-deterministic Multi-threaded Commercial Workloads. In Proc. of Computer Architecture Evaluation Using Commercial Workloads.
[3]
Alaa R. Alameldeen and David A. Wood. 2003. Variability in Architectural Simulations of Multi-Threaded Workloads. In Proceedings of the 9th International Symposium on High-Performance Computer Architecture.
[4]
David Arney, Miroslav Pajic, Julian M. Goldman, Insup Lee, Rahul Mangharam, and Oleg Sokolsky. 2010. Toward Patient Safety in Closed-Loop Medical Device Systems. In ACM/IEEE International Conference on Cyber-Physical Systems.
[5]
C. Bienia and K. Li. 2009. PARSEC 2.0: A New Benchmark Suite for Chip-Multiprocessors. In Proc. of the 5th Annual Workshop on Modeling, Benchmarking and Simulation.
[6]
B. R. Bruce, A. Akram, H. Nguyen, K. Roarty, M. Samani, M. Friborz, T. Reddy, M. D. Sinclair, and J. Lowe-Power. 2021. Enabling Reproducible and Agile Full-System Simulation. In IEEE Int’l Symp. on Performance Analysis of Systems and Software.
[7]
B. S. Cade and B. R. Noon. 2003. A gentle introduction to quantile regression for ecologists. Frontiers in Ecology and the Environment 1, 8 (2003), 412–420.
[8]
T. E. Carlson, W. Heirman, and L. Eeckhout. 2011. Sniper: Exploring the Level of Abstraction for Scalable and Accurate Parallel Multi-Core Simulation. In SC.
[9]
G. Casella and R. L Berger. 2021. Statistical inference. Cengage Learning.
[10]
Tianshi Chen, Yunji Chen, Qi Guo, Olivier Temam, Yue Wu, and Weiwu Hu. 2012. Statistical performance comparisons of computers. In IEEE International Symposium on High-Performance Comp Architecture.
[11]
A. Cimatti, E. Clarke, F. Giunchiglia, and M. Roveri. 2000. NUSMV: A New Symbolic Model Checker. International Journal of Software Tools for Technology Transfer (2000).
[12]
M. R. Clarkson, B. Finkbeiner, M. Koleini, K. K. Micinski, M. N. Rabe, and C. Sanchez. 2014. Temporal Logics for Hyperproperties. In International Conference on Principles of Security and Trust.
[13]
Charles J Clopper and Egon S Pearson. 1934. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26, 4 (1934), 404–413.
[14]
Weilong Cui and Timothy Sherwood. 2017. Estimating and Understanding Architectural Risk. In International Symposium on Microarchitecture.
[15]
F. Dannenberg, M. Kwiatkowska, C. Thachuk, and A. Turberfield. 2013. DNA Walker Circuits: Computational Potential, Design, and Verification. In Proc. 19th International Conference on DNA Computing and Molecular Programming.
[16]
A. David, K. G. Larsen, A. Legay, M. Mikuăionis, and D. B. Poulsen. 2015. Uppaal SMC Tutorial. Int’l Journal of Software Tools for Technology Transfer (2015).
[17]
C. Daws, M. Kwiatkowska, and G. Norman. 2004. Automatic Verification of the IEEE 1394 Root Contention Protocol with KRONOS and PRISM. International Journal of Network Security and Its Applications 5 (2004).
[18]
Augusto Born De Oliveira, Sebastian Fischmeister, Amer Diwan, Matthias Hauswirth, and Peter F Sweeney. 2013. Why you should care about quantile regression. In Architectural Support for Programming Languages and Operating Systems, ASPLOS ’13. 207–218. http://dl.acm.org/citation.cfm?doid=2451116.2451140
[19]
David L Dill. 1996. The Murphi Verification System. In CAV, Vol. 1102.
[20]
M. Duflot, M. Kwiatkowska, G. Norman, and D. Parker. 2006. A Formal Analysis of Bluetooth Device Discovery. International Journal of Software Tools for Technology Transfer (2006).
[21]
B. Efron and Tibshirani R.J.1993. An introduction to the bootstrap. Chapman and Hall, New York, NY. Farrell, J., Johnston, M. and Twynam, D.(1998),‘‘Volunteer motivation, satisfaction, and management at an elite sporting competition’’, Journal of Sport Management 12 (1993), 288–300.
[22]
M. Elboukhari, A. Azizi, and M. Azizi. 2010. Analysis of the Security of BB84 by Model Checking. Int’l Journal of Network Security and Its Applications 2 (2010).
[23]
L. Feng, C. Wiltsche, L. Humphrey, and U. Topcu. 2015. Controller Synthesis for Autonomous Systems Interacting with Human Operators. In Proceedings of the ACM/IEEE Sixth International Conference on Cyber-Physical Systems.
[24]
M. Fruth. 2011. Formal Methods for the Analysis of Wireless Network Protocols. Ph. D. Dissertation. University of Oxford.
[25]
Andy Georges, Dries Buytaert, and Lieven Eeckhout. 2007. Statistically rigorous java performance evaluation. ACM SIGPLAN Notices 42, 10 (2007), 57–76.
[26]
Jean Dickinson Gibbons and Subhabrata Chakraborti. 2011. Nonparametric Statistical Inference. Springer Berlin Heidelberg, Berlin, Heidelberg, 157–166. https://doi.org/10.1007/978-3-642-04898-2_420
[27]
Louis Guttman. 1977. What is not what in statistics. Journal of the Royal Statistical Society. Series D (The Statistician) 26, 2 (1977), 81–107.
[28]
H. Hansson and B. Jonsson. 1994. A Logic for Reasoning About Time and Reliability. Formal Aspects of Computing 6, 5 (1994).
[29]
A. S. Harji, P. A. Buhr, and T. Brecht. 2011. Our troubles with Linux and why you should care. In Proc. of the Second Asia-Pacific Workshop on Systems. 1–5.
[30]
Samuel Irving, Bin Li, Shaoming Chen, Lu Peng, Weihua Zhang, and Lide Duan. 2020. Computer Comparisons in the Presence of Performance Variation. Frontiers of Computer Science 14, 1 (2020).
[31]
Tomas Kalibera, Lubomir Bulej, and Petr Tuma. 2005. Benchmark precision and random initial state. In Proc. of the 2005 Int’l Symposium on Performance Evaluation of Computer and Telecommunication Systems.
[32]
Tomas Kalibera and Richard Jones. 2020. Quantifying Performance Changes with Effect Size Confidence Intervals. arxiv:2007.10899 [stat.ME]
[33]
Roger Koenker and Kevin F Hallock. 2001. Quantile regression. Journal of economic perspectives 15, 4 (2001), 143–156.
[34]
M. Kwiatkowska and G. Norman. 2002. Verifying Randomized Byzantine Agreement. In Formal Techniques for Networked and Distributed Systems.
[35]
M. Kwiatkowska, G. Norman, and D. Parker. 2005. Probabilistic Model Checking and Power-Aware Computing. In 7th International Workshop on Performability Modeling of Computer and Communication Systems.
[36]
M. Kwiatkowska, G. Norman, and D. Parker. 2006. Controller Dependability Analysis by Probabilistic Model Checking. Control Engineering Practice 15 (2006).
[37]
M. Kwiatkowska, G. Norman, and D. Parker. 2008. Using Probabilistic Model Checking in Systems Biology. ACM SIGMETRICS Performance Evaluation Review 35 (2008).
[38]
M. Kwiatkowska, G. Norman, and D. Parker. 2011. Computer Aided Verification. Springer Berlin Heidelberg, Chapter PRISM 4.0: Verification of Probabilistic Real-time Systems.
[39]
M. Kwiatkowska, G. Norman, and D. Parker. 2012. Probabilistic Verification of Herman’s Self-Stabilisation Algorithm. Formal Aspects of Computing 24 (2012).
[40]
Marta Kwiatkowska, Gethin Norman, and David Parker. 2018. Probabilistic Model Checking: Advances and Applications. In Formal System Verification: State-of the-Art and Future Trends. 73–121.
[41]
A. Legay, B. Delahaye, and S. Bensalem. 2010. Statistical model checking: An overview. In Runtime Verification, Howard Barringer, Ylies Falcone, Bernd Finkbeiner, Klaus Havelund, Insup Lee, Gordon Pace, Grigore Roşu, Oleg Sokolsky, and Nikolai Tillmann (Eds.). Vol. 6418. Springer Berlin Heidelberg.
[42]
J. Lowe-Power, A. M. Ahmad, A. Akram, M. Alian, R. Amslinger, M. Andreozzi, A. Armejach, N. Asmussen, B. Beckmann, S. Bharadwaj, G. Black, G. Bloom, B. R. Bruce, D. Rodrigues Carvalho, J. Castrillon, L. Chen, N. Derumigny, S. Diestelhorst, W. Elsasser, C. Escuin, M. Fariborz, A. Farmahini-Farahani, P. Fotouhi, R. Gambord, J. Gandhi, D. Gope, T. Grass, A. Gutierrez, B. Hanindhito, A. Hansson, S. Haria, A. Harris, T. Hayes, A. Herrera, M. Horsnell, S. Ali Raza Jafri, R. Jagtap, H. Jang, R. Jeyapaul, T. M. Jones, M. Jung, S. Kannoth, H. Khaleghzadeh, Y. Kodama, T. Krishna, T. Marinelli, C. Menard, A. Mondelli, M. Moreto, T. Mück, O. Naji, K. Nathella, H. Nguyen, N. Nikoleris, L. E. Olson, M. Orr, B. Pham, P. Prieto, T. Reddy, A. Roelke, M. Samani, A. Sandberg, J. Setoain, B. Shingarov, M. D. Sinclair, T. Ta, R. Thakur, G. Travaglini, M. Upton, N. Vaish, I. Vougioukas, W. Wang, Z. Wang, N. Wehn, C. Weis, D. A. Wood, H. Yoon, and É. F. Zulian. 2020. The gem5 Simulator: Version 20.0+. arxiv:2007.03152 [cs.AR]
[43]
A. Lungu, P. Bose, D. Sorin, S. German, and G. Janssen. 2009. Multicore Power Management: Ensuring Robustness via Early-Stage Formal Verification. In Seventh ACM-IEEE Int’l Conference on Formal Methods and Models for Codesign.
[44]
Yue Luo and Lizy K. John. 2008. Using Statistical Theory to Study Issues in Microprocessor Simulation. Technical Report TR-040225-01. University of Texas, Department of ECE.
[45]
Oded Maler and Dejan Nickovic. 2004. Monitoring temporal properties of continuous signals. In Formal Techniques, Modelling and Analysis of Timed and Fault-Tolerant Systems. Springer, 152–166.
[46]
G. Norman, D. Parker, M. Kwiatkowska, and S. Shukla. 2005. Evaluating the Reliability of NAND Multiplexing with PRISM. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 24 (2005).
[47]
G. Norman, D. Parker, M. Kwiatkowska, S. Shukla, and R. Gupta. 2003. Using Probabilistic Model Checking for Dynamic Power Management. In 3rd Workshop on Automated Verification of Critical Systems.
[48]
G. Norman and V. Shmatikov. 2006. Analysis of Probabilistic Contract Signing. Journal of Computer Security 14 (2006).
[49]
N. Oswald, V. Nagarajan, D. J. Sorin, V. Gavrielatos, T. Olausson, and R. Carr. 2022. HeteroGen: Automatic Synthesis of Heterogeneous Cache Coherence Protocols. In IEEE Int’l Symposium on High-Performance Computer Architecture.
[50]
A. Pnueli. 1977. The Temporal Logic of Programs. In 18th Annual Symposium on Foundations of Computer Science.
[51]
Kishore Kumar Pusukuri, Rajiv Gupta, and Laxmi N. Bhuyan. 2012. Thread Tranquilizer: Dynamically Reducing Performance Variation. ACM Transactions on Architecture and Code Optimization 8, 4 (2012).
[52]
A. Raghavan, Y. Luo, A. Chandawalla, M. Papaefthymiou, K. P. Pipe, T. F. Wenisch, and M. M. K. Martin. 2012. Computational Sprinting. In Proc. of the 18th Symp. on High Performance Computer Architecture.
[53]
N. Roohi, Y. Wang, M. West, G. E. Dullerud, and M. Viswanathan. 2017. Statistical Verification of the Toyota Powertrain Control Verification Benchmark. In ACM Int’l Conference on Hybrid Systems: Computation and Control.
[54]
Somayeh Sardashti, Andre Seznec, and David A. Wood. 2016. Yet Another Compressed Cache: A Low-Cost Yet Effective Compressed Cache. 13, 3 (2016).
[55]
M. Shahrad, J. Balkind, and D. Wentzlaff. 2019. Architectural Implications of Function-as-a-Service Computing. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Computer Architecture.
[56]
T. Sherwood, E. Perelman, G. Hamerly, and B. Calder. 2002. Automatically Characterizing Large Scale Program Behavior. In Proc. of the Tenth Int’l Conference on Architectural Support for Programming Languages and Operating Systems.
[57]
S. Srinivasan, P.S. Chhabra, P.K. Jaini, A. Aziz, and L. John. 1999. Formal verification of a snoop-based cache coherence protocol using symbolic model checking. In Proceedings of the Twelfth International Conference on VLSI Design.
[58]
T. Tajimi, M. Hayashi, Y. Futamase, R. Shioya, M. Goshima, and T. Tsumura. 2018. Isolation-Safe Speculative Access Control for Hardware Transactional Memory. In 25th IEEE Int’l Conference on Electronics, Circuits and Systems.
[59]
J. Vitek and T. Kalibera. 2011. Repeatability, Reproducibility, and Rigor in Systems Research. In Proc. of the Ninth ACM Int’l Conf. on Embedded Software.
[60]
Y. Wang, M. Zarei, B. Bonakdarpour, and M. Pajic. 2019. Statistical Verification of Hyperproperties for Cyber-Physical Systems. ACM Transactions on Embedded Computing Systems 18 (2019).
[61]
R. E. Wunderlich, T. F. Wenisch, B. Falsafi, and J. C. Hoe. 2003. SMARTS: Accelerating Microarchitecture Simulation via Rigorous Statistical Sampling. In Proc. of the 30th Annual Int’l Symp. on Computer Architecture.
[62]
H. Younes, G. Norman M. Kwiatkowska, and D. Parker. 2006. Numerical vs. Statistical Probabilistic Model Checking. International Journal on Software Tools for Technology Transfer 8 (2006).
[63]
M. Zarei, Y. Wang, and M. Pajic. 2020. Statistical Verification of Learning-Based Cyber-Physical Systems. In ACM International Conference on Hybrid Systems: Computation and Control.
Index Terms
- Rigorous Evaluation of Computer Processors with Statistical Model Checking
Recommendations
Conducting Laboratory Experiments Properly with Statistical Tools: An Easy Hands-On Tutorial
WSDM '19: Proceedings of the Twelfth ACM International Conference on Web Search and Data MiningThis hands-on half-day tutorial consists of two sessions. Part~I covers the following topics: Preliminaries; Paired and two-sample t-tests, confidence intervals; One-way ANOVA and two-way ANOVA without replication; Familiwise error rate. Part~II covers ...
A Survey of Statistical Model Checking
Interactive, distributed, and embedded systems often behave stochastically, for example, when inputs, message delays, or failures conform to a probability distribution. However, reasoning analytically about the behavior of complex stochastic systems is ...
Statistical model checking of distributed real-time actor systems
DS-RT '17: Proceedings of the 21st International Symposium on Distributed Simulation and Real Time ApplicationsThis paper proposes an approach to modelling and analysis of distributed real-time actor systems, which is based on the Uppaal statistical model checker (SMC). SMC was chosen because it automatizes simulations, offers a temporal logic language to ...
Comments
Information & Contributors
Information
Published In
October 2023
1528 pages
ISBN:9798400703294
DOI:10.1145/3613424
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 08 December 2023
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- NSF
Conference
MICRO '23
Sponsor:
MICRO '23: 56th Annual IEEE/ACM International Symposium on Microarchitecture
October 28 - November 1, 2023
ON, Toronto, Canada
Acceptance Rates
Overall Acceptance Rate 484 of 2,242 submissions, 22%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 296Total Downloads
- Downloads (Last 12 months)296
- Downloads (Last 6 weeks)34
Reflects downloads up to 09 Nov 2024
Other Metrics
Citations
Cited By
View all- Mienye IJere N(2024)A Survey of Decision Trees: Concepts, Algorithms, and ApplicationsIEEE Access10.1109/ACCESS.2024.341683812(86716-86727)Online publication date: 2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format