research-article

When Database Meets New Storage Devices: Understanding and Exposing Performance Mismatches via Configurations

Authors:

Xiangke LiaoAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 16, Issue 7

Pages 1712 - 1725

https://doi.org/10.14778/3587136.3587145

Published: 01 March 2023 Publication History

Abstract

NVMe SSD hugely boosts the I/O speed, with up to GB/s throughput and microsecond-level latency. Unfortunately, DBMS users can often find their high-performanced storage devices tend to deliver less-than-expected or even worse performance when compared to their traditional peers. While many works focus on proposing new DBMS designs to fully exploit NVMe SSDs, few systematically study the symptoms, root causes and possible detection methods of such performance mismatches on existing databases.

In this paper, we start with an empirical study where we systematically expose and analyze the performance mismatches on six popular databases via controlled configuration tuning. From the study, we find that all six databases can suffer from performance mismatches. Moreover, we conclude that the root causes can be categorized as databases' unawareness of new storage devices characteristics in I/O size, I/O parallelism and I/O sequentiality. We report 17 mismatches to developers and 15 are confirmed.

Additionally, we realize testing all configuration knobs yields low efficiency. Therefore, we propose a fast performance mismatch detection framework and evaluation shows that our framework brings two orders of magnitude speedup than baseline without sacrificing effectiveness.

References

[1]

ACID (atomicity, consistency, isolation, durability). https://en.wikipedia.org/wiki/ACID. [Accessed Jan. 2023].

[2]

Asynchronous IO for PostgreSQL. anarazel.de/talks/2020-01-31-fosdem-aio/aio. pdf. [Accessed Jan. 2023].

[3]

bcc Python Developer. https://github.com/iovisor/bcc/blob/master/docs/tutorial_bcc_python_developer.md. [Accessed Jan. 2023].

[4]

blktrace. https://linux.die.net/man/8/blktrace/. [Accessed Jan. 2023].

[5]

Changing page size in MySQL. https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_page_size. [Accessed Jan. 2023].

[6]

Changing page size in SQLite. https://sqlite.org/pragma.html#pragma_page_size. [Accessed Jan. 2023].

[7]

Cloud NVMes: the blind side of them. https://ragainis.lt/cloud-nvmes-the-blind-side-of-them-da927d09b378. [Accessed Jan. 2023].

[8]

Control variable. https://en.wikipedia.org/wiki/Control_variable. [Accessed Jan. 2023].

[9]

Dominator. https://en.wikipedia.org/wiki/Dominator_(graph_theory). [Accessed Jan. 2023].

[10]

eBPF: extended berkeley packet filter. https://ebpf.io/. [Accessed Jan. 2023].

[11]

Event Tracing. https://www.kernel.org/doc/html/v4.19/trace/events.html. [Accessed Jan. 2023].

[12]

Glauber Costa. How io_uring Will Revolutionize Programming in Linux. https://thenewstack.io/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/. [Accessed Jan. 2023].

[13]

Intel. Intel^® SSD 760p Series. https://www.intel.com/content/www/us/en/products/sku/134583/intel-ssd-760p-series-256gb-m-2-80mm-pcie-3-1-x4-3d2-tlc/specifications.html. [Accessed Jan. 2023].

[14]

iostat: Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions. https://github.com/sysstat/sysstat/blob/master/man/iostat.in. [Accessed Jan. 2023].

[15]

Jens Axboe. Flexible I/O Tester. https://fio.readthedocs.io/en/latest/index.html. [Accessed Jan. 2023].

[16]

Jonathan Corbet. Ringing in a new asynchronous I/O API. https://lwn.net/Articles/776703/. [Accessed Jan. 2023].

[17]

Linux-native asynchronous I/O library. http://lse.sourceforge.net/io/aio.html. [Accessed Jan. 2023].

[18]

LLVM Programmer's Manual. https://llvm.org/docs/ProgrammersManual.html. [Accessed Jan. 2023].

[19]

LLVM use-def-chains. https://llvm.org/docs/ProgrammersManual.html#iterating-over-def-use-use-def-chains. [Accessed Jan. 2023].

[20]

MariaDB. https://mariadb.org/. [Accessed Jan. 2023].

[21]

MongoDB. https://www.mongodb.com/. [Accessed Jan. 2023].

[22]

MySQL 8.0 is 36 times slower than MySQL 5.7 in Samsung 970 Pro NVMe. https://bugs.mysql.com/bug.php?id=93734/. [Accessed Jan. 2023].

[23]

MySQL. Configuring the Number of Background InnoDB I/O Threads. https://dev.mysql.com/doc/refman/8.0/en/innodb-performance-multiple_io_threads.html. [Accessed Jan. 2023].

[24]

MySQL. Using Asynchronous I/O on linux. https://dev.mysql.com/doc/refman/8.0/en/innodb-linux-native-aio.html. [Accessed Jan. 2023].

[25]

MySQL with NVMe SSD is slower than expected. https://gcore.de/en/help/linux/mysql-nvme-slow.php.

[26]

NVM Express Explained. https://nvmexpress.org/wp-content/uploads/2013/04/NVM_whitepaper.pdf. [Accessed Jan. 2023].

[27]

Oracle Database. https://www.oracle.com/database/. [Accessed Jan. 2023].

[28]

Poor IO performance - NVMe Samsung 950 Pro. https://askubuntu.com/questions/698395/. [Accessed Jan. 2023].

[29]

Postgres slower on NVMe then on Sata SSD. https://www.reddit.com/r/PostgreSQL/comments/5nrha9/postgres_slower_on_nvme_then_on_sata_ssd/. [Accessed Jan. 2023].

[30]

PostgreSQL. Asynchronous I/O for PostgreSQL (Working in Progress). https://github.com/anarazel/postgres/tree/aio. [Accessed Jan. 2023].

[31]

PostgreSQLCO.NF Parameter Documentation. https://postgresqlco.nf/doc/en/param/. [Accessed Jan. 2023].

[32]

Red-9. Database Engine Popularity Rankings. https://red9.com/database-popularity-ranking. [Accessed Jan. 2023].

[33]

Redis. https://redis.io/. [Accessed Jan. 2023].

[34]

Samsung. Samsung 980 Pro NVMe SSD. https://www.samsung.com/semiconductor/minisite/ssd/product/consumer/980pro/. [Accessed Jan. 2023].

[35]

SOLID IT. DB-Engines Ranking. https://db-engines.com/en/ranking. [Accessed Jan. 2023].

[36]

SQLite Is Serverless. https://sqlite.org/serverless.html. [Accessed Jan. 2023].

[37]

SQLite. Performance Related Assumptions. https://www.sqlite.org/fileio.html#fs_performance. [Accessed Jan. 2023].

[38]

TaintChecker: a clang static checker that carries out tainting analysis. https://github.com/franchiotta/taintchecker/. [Accessed Jan. 2023].

[39]

Taintgrind: a valgrind taint analysis tool. https://github.com/wmkhoo/taintgrind. [Accessed Jan. 2023].

[40]

The SQLite OS Interface or "VFS". https://www.sqlite.org/vfs.html.

[41]

Transitive relation. https://en.wikipedia.org/wiki/Transitive_relation. [Accessed Jan. 2023].

[42]

Western Digital. WD_BLACK SN850 NVMe SSD. https://www.westerndigital.com/products/internal-drives/wd-black-sn850-nvme-ssd/. [Accessed Jan. 2023].

[43]

Mona Attariyan, Michael Chow, and Jason Flinn. X-ray: Automating root-cause diagnosis of performance anomalies in production software. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation (OSDI), 2012.

[44]

Oana Balmau, Diego Didona, Rachid Guerraoui, Willy Zwaenepoel, Huapeng Yuan, Aashray Arora, Karan Gupta, and Pavan Konka. TRIAD: creating synergies between memory, disk and log in log structured key-value stores. In 2017 USENIX Annual Technical Conference (ATC), 2017.

[45]

Supratik Chakraborty, Daniel J Fremont, Kuldeep S Meel, Sanjit A Seshia, and Moshe Y Vardi. Distribution-aware sampling and weighted model counting for sat. In Proceedings of the Association for the Advancement of ArtiĄcial Intelligence (AAAI), 2014.

[46]

Feng Chen, Binbing Hou, and Rubao Lee. Internal parallelism of flash memory-based solid-state drives. ACM Trans. Storage, 12(3), 2016.

Digital Library

[47]

Feng Chen, David A. Koufaty, and Xiaodong Zhang. Understanding intrinsic characteristics and system implications of flash memory based solid state drives. In Proceedings of the Eleventh International Joint Conference on Measurement and Modeling of Computer Systems (SIGMETRICS), 2009.

Digital Library

[48]

Qingrong Chen, Teng Wang, Owolabi Legunsen, Shanshan Li, and Tianyin Xu. Understanding and discovering software configuration dependencies in cloud and datacenter systems. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 362--374, 2020.

Digital Library

[49]

Alexander Conway, Abhishek Gupta, Vijay Chidambaram, Martin Farach-Colton, Richard P. Spillane, Amy Tai, and Rob Johnson. Splinterdb: Closing the bandwidth gap for nvme key-value stores. In 2020 USENIX Annual Technical Conference (ATC), 2020.

[50]

Keith D Cooper, Timothy J Harvey, and Ken Kennedy. A simple, fast dominance algorithm. Software Practice & Experience, 4, 2001.

[51]

Monika Dhok and Murali Krishna Ramanathan. Directed test generation to detect loop inefficiencies. In European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE), 2016.

Digital Library

[52]

Pedram Ghodsnia, Ivan T. Bowman, and Anisoara Nica. Parallel I/O aware query optimization. In International Conference on Management of Data (SIGMOD), 2014.

[53]

X. Guo, X. Liu, E. Zhu, X. Zhu, M. Li, X. Xu, and J. Yin. Adaptive self-paced deep clustering with data augmentation. IEEE Transactions on Knowledge and Data Engineering, 2019.

[54]

Sibsankar Haldar. SQLite Database System Design and Implementation. Sibsankar Haldar, 2016. [Accessed Jan. 2023].

[55]

Xue Han, Tingting Yu, and David Lo. Perflearner: Learning from bug reports to understand and generate performance test frames. In International Conference on Automated Software Engineering (ASE), 2018.

Digital Library

[56]

Haochen He, Zhouyang Jia, Shanshan Li, Erci Xu, Tingting Yu, Yue Yu, Ji Wang, and Xiangke Liao. Cp-detector: Using configuration-related performance properties to expose performance bugs. In 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2020.

[57]

Jun He, Sudarsun Kannan, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. The unwritten contract of solid state drives. In Proceedings of the Twelfth European Conference on Computer Systems (EuroSys), 2017.

Digital Library

[58]

Sen He, Glenna Manns, John Saunders, Wei Wang, Lori Pollock, and Mary Lou Soffa. A statistics-based performance testing methodology for cloud applications. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE), 2019.

Digital Library

[59]

Yang Hu, Hong Jiang, Dan Feng, Lei Tian, Hao Luo, and Chao Ren. Exploring and exploiting the multilevel parallelism inside ssds for improved performance and endurance. IEEE Trans. Computers, 62(6), 2013.

Digital Library

[60]

Yigong Hu, Gongqi Huang, and Peng Huang. Automated reasoning and detection of specious configuration in large systems with symbolic execution. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI), 2020.

Digital Library

[61]

Jaehyun Hwang, Midhul Vuppalapati, Simon Peter, and Rachit Agarwal. Rearchitecting linux storage stack for μs latency and high throughput. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2021.

[62]

Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. Understanding and detecting real-world performance bugs. In Conference on Programming Language Design and Implementation (PLDI), 2012.

Digital Library

[63]

M. F. Johansen, Ø. Haugen, and F. Fleurey. An algorithm for generating t-wise covering arrays from large feature models. In Proceedings of the International Software Product Line Conference (SPLC), 2012.

Digital Library

[64]

Aarati Kakaraparthy, Jignesh M. Patel, Kwanghyun Park, and Brian Kroth. Optimizing databases by learning hidden parameters of solid state drives. Proc. VLDB Endow., 13(4), 2019.

Digital Library

[65]

Christian Kaltenecker, Alexander Grebhahn, Norbert Siegmund, Jianmei Guo, and Sven Apel. Distance-based sampling of software configuration spaces. In IEEE/ACM 41st International Conference on Software Engineering (ICSE), 2019.

Digital Library

[66]

Christian Kaltenecker, Alexander Grebhahn, Norbert Siegmund, Jianmei Guo, and Sven Apel. Distance-based sampling of software configuration spaces. In Proceedings of the 41st International Conference on Software Engineering (ICSE), 2019.

Digital Library

[67]

Ali Khakifirooz, Sriram Balasubrahmanyam, Richard Fastow, Kristopher H Gaewsky, Chang Wan Ha, Rezaul Haque, Owen W Jungroth, Steven Law, Aliasgar S Madraswala, Binh Ngo, et al. A 1Tb 4b/Cell 144-Tier Floating-Gate 3D-NAND Flash Memory with 40MB/s Program Throughput and 13.8 Gb/mm 2 Bit Density. In Proceedings of the 68th IEEE International Solid-State Circuits Conference (ISSCC), 2021.

[68]

Chulbum Kim, Doo-Hyun Kim, Woopyo Jeong, Hyun-Jin Kim, Il Han Park, Hyun-Wook Park, JongHoon Lee, JiYoon Park, Yang-Lo Ahn, Ji Young Lee, et al. A 512-Gb 3-b/Cell 64-stacked WL 3-D-NAND flash memory. IEEE Journal of Solid-State Circuits, 53(1), 2017.

[69]

Baptiste Lepers, Oana Balmau, Karan Gupta, and Willy Zwaenepoel. Kvell: the design and implementation of a fast persistent key-value store. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP), 2019.

Digital Library

[70]

Chi Li, Shu Wang, Henry Hoffmann, and Shan Lu. Statically inferring performance properties of software configurations. In Proceedings of the Fifteenth European Conference on Computer Systems (EuroSys), 2020.

Digital Library

[71]

Chi Li, Shu Wang, Henry Hoffmann, and Shan Lu. Statically inferring performance properties of software configurations. In Fifteenth EuroSys Conference (EuroSys), 2020.

Digital Library

[72]

Yinan Li, Bingsheng He, Jun Yang, Qiong Luo, and Ke Yi. Tree indexing on solid state drives. Proc. VLDB Endow., 3(1), 2010.

Digital Library

[73]

Xiaojian Liao, Youyou Lu, Erci Xu, and Jiwu Shu. Write dependency disentanglement with horae. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI), 2020.

Digital Library

[74]

Xiaojian Liao, Youyou Lu, Zhe Yang, and Jiwu Shu. Crash consistent non-volatile memory express. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (SOSP), 2021.

Digital Library

[75]

Aleksander Maricq, Dmitry Duplyakin, Ivo Jimenez, Carlos Maltzahn, Ryan Stutsman, and Robert Ricci. Taming performance variability. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation (OSDI), 2018.

Digital Library

[76]

D. Marijan, A. Gotlieb, S. Sen, and A. Hervieu. Practical pairwise testing for software product lines. In Proceedings of the International Software Product Line Conference (SPLC), 2013.

Digital Library

[77]

Changwoo Min, Kangnyeon Kim, Hyunjin Cho, Sang-Won Lee, and Young Ik Eom. SFS: random write considered harmful in solid state drives. In Proceedings of the 10th USENIX conference on File and Storage Technologies (FAST), 2012.

[78]

Adrian Nistor, Po-Chun Chang, Cosmin Radoi, and Shan Lu. Caramel: Detecting and fixing performance problems that have non-intrusive fixes. In International Conference on Software Engineering (ICSE), 2015.

[79]

OREILLY. DBWR---the Database Writer. https://www.oreilly.com/library/view/oracle-database-administration/1565925165/ch10s02s01s01.html. [Accessed Jan. 2023].

[80]

Anastasios Papagiannis, Giorgos Saloustros, Pilar González-Férez, and Angelos Bilas. Tucana: Design and implementation of a fast and efficient scale-up key-value store. In 2016 USENIX Annual Technical Conference (ATC), 2016.

[81]

Michael Pradel, Markus Huggler, and Thomas R. Gross. Performance regression testing of concurrent classes. In International Symposium on Software Testing & Analysis (ISSTA), 2014.

Digital Library

[82]

Pandian Raju, Rohan Kadekodi, Vijay Chidambaram, and Ittai Abraham. Pebblesdb: Building key-value stores using fragmented log-structured merge trees. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP), 2017.

Digital Library

[83]

Hongchan Roh, Sanghyun Park, Sungho Kim, Mincheol Shin, and Sang-Won Lee. B+-tree index optimization by exploiting internal parallelism of flash-based solid state drives. Proc. VLDB Endow., 5(4), 2011.

Digital Library

[84]

Jason Sewall, Jatin Chhugani, Changkyu Kim, Nadathur Satish, and Pradeep Dubey. PALM: parallel architecture-friendly latch-free modifications to B+ trees on many-core processors. Proc. VLDB Endow., 4(11), 2011.

Digital Library

[85]

Pengfei Su, Shasha Wen, Hailong Yang, Milind Chabbi, and Xu Liu. Redundant loads: A software inefficiency indicator. In International Conference on Software Engineering (ICSE), 2019.

[86]

TunLi, WanweiLiu, JuanChen, XiaoguangMao, and XinjunMao. Towards connecting discrete mathematics and software engineering. TSINGHUA SCIENCE AND TECHNOLOGY, 25(3), 2020.

[87]

Alexandru Uta, Alexandru Custura, Dmitry Duplyakin, Ivo Jimenez, Jan Rellermeyer, Carlos Maltzahn, Robert Ricci, and Alexandru Iosup. Is big data performance reproducible in modern cloud networks? In 17th USENIX symposium on networked systems design and implementation (NSDI), 2020.

[88]

Dana Van Aken, Andrew Pavlo, Geoffrey J Gordon, and Bohan Zhang. Automatic database management system tuning through large-scale machine learning. In Proceedings of the 2017 ACM international conference on management of data (SIGMOD), 2017.

Digital Library

[89]

Teng Wang, Zhouyang Jia, Shanshan Li, Si Zheng, Yue Yu, Erci Xu, Shaoliang Peng, and Xiangke Liao. Understanding and detecting on-the-fly configuration bugs. In Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023.

[90]

Youjip Won, Jaemin Jung, Gyeongyeol Choi, Joontaek Oh, Seongbae Son, Joo Young Hwang, and Sangyeun Cho. Barrier-enabled io stack for flash storage. In USENIX Conference on File and Storage Technologies (FAST), 2018.

[91]

Qiumin Xu, Huzefa Siyamwala, Mrinmoy Ghosh, Tameesh Suri, Manu Awasthi, Zvika Guz, Anahita Shayesteh, and Vijay Balakrishnan. Performance analysis of nvme ssds and their implication on real world databases. In Proceedings of the 8th ACM International Systems and Storage Conference (SYSTOR), 2015.

Digital Library

[92]

Tianyin Xu, Jiaqi Zhang, Peng Huang, Jing Zheng, Tianwei Sheng, Ding Yuan, Yuanyuan Zhou, and Shankar Pasupathy. Do not blame users for misconfigurations. In ACM SIGOPS 24th Symposium on Operating Systems Principles (SOSP), 2013.

Digital Library

[93]

Tingting Yu and Michael Pradel. Syncprof: Detecting, localizing, and optimizing synchronization bottlenecks. In International Symposium on Software Testing & Analysis (ISSTA), 2016.

Digital Library

[94]

Bohan Zhang, Dana Van Aken, Justin Wang, Tao Dai, Shuli Jiang, Jacky Lao, Siyuan Sheng, Andrew Pavlo, and Geoffrey J. Gordon. A demonstration of the ottertune automatic database management system tuning service. Proc. VLDB Endow., 11(12), 2018.

Digital Library

[95]

Ji Zhang, Yu Liu, Ke Zhou, Guoliang Li, Zhili Xiao, Bin Cheng, Jiashu Xing, Yangtao Wang, Tianheng Cheng, Li Liu, et al. An end-to-end automatic cloud database tuning system using deep reinforcement learning. In Proceedings of the 2019 International Conference on Management of Data (SIGMOD), 2019.

Digital Library

[96]

Shulin Zhou, Xiaodong Liu, Shanshan Li, Wei Dong, Xiangke Liao, and Yun Xiong. Confmapper: Automated variable finding for configuration items in source code. In 2016 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), 2016.

[97]

Yuqing Zhu, Jianxun Liu, Mengying Guo, Yungang Bao, Wenlong Ma, Zhuoyue Liu, Kunpeng Song, and Yingchun Yang. Bestconfig: tapping the performance potential of systems via automatic configuration tuning. In Proceedings of the 2017 Symposium on Cloud Computing (SoCC), 2017.

Digital Library

Cited By

Xu JQiu YChen YWang YLin WLin YZhao SLiu YWang YChen W(2024)Performance Characterization of SmartNIC NVMe-over-Fabrics Target OffloadingProceedings of the 17th ACM International Systems and Storage Conference10.1145/3688351.3689154(14-24)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3688351.3689154

Recommendations

Adapting database storage for new hardware
Understanding the Impacts of Solid-State Storage on the Hadoop Performance
CBD '13: Proceedings of the 2013 International Conference on Advanced Cloud and Big Data

The superior I/O performance of solid-state storage (e.g., solid-state drives) makes it become an attractive replacement for the traditional magnetic storage (e.g., harddisk drives). More and more storage systems start to integrate solid-state storage ...
Modeling and performance of MEMS-based storage devices
Special issue on proceedings of ACM SIGMETRICS 2000

MEMS-based storage devices are seen by many as promising alternatives to disk drives. Fabricated using conventional CMOS processes, MEMS-based storage consists of thousands of small, mechanical probe tips that access gigabytes of high-density, ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 16, Issue 7

March 2023

203 pages

ISSN:2150-8097

Editors:
Georgia Koutrika
Athena Research Center
,
Jun Yang
Duke University

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 March 2023

Published in PVLDB Volume 16, Issue 7

Check for updates

Badges

Artifacts Available / v1.1

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
169
Total Downloads

Downloads (Last 12 months)132
Downloads (Last 6 weeks)11

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xu JQiu YChen YWang YLin WLin YZhao SLiu YWang YChen W(2024)Performance Characterization of SmartNIC NVMe-over-Fabrics Target OffloadingProceedings of the 17th ACM International Systems and Storage Conference10.1145/3688351.3689154(14-24)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3688351.3689154

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents