research-article

Toward Virtual Machine Image Management for Persistent Memory

Authors:

Xiaoguang Liu, and

Gang WangAuthors Info & Claims

ACM Transactions on Storage (TOS), Volume 17, Issue 3

Article No.: 20, Pages 1 - 24

https://doi.org/10.1145/3450976

Published: 16 August 2021 Publication History

Abstract

Persistent memory’s (PM) byte-addressability and high capacity will also make it emerging for virtualized environment. Modern virtual machine monitors virtualize PM using either I/O virtualization or memory virtualization. However, I/O virtualization will sacrifice PM’s byte-addressability, and memory virtualization does not get the chance of PM image management. In this article, we enhance QEMU’s memory virtualization mechanism. The enhanced system can achieve both PM’s byte-addressability inside virtual machines and PM image management outside the virtual machines. We also design pcow, a virtual machine image format for PM, which is compatible with our enhanced memory virtualization and supports storage virtualization features including thin-provisioning, base image, snapshot, and striping. Address translation is performed with the help of the Extended Page Table, thus much faster than image formats implemented in I/O virtualization. We also optimize pcow considering PM’s characteristics. We perform exhaustive performance evaluations on an x86 server equipping with Intel’s Optane DC persistent memory. The evaluation demonstrates that our scheme boosts the overall performance by up to 50× compared with qcow2, an image format implemented in I/O virtualization, and brings almost no performance overhead compared with the native memory virtualization. The striping feature can also scale-out the virtual PM’s bandwidth performance.

References

[1]

Redis Labs. 2009. redis. Retrieved from https://redis.io/.

[2]

Redis Labs. 2009. redis-benchmark. Retrieved from https://redis.io/topics/benchmarks.

[3]

VMware. 2011. Virtual Disk Format 5.0. Retrieved from https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf.

[4]

Jens Axboe, et al. 2014. Flexible I/O Tester. Retrieved from https://github.com/axboe/fio.

[5]

Intel. 2014. Persistent Memory Development Kit. Retrieved from https://pmem.io/pmdk/.

[6]

Intel. 2017. BTT - Block Translation Table, Linux Kernel Documentation. Retrieved from https://www.kernel.org/doc/Documentation/nvdimm/btt.txt.

[7]

Kevin Wolf, et al. 2017. Qcow2 Image Format, QEMU Documentation. Retrieved from https://github.com/qemu/qemu/blob/master/docs/interop/qcow2.txt.

[8]

VMware. 2018. PMem (Persistant Memory) NVDIMM support in vSphere. Retrieved from https://storagehub.vmware.com/t/vsphere-storage/vsphere-6-7-core-storage-1/pmem-persistant-memory-nvdimm-support-in-vsphere/.

[9]

Intel. 2018. pmem-redis. Retrieved from https://github.com/pmem/pmem-redis.

[10]

Intel. 2018. Using Persistent Memory Devices with the Linux Device Mapper. Retrieved from https://pmem.io/2018/05/15/using_persistent_memory_devices_with_the_linux_device_mapper.html.

[11]

Intel. 2019. Intel Optane DC Persistent Memory. Retrieved from https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html.

[12]

Avi Kivity, Anthony Liguori, et al. 2019. The memory API, QEMU Documentation. Retrieved from https://github.com/qemu/qemu/blob/master/docs/devel/memory.rst.

[13]

Pankaj Gupta. 2019. Virtio-pmem, patchwork. Retrieved from https://patchwork.kernel.org/cover/11010055.

[14]

Dmytro Apalkov, Alexey Khvalkovskiy, Steven Watts, Vladimir Nikitin, Xueti Tang, Daniel Lottis, Kiseok Moon, Xiao Luo, Eugene Chen, Adrian Ong et al. 2013. Spin-transfer torque magnetic random access memory (STT-MRAM). ACM J. Emerg. Technol. Comput. Syst. 9, 2 (2013), 13.

Digital Library

[15]

Qingshu Chen, Liang Liang, Yubin Xia, Haibo Chen, and Hyunsoo Kim. 2016. Mitigating sync amplification for copy-on-write virtual disk. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST’16). 241–247.

Digital Library

[16]

Deukyeon Hwang, Wook-Hee Kim, Youjip Won, and Beomseok Nam. 2018. Endurable transient inconsistency in byte-addressable persistent B+-tree. In Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST). 187.

Digital Library

[17]

Arpit Joshi, Vijay Nagarajan, Marcelo Cintra, and Stratis Viglas. 2018. DHTM: Durable hardware transactional memory. In Proceedings of the International Symposium on Computer Architecture.

Digital Library

[18]

Sudarsun Kannan, Nitish Bhat, Ada Gavrilovska, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. 2018. Redesigning LSMs for nonvolatile memory with NoveLSM. In Proceedings of the USENIX Annual Technical Conference (ATC’18). 993–1005.

Digital Library

[19]

Liang Liang, Rong Chen, Haibo Chen, Yubin Xia, KwanJong Park, Binyu Zang, and Haibing Guan. 2016. A case for virtualizing persistent memory. In Proceedings of the ACM Symposium on Cloud Computing (SOCC’16). 126–140.

Digital Library

[20]

Youyou Lu, Jiwu Shu, Long Sun, and Onur Mutlu. 2014. Loose-ordering consistency for persistent memory. In Proceedings of the 32nd International Conference on Computer Design (ICCD’14). IEEE, 216–223.

[21]

Richard McDougall and Jennifer Anderson. 2010. Virtualization performance: Perspectives and challenges ahead. ACM SIGOPS Oper. Syst. Rev. 44, 4 (2010), 40–56.

Digital Library

[22]

Timothy Merrifield and H. Reza Taheri. 2016. Performance implications of extended page tables on virtualized x86 processors. ACM SIGPLAN Notices 51, 7 (2016), 25–35.

Digital Library

[23]

Bo Peng, Haozhong Zhang, Jianguo Yao, Yaozu Dong, Yu Xu, and Haibing Guan. 2018. MDev-NVMe: A NVMe storage virtualization solution with mediated pass-through. In Proceedings of the USENIX Annual Technical Conference (ATC’18). 665–676.

Digital Library

[24]

Thanumalayan Sankaranarayana Pillai, Vijay Chidambaram, Ramnatthan Alagappan, Samer Al-Kiswany, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2015. Crash consistency. Queue 13, 7 (2015), 20.

Digital Library

[25]

Simone Raoux, Geoffrey W. Burr, Matthew J. Breitwisch, Charles T. Rettner, Y.-C. Chen, Robert M. Shelby, Martin Salinga, Daniel Krebs, S.-H. Chen, H.-L. Lung, et al. 2008. Phase-change random access memory: A scalable technology. IBM J. Res. Dev. 52, 4.5 (2008), 465–479.

Digital Library

[26]

Andy Rudoff. 2017. Persistent memory programming. Login: Usenix Mag. 42 (2017), 34–40.

[27]

Rusty Russell. 2008. virtio: Towards a de-facto standard for virtual I/O devices. ACM SIGOPS Oper. Syst. Rev. 42, 5 (2008), 95–103.

Digital Library

[28]

SNIA NVM Programming Technical Working Group. 2017. NVM Programming Model (Version 1.2).

[29]

Xiang Song, Jian Yang, and Haibo Chen. 2014. Architecting flash-based solid-state drive for high-performance I/O virtualization. Comput. Architect. Lett. 13, 2 (2014), 61–64.

Digital Library

[30]

Chunqiang Tang. 2011. FVD: A high-performance virtual machine image format for cloud. In Proceedings of the USENIX Annual Technical Conference (ATC’11). 16–24.

Digital Library

[31]

Zev Weiss, Sriram Subramanian, Swaminathan Sundararaman, Nisha Talagala, Andrea C. Arpaci-Dusseau, and Remzi H Arpaci-Dusseau. 2015. ANViL: Advanced virtualization for modern non-volatile memory devices. In Proceedings of the 13rd USENIX Conference on File and Storage Technologies (FAST’15). 111–118.

Digital Library

[32]

Cong Xu, Dimin Niu, Naveen Muralimanohar, Rajeev Balasubramonian, Tao Zhang, Shimeng Yu, and Yuan Xie. 2015. Overcoming the challenges of crossbar resistive memory architectures. In Proceedings of the 21st International Symposium on High Performance Computer Architecture (HPCA’15). IEEE, 476–488.

[33]

Jian Xu, Lu Zhang, Amirsaman Memaripour, Akshatha Gangadharaiah, Amit Borase, Tamires Brito Da Silva, Steven Swanson, and Andy Rudoff. 2017. NOVA-fortis: A fault-tolerant non-volatile main memory file system. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP’17). 478–496.

Digital Library

[34]

Jiwei Xu, Wenbo Zhang, Zhenyu Zhang, Tao Wang, and Tao Huang. 2016. Clustering-based acceleration for virtual machine image deduplication in the cloud environment. J. Syst. Softw. 121 (2016), 144–156.

Digital Library

[35]

J. Zhang, L. Cui, P. Li, X. Liu, and G. Wang. 2019. Towards virtual machine image management for persistent memory. In Proceedings of the 35th Symposium on Mass Storage Systems and Technologies (MSST’19). 116–125.

[36]

Jiachen Zhang, Peng Li, Bo Liu, Trent G. Marbach, Xiaoguang Liu, and Gang Wang. 2018. Performance analysis of 3D XPoint SSDs in virtualized and non-virtualized environments. In Proceedings of the 24th International Conference on Parallel and Distributed Systems (ICPADS’18). IEEE, 51–60.

[37]

Yiying Zhang and Steven Swanson. 2015. A study of application performance with non-volatile main memory. In Proceedings of the 31st Symposium on Mass Storage Systems and Technologies (MSST’15). IEEE, 1–10.

[38]

Pengfei Zuo and Yu Hua. 2017. A write-friendly hashing scheme for non-volatile memory systems. In Proceedings of the 33rd Symposium on Mass Storage Systems and Technologies (MSST’17).

Cited By

Liu YRen YLiu MLi HGuo HMiao XHu XChen HMa XWon Y(2024)Optimizing file systems on heterogeneous memory by integrating DRAM cache with virtual memory managementProceedings of the 22nd USENIX Conference on File and Storage Technologies10.5555/3650697.3650702(71-88)Online publication date: 27-Feb-2024
https://dl.acm.org/doi/10.5555/3650697.3650702
Sheng BHan RXiao FGuo ZGui L(2024)MetaFormerProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435508:1(1-27)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643550
Wang FLv YZhu MDing HHan J(2024)XRF55Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435438:1(1-34)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643543
Show More Cited By

Index Terms

Toward Virtual Machine Image Management for Persistent Memory

Recommendations

A Case for Virtualizing Persistent Memory
SoCC '16: Proceedings of the Seventh ACM Symposium on Cloud Computing

With the proliferation of software and hardware support for persistent memory (PM) like PCM and NV-DIMM, we envision that PM will soon become a standard component of commodity cloud, especially for those applications demanding high performance and low ...
Read More
Efficient memory virtualization for Cross-ISA system mode emulation
VEE '14

Cross-ISA system-mode emulation has many important applications. For example, Cross-ISA system-mode emulation helps computer architects and OS developers trace and debug kernel execution-flow efficiently by emulating a slower platform (such as ARM) on a ...
Read More
Efficient memory virtualization for Cross-ISA system mode emulation
VEE '14: Proceedings of the 10th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments

Cross-ISA system-mode emulation has many important applications. For example, Cross-ISA system-mode emulation helps computer architects and OS developers trace and debug kernel execution-flow efficiently by emulating a slower platform (such as ARM) on a ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Storage

ACM Transactions on Storage Volume 17, Issue 3

August 2021

227 pages

ISSN:1553-3077

EISSN:1553-3093

DOI:10.1145/3477268

Editor:
Sam H. Noh
Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea

Issue’s Table of Contents

Copyright © 2021 Association for Computing Machinery.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 August 2021

Accepted: 01 February 2021

Revised: 01 November 2020

Received: 01 July 2020

Published in TOS Volume 17, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Science Foundation of China
Science and Technology Development Plan of Tianjin
Fundamental Research Funds for the Central Universities
TKLNDST

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
216
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)2

Other Metrics

View Author Metrics

Citations

Cited By

Liu YRen YLiu MLi HGuo HMiao XHu XChen HMa XWon Y(2024)Optimizing file systems on heterogeneous memory by integrating DRAM cache with virtual memory managementProceedings of the 22nd USENIX Conference on File and Storage Technologies10.5555/3650697.3650702(71-88)Online publication date: 27-Feb-2024
https://dl.acm.org/doi/10.5555/3650697.3650702
Sheng BHan RXiao FGuo ZGui L(2024)MetaFormerProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435508:1(1-27)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643550
Wang FLv YZhu MDing HHan J(2024)XRF55Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435438:1(1-34)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643543
Liao PWang XAn LMao SZhao TYang C(2024)TFSemantic: A Time–Frequency Semantic GAN Framework for Imbalanced Classification Using Radio SignalsACM Transactions on Sensor Networks10.1145/361409620:4(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3614096
Cai MJiang XShen JYe B(2024)SplitDB: Closing the Performance Gap for LSM-Tree-Based Key-Value StoresIEEE Transactions on Computers10.1109/TC.2023.332698273:1(206-220)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TC.2023.3326982
Yao TLi YLi YZhu YWang GYue J(2023)Cross-modal Semantically Augmented Network for Image-text MatchingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363135620:4(1-18)Online publication date: 11-Dec-2023
https://dl.acm.org/doi/10.1145/3631356
Zhang YZhang X(2023)Incentive Mechanism with Task Bundling for Mobile Crowd SensingACM Transactions on Sensor Networks10.1145/358178819:3(1-23)Online publication date: 17-Apr-2023
https://dl.acm.org/doi/10.1145/3581788
Zhao MTang HXie PDai SSebe NWang W(2023)Bidirectional Transformer GAN for Long-term Human Motion PredictionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/357935919:5(1-19)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3579359
Lin LDeng YZhou YZhu Y(2023)InDe: An Inline Data Deduplication Approach via Adaptive Detection of Valid Container UtilizationACM Transactions on Storage10.1145/356842619:1(1-27)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3568426
Ahmed ULin JSrivastava G(2023)Deep Hierarchical Attention Active Learning for Mental Disorder Unlabeled Data in AIoMTACM Transactions on Sensor Networks10.1145/351930419:3(1-18)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1145/3519304
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents