Abstract
The use of CXL-based memory expanders introduces increased latency compared to local memory due to control and transmission overheads. This latency difference negatively impacts tasks that are sensitive to latency. While cache prefetching has traditionally been used to mitigate memory latency, addressing this performance gap requires improved CPU prefetch coverage. However, tuning a CPU prefetcher for CXL memory necessitates costly CPU modifications and can result in cache pollution and wasted memory bandwidth. To address these challenges, we propose a solution called Polaris, a novel CXL memory expander that integrates a hardware prefetcher in the CXL memory controller chip. Polaris analyzes incoming memory requests and prefetches cachelines to a dedicated SRAM buffer without requiring modifications to CPUs or software. In cases where prefetch hits occur, Polaris establishes a “shortcut” for rapid memory access, significantly reducing the performance gap between CXL and local DDR memory. Furthermore, if small CPU changes are allowed, such as extending Intel’s DDIO, Polaris can further minimize CXL memory access overheads by actively pushing high-confidence prefetches to the CPU’s last-level cache (LLC). Extensive experiments demonstrate that, in conjunction with various CPU-side prefetchers, Polaris enables up to 85% of common workloads (on average, 43%) to effectively tolerate CXL memory’s longer latency.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We say the CXL latency is “effectively tolerated” if the performance gap between CXL and local memory is within 5%.
References
Parsec 2.1, 2022.9. https://parsec.cs.princeton.edu/
Pythia’s github repo, 2022.9. https://github.com/CMU-SAFARI/Pythia
Aguilera, M.K., et al.: Remote regions: a simple abstraction for remote memory. In: 2018 USENIX Annual Technical Conference (USENIX ATC 18), pp. 775–787 (2018)
Al Maruf, H., Chowdhury, M.: Effectively prefetching remote memory with leap. In: 2020 USENIX Annual Technical Conference (USENIX ATC 20), pp. 843–857 (2020)
Amaro, E., et al.: Can far memory improve job throughput? In: Proceedings of the Fifteenth European Conference on Computer Systems, pp. 1–16 (2020)
Bakhshalipour, M., Lotfi-Kamran, P., Sarbazi-Azad, H.: Domino temporal data prefetcher. In: 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 131–142. IEEE (2018)
Bakhshalipour, M., Shakerinava, M., Lotfi-Kamran, P., Sarbazi-Azad, H.: Bingo spatial data prefetcher. In: 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 399–411. IEEE (2019)
Beamer, S., Asanović, K., Patterson, D.: The gap benchmark suite, arXiv preprint arXiv:1508.03619 (2015)
Bera, R., et al.: Hermes: accelerating long-latency load requests via perceptron-based off-chip load prediction. In: 55th IEEE/ACM International Symposium on Microarchitecture, MICRO 2022, Chicago, IL, USA, 1–5 October 2022. IEEE, 2022, pp. 1–18 (2022). https://doi.org/10.1109/MICRO56248.2022.00015
Bera, R., Kanellopoulos, K., Nori, A., Shahroodi, T., Subramoney, S., Mutlu, O.: Pythia: a customizable hardware prefetching framework using online reinforcement learning. In: MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, 2021, pp. 1121–1137 (2021)
Bera, R., Nori, A.V., Mutlu, O., Subramoney, S.: Dspatch: dual spatial pattern prefetcher. In: Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, 2019, pp. 531–544 (2019)
Bhatia, E., Chacon, G., Pugsley, S., Teran, E., Gratz, P.V., Jiménez, D.A.: Perceptron-based prefetch filtering. In: 2019 ACM/IEEE 46th Annual International Symposium on Computer Architecture (ISCA), pp. 1–13. IEEE (2019)
Bloom, B.H.: Space/time trade-offs in hash coding with allowable errors. Commun. ACM 13(7), 422–426 (1970). https://doi.org/10.1145/362686.362692
Bucek, J., Lange, K.-D., Kistowski, J.V.: SPEC CPU2017: next-generation compute benchmark. In: Companion of the 2018 ACM/SPEC International Conference on Performance Engineering, pp. 41–42 (2018)
Calciu, I., et al.: Rethinking software runtimes for disaggregated memory. In: Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 79–92 (2021)
ChampSim, Champsim simulator, 2022.9. https://github.com/ChampSim/ChampSim
C. foundation, Cxl 3.0 specification, 2022.9. https://www.computeexpresslink.org/download-the-specification
Gao, Y., et al.: When cloud storage meets \(\{RDMA\}\). In: 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pp. 519–533 (2021)
Gouk, D., Lee, S., Kwon, M., Jung, M.: Direct access,\(\{High-Performance\}\) memory disaggregation with \(\{DirectCXL\}\). In: 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp. 287–294 (2022)
Gu, J., Lee, Y., Zhang, Y., Chowdhury, M., Shin, K.G.: Efficient memory disaggregation with infiniswap. In: 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pp. 649–667 (2017)
Henning, J.L.: SPEC CPU2006 benchmark descriptions. ACM SIGARCH Comput. Archit. News 34(4), 1–17 (2006)
Huggahalli, R., Iyer, R., Tetrick, S.: Direct cache access for high bandwidth network i/o. In: 32nd International Symposium on Computer Architecture (ISCA’05), pp. 50–59. IEEE (2005)
Hynix, S.: Sk hynix cxl memory expander, 2022.9. https://news.skhynix.com/sk-hynix-develops-ddr5-dram-cxltm-memory-to-expand-the-cxl-memory-ecosystem/
Intel, Intel data-direct io, 2022.9. https://www.intel.cn/content/www/cn/zh/io/data-direct-i-o-technology.html
Kim, J., Pugsley, S.H., Gratz, P.V., Reddy, A.N., Wilkerson, C., Chishti, Z.: Path confidence based lookahead prefetching. In: 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 1–12. IEEE (2016)
Kumar, A., Huggahalli, R., Makineni, S.: Characterization of direct cache access on multi-core systems and 10gbe. In: 2009 IEEE 15th International Symposium on High Performance Computer Architecture, pp. 341–352. IEEE (2009)
León, E.A., Ferreira, K.B., Maccabe, A.B.: Reducing the impact of the memorywall for I/O using cache injection. In: 2007 15th Annual IEEE Symposium on High-Performance Interconnects (HOTI), pp. 143–150. IEEE (2007)
Li, H., et al.: First-generation memory disaggregation for cloud platforms, arXiv preprint arXiv:2203.00241 (2022)
Lim, K., Chang, J., Mudge, T., Ranganathan, P., Reinhardt, S.K., Wenisch, T.F.: Disaggregated memory for expansion and sharing in blade servers. ACM SIGARCH Comput. Archit. News 37(3), 267–278 (2009)
Maruf, H.A., et al.: TPP: transparent page placement for cxl-enabled tiered memory, arXiv preprint arXiv:2206.02878 (2022)
Michaud, P.: Best-offset hardware prefetching. In: 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 469–480 (2016)
Muralimanohar, N., Balasubramonian, R., Jouppi, N.P.: Cacti 6.0: a tool to model large caches. HP Lab. 27, 28 (2009)
Nassif, N., et al.: Sapphire rapids: the next-generation intel Xeon scalable processor. In: 2022 IEEE International Solid-State Circuits Conference (ISSCC), vol. 65, pp. 44–46. IEEE (2022)
NCSU, Freepdk45. https://www.eda.ncsu.edu/wiki/FreePDK45:Contents
OpenXiangShan, Xiangshan riscv cpu, 2022.9. https://github.com/OpenXiangShan/XiangShan
Pugsley, S.H., et al.: Sandbox prefetching: safe run-time evaluation of aggressive prefetchers. In: IEEE 20th International Symposium on High Performance Computer Architecture (HPCA), pp. 626–637. IEEE (2014)
Ruan, Z., Schwarzkopf, M., Aguilera, M.K., Belay, A.: \(\{AIFM\}\):\(\{High-Performance\}\),\(\{Application-Integrated\}\) far memory. In: 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 315–332 (2020)
Samsung, Smdk, 2022.9. https://github.com/OpenMPDK/SMDK.git
Shakerinava, M., Bakhshalipour, M., Lotfi-Kamran, P., Sarbazi-Azad, H.: Multi-lookahead offset prefetching. The Third Data Prefetching Championship (2019)
Shan, Y., Huang, Y., Chen, Y., Zhang, Y.: \(\{LegoOS\}\): a disseminated, distributed \(\{OS\}\) for hardware resource disaggregation. In: 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 69–87 (2018)
Shevgoor, M., Koladiya, S., Balasubramonian, R., Wilkerson, C., Pugsley, S.H., Chishti, Z.: Efficiently prefetching complex address patterns. In: 2015 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 41–152. IEEE (2015)
Sumsung, Expanding the limits of memory bandwidth and density: Samsung’s cxl dram memory expander, 2022.9. https://semiconductor.samsung.com/newsroom/tech-blog/expanding-the-limits-of-memory-bandwidth-and-density-samsungs-cxl-dram-memory-expander/
Tang, D., Bao, Y., Hu, W., Chen, M.: DMA cache: using on-chip storage to architecturally separate I/O data from CPU data for improving I/O performance. In: HPCA-16: The Sixteenth International Symposium on High-Performance Computer Architecture, pp. 1–12. IEEE (2010)
Viswanathan, V.: Disclosure of H/W prefetcher control on some intel processors. Intel SW Developer Zone (2014)
Wang, C., et al.: Semeru: a \(\{Memory-Disaggregated\}\) managed runtime. In: 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 261–280 (2020)
Wiki, Pcie 5.0, 2022.9. https://en.wikipedia.org/wiki/PCI_Express
Acknowledgment
This work is supported by Key-Area Research and Development Program of Guangdong Province (2021B0101310002), NSFC (61832020, 62032001, 92064006) and 111 Project (B18001).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhou, Z. et al. (2024). Polaris: Enhancing CXL-based Memory Expanders with Memory-side Prefetching. In: Li, C., Li, Z., Shen, L., Wu, F., Gong, X. (eds) Advanced Parallel Processing Technologies. APPT 2023. Lecture Notes in Computer Science, vol 14103. Springer, Singapore. https://doi.org/10.1007/978-981-99-7872-4_2
Download citation
DOI: https://doi.org/10.1007/978-981-99-7872-4_2
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-7871-7
Online ISBN: 978-981-99-7872-4
eBook Packages: Computer ScienceComputer Science (R0)
