research-article

Dissecting Cloud Gaming Performance with DECAF

Authors:

Muhammad ShahzadAuthors Info & Claims

Proceedings of the ACM on Measurement and Analysis of Computing Systems, Volume 5, Issue 3

Article No.: 31, Pages 1 - 27

https://doi.org/10.1145/3491043

Published: 15 December 2021 Publication History

Abstract

Cloud gaming platforms have witnessed tremendous growth over the past two years with a number of large Internet companies including Amazon, Facebook, Google, Microsoft, and Nvidia publicly launching their own platforms. While cloud gaming platforms continue to grow, the visibility in their performance and relative comparison is lacking. This is largely due to absence of systematic measurement methodologies which can generally be applied. As such, in this paper, we implement DECAF, a methodology to systematically analyze and dissect the performance of cloud gaming platforms across different game genres and game platforms. DECAF is highly automated and requires minimum manual intervention. By applying DECAF, we measure the performance of three commercial cloud gaming platforms including Google Stadia, Amazon Luna, and Nvidia GeForceNow, and uncover a number of important findings. First, we find that processing delays in the cloud comprise majority of the total round trip delay experienced by users, accounting for as much as 73.54% of total user-perceived delay. Second, we find that video streams delivered by cloud gaming platforms are characterized by high variability of bitrate, frame rate, and resolution. Platforms struggle to consistently serve 1080p/60 frames per second streams across different game genres even when the available bandwidth is 8-20× that of platform's recommended settings. Finally, we show that game platforms exhibit performance cliffs by reacting poorly to packet losses, in some cases dramatically reducing the delivered bitrate by up to 6.6× when loss rates increase from 0.1% to 1%. Our work has important implications for cloud gaming platforms and opens the door for further research on comprehensive measurement methodologies for cloud gaming.

References

[1]

Amazon Luna. https://www.amazon.com/luna/landing-page.

[2]

Cedexis-Citrix. https://www.cedexis.com/.

[3]

FFmpeg. https://www.ffmpeg.org/.

[4]

G.1072 : Opinion model predicting gaming quality of experience for cloud gaming services. https://www.itu.int/rec/TREC-G.1072/en.

[5]

GaiKai. https://www.crunchbase.com/organization/gaikai.

[6]

Gaming Monitor Alienware - AW2521HFL. https://www.bestbuy.com/site/alienware-aw2521hfl-24--5-ips-led-fhdfreesync-and-g-sync-compatible-monitor-displayport-hdmi-usb-lunar-light/6406940.p'skuId=6406940.

[7]

GeForce Now. https://www.nvidia.com/en-us/geforce-now/.

[8]

Google Stadia. https://stadia.google.com/.

[9]

Hulu. https://www.hulu.com/.

[10]

International Center For History Of Electronic Games. https://www.museumofplay.org/about/icheg.

[11]

Netflix. https://www.netflix.com/.

[12]

OnLive. http://onlive.com/.

[13]

Ookla. https://www.ookla.com/.

[14]

Open vSwitch. https://www.openvswitch.org/.

[15]

Prime video. https://www.primevideo.com/.

[16]

The Technology Behind A Low Latency Cloud Gaming Service. https://parsec.app/blog/description-of-parsectechnology-b2738dcc3842.

[17]

The Cloud Gaming Market Will Pass the Billion-Dollar Mark in 2021 with Revenues of 1.4B. https: //newzoo.com/insights/articles/cloud-gaming-market-first-billion-dollar-year-23--7-million-paying-users-willgenerate-revenues-of-1--4-billion-in-2021/.

[18]

ThousandEyes. https://www.thousandeyes.com/.

[19]

VMAF - Video Multi-Method Assessment Fusion. https://github.com/Netflix/vmaf.

[20]

X-Cloud cloud gaming. https://www.xbox.com/en-US/xbox-game-pass/cloud-gaming.

[21]

X-Cloud game pass. https://www.xbox.com/en-US/xbox-game-pass/cloud-gaming?xr=shellnav.

[22]

Yahoo. https://money.yahoo.com/cloud-gaming-market-growth-trends-141300368.html.

[23]

Google Stadia bandwidth requirements - what kind of connection will you need to stream games? https://www. gamesradar.com/google-stadia-bandwidth-requirement/.

[24]

Assassins Creed Valhalla. https://www.ubisoft.com/en-us/game/assassins-creed/valhalla.

[25]

Cloud Gaming, Meet Facebook Gaming. https://www.facebook.com/fbgaminghome/blog/cloud-gaming-meetfacebook-gaming.

[26]

Crew. https://www.ubisoft.com/en-us/game/the-crew/the-crew.

[27]

Crew 2. https://www.ubisoft.com/en-us/game/the-crew/the-crew-2.

[28]

Far Cry 5. https://www.ubisoft.com/en-us/game/far-cry/far-cry-5.

[29]

I used cloud gaming exclusively for a month. Here's what happened. https://www.digitaltrends.com/gaming/i-usedcloud-gaming-for-a-month-shadow-stadia-geforce-now/.

[30]

To succeed, cloud gaming needs to diappear. https://www.theverge.com/2021/6/23/22547334/cloud-gaming-xboxxcloud-microsoft-streaming-google-stadia-amazon.

[31]

5 reasons cloud gaming isn't doing it for me -- yet. https://www.cnet.com/tech/computing/5-reasons-cloud-gamingisnt-doing-it-for-me-yet/.

[32]

Bandwidth, data usage, and stream quality. https://support.google.com/stadia/answer/9607891?hl=en.

[33]

The Chromium Project. https://www.chromium.org/.

[34]

DECAF: Dissecting Cloud Gaming Performace. https://github.com/decafCG/decaf.

[35]

Geforce Now System Requirements. https://www.nvidia.com/en-us/geforce-now/system-reqs/.

[36]

Iperf. https://iperf.fr/.

[37]

Maxmind. https://www.maxmind.com/en/home.

[38]

Python Imaging Library. https://pypi.org/project/Pillow/.

[39]

Python-tesseract: an optical character recognition tool for python. https://pypi.org/project/pytesseract/.

[40]

Python win32con Module. https://www.programcreek.com/python/index/475/win32con.

[41]

pywin32. https://pypi.org/project/pywin32/.

[42]

What Is Amazon Luna? https://www.amazon.com/gp/help/customer/display.html?nodeId=G8WBF7CLZX7W345R. Proc. ACM Meas. Anal. Comput. Syst., Vol. 5, No. 3, Article 31. Publication date: December 2021. 31:24 Hassan Iqbal, Ayesha Khalid, & Muhammad Shahzad

[43]

Adnan Ahmed, Zubair Shafiq, and Amir Khakpour. QoE Analysis of a Large-Scale Live Video Streaming Event (ACM SIGMETRICS '16).

[44]

Zahaib Akhtar, Yun Seong Nam, Ramesh Govindan, Sanjay Rao, Jessica Chen, Ethan Katz-Bassett, Bruno Ribeiro, Jibin Zhan, and Hui Zhang. Oboe: Auto-tuning Video ABR Algorithms to Network Conditions (ACM SIGCOMM '18).

[45]

Todd Arnold, Jia He, Weifan Jiang, Matt Calder, Italo Cunha, Vasileios Giotsas, and Ethan Katz-Bassett. Cloud Provider Connectivity in the Flat Internet (ACM IMC '20).

[46]

Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, and Hui Zhang. Developing a Predictive Model of Quality of Experience for Internet Video (ACM SIGCOMM '13).

[47]

Hafiz Mohsin Bashir, Abdullah Bin Faisal, M Asim Jamshed, Peter Vondras, Ali Musa Iftikhar, Ihsan Ayyub Qazi, and Fahad R. Dogar. Reducing Tail Latency Using Duplication: A Multi-Layered Approach (ACM CoNEXT '19).

[48]

Jan A Bergstra and CA Middelburg. ITU-T Recommendation G. 107: The E-Model, a computational model for use in transmission planning. (2003).

[49]

Wei Cai, Ryan Shea, Chun-Ying Huang, Kuan-Ta Chen, Jiangchuan Liu, and Cheng-Hsin Hsu. A Survey on Cloud Gaming: Future of Computer Games. IEEE Access (2016).

[50]

Eduardo F Camacho and Carlos Bordons Alba. Model predictive control. Springer science & business media.

[51]

Marc Carrascosa and Boris Bellalta. Cloud-gaming:Analysis of Google Stadia traffic. arXiv:2009.09786 [cs.NI]

[52]

Kuan-Ta Chen, Yu-Chun Chang, Po-Han Tseng, Chun-Ying Huang, and Chin-Laung Lei. Measuring the Latency of Cloud Gaming Systems (ACM MM '11).

[53]

Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei. How Sensitive Are Online Gamers to Network Quality? Commun. ACM (2006).

[54]

Yi-Ching Chiu, Brandon Schlinker, Abhishek Balaji Radhakrishnan, Ethan Katz-Bassett, and Ramesh Govindan. Are We One Hop Away from a Better Internet? (ACM IMC '15).

[55]

Jonathan Deber, Ricardo Jota, Clifton Forlines, and Daniel Wigdor. How Much Faster is Fast Enough? User Perception of Latency & Latency Improvements in Direct and Indirect Touch (ACM SIGCHI '15).

[56]

Florin Dobrian, Vyas Sekar, Asad Awan, Ion Stoica, Dilip Joseph, Aditya Ganjam, Jibin Zhan, and Hui Zhang. Understanding the Impact of Video Quality on User Engagement (ACM SIGCOMM '11).

[57]

Andrea Di Domenico, Gianluca Perna, Martino Trevisan, Luca Vassio, and Danilo Giordano. A network analysis on cloud gaming: Stadia, GeForce Now and PSNow. arXiv:2012.06774 [cs.NI]

[58]

Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series.

[59]

Mohamed Hegazy, Khaled Diab, Mehdi Saeedi, Boris Ivanovic, Ihab Amer, Yang Liu, Gabor Sines, and Mohamed Hefeeda. Content-Aware Video Encoding for Cloud Gaming.

[60]

Te-Yuan Huang, Nikhil Handigol, Brandon Heller, Nick McKeown, and Ramesh Johari. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard (ACM IMC '12).

[61]

Niels Justesen, Philip Bontrager, Julian Togelius, and Sebastian Risi. Deep learning for video game playing. IEEE Transactions on Games 12 (2019).

[62]

Teemu Kämäräinen, Matti Siekkinen, Antti Ylä-Jääski, Wenxiao Zhang, and Pan Hui. A Measurement Study on Achieving Imperceptible Latency in Mobile Cloud Gaming. Association for Computing Machinery.

[63]

Guillaume Lample and Devendra Singh Chaplot. Playing FPS games with deep reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence.

[64]

Xing Liu, Bo Han, Feng Qian, and Matteo Varvello. LIME: Understanding Commercial 360° Live Video Streaming Services (MMSys '19).

[65]

Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. Neural Adaptive Video Streaming with Pensieve (ACM SIGCOMM '17).

[66]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529--533.

[67]

Donald A. Norman. The design of everyday things. Basic Books.

Digital Library

[68]

Jesse Schell. The art of game design: a book of lenses. CRC Press, Taylor & Francis Group.

[69]

Steven Schmidt, Saman Zadtootaghaj, Saeed Shafiee Sabet, and Sebastian Möller. Modeling and Understanding the Quality of Experience of Online Mobile Gaming Services (IEEE QoMEX '21).

[70]

C.E. Shannon. Communication in the Presence of Noise. Proceedings of the IRE (1949).

[71]

Iraj Sodagar. The MPEG-DASH Standard for Multimedia Streaming Over the Internet. IEEE MultiMedia (2011).

Digital Library

[72]

German Sviridov, Cedric Beliard, Andrea Bianco, Paolo Giaccone, and Dario Rossi. Removing human players from the loop: AI-assisted assessment of Gaming QoE (IEEE INFOCOM WKSHPS '20).

[73]

Bolun Wang, Xinyi Zhang, Gang Wang, Haitao Zheng, and Ben Y. Zhao. Anatomy of a Personalized Livestreaming System (ACM IMC '16). Proc. ACM Meas. Anal. Comput. Syst., Vol. 5, No. 3, Article 31. Publication date: December 2021. Dissecting Cloud Gaming Performance with DECAF 31:25

[74]

S. Wang and S. Dey. Modeling and Characterizing User Experience in a Cloud Server Based Mobile Gaming Approach (IEEE GLOBECOM '09).

[75]

Marek Wydmuch, Micha Kempka, and Wojciech Jakowski. Vizdoom competitions: Playing doom from pixels. IEEE Transactions on Games 11, 3 (2018), 248--259.

[76]

Zheng Xue, Di Wu, Jian He, Xiaojun Hei, and Yong Liu. Playing High-End Video Games in the Cloud: A Measurement Study. IEEE Transactions on Circuits and Systems for Video Technology (2015).

Digital Library

[77]

Jun Yi, Shiqing Luo, and Zhisheng Yan. A Measurement Study of YouTube 360° Live Video Streaming (ACM NOSSDAV '19).

[78]

Saman Zadtootaghaj, Steven Schmidt, Saeed Shafiee Sabet, Sebastian Möller, and Carsten Griwodz. Quality estimation models for gaming video streaming services using perceptual video quality dimensions (ACM MM '20).

[79]

Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint arXiv:1806.01830 (2018).

[80]

Timothy Zhu, Alexey Tumanov, Michael A. Kozuch, Mor Harchol-Balter, and Gregory R. Ganger. PriorityMeister: Tail Latency QoS for Shared Networked Storage (ACM SOCC '14).

Cited By

Lyu MWang YSivaraman V(2024)Do Cloud Games Adapt to Client Settings and Network Conditions?2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619817(482-488)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619817
Mohan NFerguson ACech HBose RRenatin PMarina MOtt JChua TNgo CKa-Wei Lee RKumar RLauw H(2024)A Multifaceted Look at Starlink PerformanceProceedings of the ACM Web Conference 202410.1145/3589334.3645328(2723-2734)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645328
Zhang YKutscher DCui Y(2024)Networked Metaverse Systems: Foundations, Gaps, Research DirectionsIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.34260985(5488-5539)Online publication date: 2024
https://doi.org/10.1109/OJCOMS.2024.3426098
Show More Cited By

Index Terms

Dissecting Cloud Gaming Performance with DECAF
1. Networks
  1. Network performance evaluation
    1. Network measurement
    2. Network performance analysis

Recommendations

Dissecting Cloud Gaming Performance with DECAF
SIGMETRICS/PERFORMANCE '22: Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems

Cloud gaming platforms have witnessed tremendous growth over the past two years, with a number of large Internet companies including Amazon, Facebook, Google, Microsoft, and Nvidia publicly launching their own platforms. However, there is an absence of ...
Dissecting Cloud Gaming Performance with DECAF
SIGMETRICS '22

Cloud gaming platforms have witnessed tremendous growth over the past two years, with a number of large Internet companies including Amazon, Facebook, Google, Microsoft, and Nvidia publicly launching their own platforms. However, there is an absence of ...
Dissecting the protocol and network traffic of the OnLive cloud gaming platform

Cloud gaming is a new paradigm that is envisaged to play a pivotal role in the video game industry in forthcoming years. Cloud gaming, or gaming on demand, is a type of online gaming that allows on-demand streaming of game content onto non-specialised ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Measurement and Analysis of Computing Systems

Proceedings of the ACM on Measurement and Analysis of Computing Systems Volume 5, Issue 3

POMACS

December 2021

435 pages

EISSN:2476-1249

DOI:10.1145/3506735

Editors:
Augustin Chaintreau
Columbia University
,
Leana Golubchik
University of Southern California
,
Zhi-Li Zhang
University of Minnesota

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 December 2021

Published in POMACS Volume 5, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
723
Total Downloads

Downloads (Last 12 months)216
Downloads (Last 6 weeks)9

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lyu MWang YSivaraman V(2024)Do Cloud Games Adapt to Client Settings and Network Conditions?2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619817(482-488)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619817
Mohan NFerguson ACech HBose RRenatin PMarina MOtt JChua TNgo CKa-Wei Lee RKumar RLauw H(2024)A Multifaceted Look at Starlink PerformanceProceedings of the ACM Web Conference 202410.1145/3589334.3645328(2723-2734)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645328
Zhang YKutscher DCui Y(2024)Networked Metaverse Systems: Foundations, Gaps, Research DirectionsIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.34260985(5488-5539)Online publication date: 2024
https://doi.org/10.1109/OJCOMS.2024.3426098
Hoßfeld THeegaard PVarela MJarschel M(2024)User-centric Markov reward model for state-dependent Erlang loss systemsPerformance Evaluation10.1016/j.peva.2024.102425165(102425)Online publication date: Aug-2024
https://doi.org/10.1016/j.peva.2024.102425
Shi BZhang L(2024)Event Sequence-Driven Generalized and Accurate End-to-End Streaming Latency MeasurementMachine Learning and Intelligent Communication10.1007/978-3-031-71716-1_10(119-131)Online publication date: 20-Sep-2024
https://doi.org/10.1007/978-3-031-71716-1_10
Cheng RLiu KWu NHan B(2023)Enriching Telepresence with Semantic-driven Holographic CommunicationProceedings of the 22nd ACM Workshop on Hot Topics in Networks10.1145/3626111.3628184(147-156)Online publication date: 28-Nov-2023
https://dl.acm.org/doi/10.1145/3626111.3628184
Ky JMathieu BLahmadi ABoutaba R(2023)ML Models for Detecting QoE Degradation in Low-Latency Applications: A Cloud-Gaming Case StudyIEEE Transactions on Network and Service Management10.1109/TNSM.2023.329380620:3(2295-2308)Online publication date: 11-Jul-2023
https://dl.acm.org/doi/10.1109/TNSM.2023.3293806
Ky JMathieu BLahmadi ABoutaba R(2022)Assessing Unsupervised Machine Learning Solutions for Anomaly Detection in Cloud Gaming SessionsProceedings of the 18th International Conference on Network and Service Management10.5555/3581644.3581714(1-7)Online publication date: 31-Oct-2022
https://dl.acm.org/doi/10.5555/3581644.3581714
Ky JMathieu BLahmadi ABoutaba R(2022)Assessing Unsupervised Machine Learning solutions for Anomaly Detection in Cloud Gaming Sessions2022 18th International Conference on Network and Service Management (CNSM)10.23919/CNSM55787.2022.9964533(367-373)Online publication date: 31-Oct-2022
https://doi.org/10.23919/CNSM55787.2022.9964533
Cheng RWu NVarvello MChen SHan BBarakat CPelsser CBenson TChoffnes D(2022)Are we ready for metaverse?Proceedings of the 22nd ACM Internet Measurement Conference10.1145/3517745.3561417(504-518)Online publication date: 25-Oct-2022
https://dl.acm.org/doi/10.1145/3517745.3561417
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents