research-article

Elo-MMR: A Rating System for Massive Multiplayer Competitions

Authors:

Paul LiuAuthors Info & Claims

WWW '21: Proceedings of the Web Conference 2021

Pages 1772 - 1784

https://doi.org/10.1145/3442381.3450091

Published: 03 June 2021 Publication History

Abstract

Skill estimation mechanisms, colloquially known as rating systems, play an important role in competitive sports and games. They provide a measure of player skill, which incentivizes competitive performances and enables balanced match-ups. In this paper, we present a novel Bayesian rating system for contests with many participants. It is widely applicable to competition formats with discrete ranked matches, such as online programming competitions, obstacle courses races, and video games. The system’s simplicity allows us to prove theoretical bounds on its robustness and runtime. In addition, we show that it is incentive-compatible: a player who seeks to maximize their rating will never want to underperform. Experimentally, the rating system surpasses existing systems in prediction accuracy, and computes faster than existing systems by up to an order of magnitude.

References

[1]

CodeChef Rating Mechanism. codechef.com/ratings

[2]

Codeforces: Results of 2019. codeforces.com/blog/entry/73683

[3]

Farming Volatility: How a major flaw in a well-known rating system takes over the GBL leaderboard. reddit.com/r/TheSilphRoad/comments/hwff2d/farming_volatility_how_a_major_flaw_in_a/

[4]

Halo Xbox video game franchise: in numbers. telegraph.co.uk/technology/video-games/11223730/Halo-in-numbers.html

[5]

Kaggle milestone: 5 million registered users!kaggle.com/general/164795

[6]

Kaggle Progression System. kaggle.com/progression

[7]

LeetCode New Contest Rating Algorithm. leetcode.com/discuss/general-discussion/468851/New-Contest-Rating-Algorithm-(Coming-Soon)

[8]

Open Codeforces Rating System. codeforces.com/blog/entry/20762

[9]

Topcoder Algorithm Competition Rating System. topcoder.com/community/competitive-programming/how-to-compete/ratings

[10]

Why Are Obstacle-Course Races So Popular?theatlantic.com/health/archive/2018/07/why-are-obstacle-course-races-so-popular/565130/

[11]

Sharad Agarwal and Jacob R. Lorch. 2009. Matchmaking for online games and other latency-sensitive P2P systems. In SIGCOMM 2009. 315–326.

Digital Library

[12]

Mark Yuying An. 1997. Log-concave probability distributions: Theory and statistical testing. (1997).

[13]

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika (1952), 324–345.

[14]

Shuo Chen and Thorsten Joachims. 2016. Modeling Intransitivity in Matchup and Comparison Data. In WSDM 2016. 227–236.

[15]

Rémi Coulom. [n.d.]. Whole-history rating: A Bayesian rating system for players of time-varying strength. In CG 2008. Springer, 113–124.

[16]

Pierre Dangauthier, Ralf Herbrich, Tom Minka, and Thore Graepel. 2007. TrueSkill Through Time: Revisiting the History of Chess. In NeurIPS 2007. 337–344.

[17]

Arpad E. Elo. 1961. New USCF rating system. Chess Life (1961), 160–161.

[18]

RNDr Michal Forišek. 2009. Theoretical and Practical Aspects of Programming Contest Ratings. (2009).

[19]

David A Freedman. 1963. On the asymptotic behavior of Bayes’ estimates in the discrete case. The Annals of Mathematical Statistics(1963), 1386–1403.

[20]

Mark E Glickman. 1995. A comprehensive guide to chess ratings. American Chess Journal(1995), 59–102.

[21]

Mark E Glickman. 1999. Parameter estimation in large dynamic paired comparison experiments. Applied Statistics (1999), 377–394.

[22]

Mark E Glickman. 2012. Example of the Glicko-2 system. Boston University (2012), 1–6.

[23]

Linxia Gong, Xiaochuan Feng, Dezhi Ye, Hao Li, Runze Wu, Jianrong Tao, Changjie Fan, and Peng Cui. 2020. OptMatch: Optimized Matchmaking via Modeling the High-Order Interactions on the Arena. In KDD 2020. 2300–2310.

Digital Library

[24]

Ralf Herbrich, Tom Minka, and Thore Graepel. 2006. TrueSkillTM: A Bayesian Skill Rating System. In NeurIPS 2006. 569–576.

[25]

Tzu-Kuo Huang, Chih-Jen Lin, and Ruby C. Weng. 2006. Ranking individuals by group comparisons. In ICML 2006. 425–432.

Digital Library

[26]

Stephanie Kovalchik. 2020. Extension of the Elo rating system to margin of victory. Int. J. Forecast. (2020).

[27]

Yao Li, Minhao Cheng, Kevin Fujii, Fushing Hsieh, and Cho-Jui Hsieh. 2018. Learning from Group Comparisons: Exploiting Higher Order Interactions. In NeurIPS 2018. 4986–4995.

[28]

Tom Minka, Ryan Cleven, and Yordan Zaykov. 2018. TrueSkill 2: An improved Bayesian skill rating system. Technical Report MSR-TR-2018-8. Microsoft.

[29]

T. Minka, J.M. Winn, J.P. Guiver, Y. Zaykov, D. Fabian, and J. Bronskill. /Infer.NET 0.3. Microsoft Research Cambridge. http://dotnet.github.io/infer.

[30]

Sergey I. Nikolenko, Alexander, and V. Sirotkin. 2010. Extensions of the TrueSkill TM rating system. In In Proceedings of the 9th International Conference on Applications of Fuzzy Systems and Soft Computing. 151–160.

[31]

Jerneja Premelč, Goran Vučković, Nic James, and Bojan Leskošek. 2019. Reliability of judging in DanceSport. Front. Psychol. (2019), 1001.

[32]

Josh Stone and Nicholas D Matsakis. The Rayon library (Rust Crate). crates.io/crates/rayon

[33]

Ruby C. Weng and Chih-Jen Lin. 2011. A Bayesian Approximation Method for Online Ranking. J. Mach. Learn. Res.(2011), 267–300.

[34]

John Michael Winn. 2019. Model-based machine learning.

[35]

Lin Yang, Stanko Dimitrov, and Benny Mantin. 2014. Forecasting sales of new virtual goods with the Elo rating system. RPM (2014), 457–469.

Cited By

Fan GZhang CWang KLi YChen JXu Z(2024)CUPID: Improving Battle Fairness and Position Satisfaction in Online MOBA Games with a Re-matchmaking SystemProceedings of the ACM on Human-Computer Interaction10.1145/36869788:CSCW2(1-39)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686978
Yuksel C(2024)Skill-Based Matchmaking for Competitive Two-Player GamesProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36513037:1(1-19)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3651303
Du BYang JWu SZhang ZLiu Y(2024)Enhancing Programming Competition Performance: A Data-Driven Approach to Personalized Training2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C63300.2024.00059(417-422)Online publication date: 1-Jul-2024
https://doi.org/10.1109/QRS-C63300.2024.00059
Show More Cited By

Recommendations

TeamSkill: modeling team chemistry in online multi-player games
PAKDD'11: Proceedings of the 15th Pacific-Asia conference on Advances in knowledge discovery and data mining - Volume Part II

In this paper, we introduce a framework for modeling elements of "team chemistry" in the skill assessment process using the performances of subsets of teams and four approaches which make use of this framework to estimate the collective skill of a team. ...
Competing sellers in online markets: reserve prices, shill bidding, and auction fees
AAMAS '06: Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems

In this paper, we consider competition between sellers offering similar items in concurrent online auctions, where each seller must set its individual auction parameters (such as the reserve price) in such a way as to attract buyers. We show that there ...
Elo Rating System

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '21: Proceedings of the Web Conference 2021

April 2021

4054 pages

ISBN:9781450383127

DOI:10.1145/3442381

Editors:
Jure Leskovec
Stanford
,
Marko Grobelnik
Jožef Stefan Institute
,
Marc Najork
Google
,
Jie Tang
Tsinghua University
,
Leila Zia
Wikimedia Foundation

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '21

Sponsor:

SIGWEB

WWW '21: The Web Conference 2021

April 19 - 23, 2021

Ljubljana, Slovenia

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
393
Total Downloads

Downloads (Last 12 months)120
Downloads (Last 6 weeks)10

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fan GZhang CWang KLi YChen JXu Z(2024)CUPID: Improving Battle Fairness and Position Satisfaction in Online MOBA Games with a Re-matchmaking SystemProceedings of the ACM on Human-Computer Interaction10.1145/36869788:CSCW2(1-39)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686978
Yuksel C(2024)Skill-Based Matchmaking for Competitive Two-Player GamesProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36513037:1(1-19)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3651303
Du BYang JWu SZhang ZLiu Y(2024)Enhancing Programming Competition Performance: A Data-Driven Approach to Personalized Training2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C63300.2024.00059(417-422)Online publication date: 1-Jul-2024
https://doi.org/10.1109/QRS-C63300.2024.00059
Toda KInoue STobe Y(2024)Optimization of Player-Combinations in Multiplayer Games2024 International Conference on Information Networking (ICOIN)10.1109/ICOIN59985.2024.10572104(746-750)Online publication date: 17-Jan-2024
https://doi.org/10.1109/ICOIN59985.2024.10572104
Zhang YWeiss T(2024)Crowd-sourced Evaluation of Combat Animations2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00015(60-65)Online publication date: 17-Jan-2024
https://doi.org/10.1109/AIxVR59861.2024.00015
Powell B(2023)Generalizing the Elo rating system for multiplayer games and races: why endurance is better than speedJournal of Quantitative Analysis in Sports10.1515/jqas-2023-000419:3(223-243)Online publication date: 30-Jun-2023
https://doi.org/10.1515/jqas-2023-0004
Wang J(2023)Graph Embedding Augmented Skill Rating SystemIEEE Transactions on Games10.1109/TG.2022.322184915:3(460-468)Online publication date: Sep-2023
https://doi.org/10.1109/TG.2022.3221849
C. AD. CD. V. PChandavarkar B(2022)Transparency in Content and Source ModerationAdvances in Data Science and Artificial Intelligence10.1007/978-3-031-16178-0_31(445-454)Online publication date: 29-Sep-2022
https://doi.org/10.1007/978-3-031-16178-0_31

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten