The Surprising Benefits of Base Rate Neglect in Robust Aggregation
Authors: 
Yuqing Kong, Shu Wang, Ying WangAuthors Info & Claims
Yuqing Kong, Shu Wang, Ying WangAuthors Info & ClaimsPage 1131
Abstract
Robust aggregation integrates predictions from multiple experts without knowledge of the experts' information structures. Prior work assumes experts are Bayesian, providing predictions as perfect posteriors based on their signals. However, real-world experts often deviate systematically from Bayesian reasoning. Our work considers experts who tend to ignore the base rate. We find that a certain degree of base rate neglect helps with robust forecast aggregation.
Specifically, we consider a forecast aggregation problem with two experts who each predict a binary world state after observing private signals. Unlike previous work, we model experts exhibiting base rate neglect, where they incorporate the base rate information to degree λ ∈ [0, 1], with λ = 0 indicating complete ignorance and λ = 1 perfect Bayesian updating. To evaluate aggregators' performance, we adopt Arieli et al. [2018]'s worst-case regret model, which measures the maximum regret across the set of considered information structures compared to an omniscient benchmark.
Our results reveal the surprising V-shape of regret as a function of λ. That is, predictions with an intermediate base rate consideration degree λ < 1 counter-intuitively lead to lower regret than perfect Bayesian posteriors with λ = 1. We theoretically prove that the regret cure of any agregator is single-troughted. Through numerical analyses, we also find the regrets of many aggregators are non-monotone, thus validating the V-shaped regret. Furthermore, the "tight" lower bound of the optimal regret regarding each λ degree is also V-shaped.
Meanwhile, existing aggregators, including the simple average aggregator and the average prior aggregator, only work well for a small range of prior consideration degree λ. We provide a new closed-form aggregator that generally performs well for all λ. Our aggregator can achieve a near-zero regret when the degree λ equals 0.5, which almost recovers the perfect Bayesian posterior that an omniscient aggregator provides.
Finally, we conduct an empirical study to test the base rate neglect model and evaluate various aggregators' performance. Our findings reveal that 57% of predictions lie outside the theoretical range where λ ∈ [0, 1]. For inside subsample which exhibits base rate neglect, our aggregator outperforms the simple average and average prior aggregators. We also observe that some degree of base rate neglect can benefit aggregation.
A full version of this paper can be found at https://arxiv.org/abs/2406.13490.
Index Terms
- The Surprising Benefits of Base Rate Neglect in Robust Aggregation
Index terms have been assigned to the content through auto-classification.
Recommendations
Robust Decision Aggregation with Second-order Information
WWW '24: Proceedings of the ACM Web Conference 2024We consider a decision aggregation problem with two experts who each make a binary recommendation after observing a private signal about an unknown binary world state. An agent, who does not know the joint information structure between signals and states,...
Forecast Aggregation
EC '17: Proceedings of the 2017 ACM Conference on Economics and ComputationBayesian experts with a common prior that are exposed to different evidence possibly make contradicting probabilistic forecasts. A policy maker who receives the forecasts must aggregate them in the best way possible. This is a challenge whenever the ...
Securing secure aggregation: mitigating multi-round privacy leakage in federated learning
AAAI'23/IAAI'23/EAAI'23: Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial IntelligenceSecure aggregation is a critical component in federated learning (FL), which enables the server to learn the aggregate model of the users without observing their local models. Conventionally, secure aggregation algorithms focus only on ensuring the ...
Comments
Information & Contributors
Information
Published In
July 2024
1340 pages
ISBN:9798400707049
DOI:10.1145/3670865
- Chair:
- Dirk Bergemann,
- Program Chairs:
- Robert Kleinberg,
- Daniela Saban
Copyright © 2024 Copyright held by the owner/author(s).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 17 December 2024
Check for updates
Qualifiers
- Extended-abstract
Funding Sources
Conference
EC '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 664 of 2,389 submissions, 28%
Upcoming Conference
EC '25
- Sponsor:
- sigecom
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 21Total Downloads
- Downloads (Last 12 months)21
- Downloads (Last 6 weeks)21
Reflects downloads up to 16 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in