Introducing CausalBench: A Flexible Benchmark Framework for Causal Analysis and Machine Learning
Authors: 
Ahmet Kapkiç, Pratanu Mandal, Shu Wan, Paras Sheth, Abhinav Gorantla, Yoonhyuk Choi, 
Huan Liu, K. Selçuk CandanAuthors Info & Claims
Ahmet Kapkiç, Pratanu Mandal, Shu Wan, Paras Sheth, Abhinav Gorantla, Yoonhyuk Choi, Huan Liu, K. Selçuk CandanAuthors Info & ClaimsPages 5220 - 5224
Abstract
While witnessing the exceptional success of machine learning (ML) technologies in many applications, users are starting to notice a critical shortcoming of ML: correlation is a poor substitute for causation. The conventional way to discover causal relationships is to use randomized controlled experiments (RCT); in many situations, however, these are impractical or sometimes unethical. Causal learning from observational data offers a promising alternative. While being relatively recent, causal learning aims to go far beyond conventional machine learning, yet several major challenges remain. Unfortunately, advances are hampered due to the lack of unified benchmark datasets, algorithms, metrics, and evaluation service interfaces for causal learning. In this paper, we introduce CausalBench, a transparent, fair, and easy-to-use evaluation platform, aiming to (a) enable the advancement of research in causal learning by facilitating scientific collaboration in novel algorithms, datasets, and metrics and (b) promote scientific objectivity, reproducibility, fairness, and awareness of bias in causal learning research. CausalBench provides services for benchmarking data, algorithms, models, and metrics, impacting the needs of a broad of scientific and engineering disciplines.
References
[1]
Fahim Tasneema Azad, K. Selçuk Candan, Ahmet Kapkiç, Mao-Lin Li, Huan Liu, Pratanu Mandal, Paras Sheth, Bilgehan Arslan, Gerardo Chowell-Puente, John Sabo, Rebecca Muenich, Javier Redondo Anton, and Maria Luisa Sapino. 2024. A Vision for Spatio-Causal Situation Awareness, Forecasting, and Planning. ACM Transactions on Spatial Algorithms and Systems (TSAS) (2024). Accepted for publication.
[2]
Andrew P. Bradley. 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition 30, 7 (1997), 1145--1159. https://doi.org/10.1016/S0031--3203(96)00142--2
[3]
Debo Cheng, Jiuyong Li, Lin Liu, Jixue Liu, and Thuc Duy Le. 2024. Data-Driven Causal Effect Estimation Based on Graphical Causal Modelling: A Survey. ACM Comput. Surv. 56, 5, Article 127 (jan 2024), 37 pages. https://doi.org/10.1145/3636423
[4]
Lu Cheng, Ruocheng Guo, Raha Moraffah, Paras Sheth, K. Selçuk Candan, and Huan Liu. 2022. Evaluation Methods and Measures for Causal Learning Algorithms. IEEE Transactions on Artificial Intelligence 3, 6 (2022), 924--943. https://doi.org/10.1109/TAI.2022.3150264
[5]
Yoonhyuk Choi, Jiho Choi, Taewook Ko, Hyungho Byun, and Chong-Kwon Kim. 2022. Review-Based Domain Disentanglement without Duplicate Users or Contexts for Cross-Domain Recommendation. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM '22). Association for Computing Machinery, New York, NY, USA, 293--303. https://doi.org/10.1145/3511808.3557434
[6]
Iain M Cockburn, Rebecca Henderson, and Scott Stern. 2018. The impact of artificial intelligence on innovation. Vol. 24449. National bureau of economic research Cambridge, MA, USA.
[7]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248--255. https://doi.org/10.1109/CVPR.2009.5206848
[8]
Li Deng. 2012. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine 29, 6 (2012), 141--142.
[9]
Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http://archive.ics.uci.edu/ml
[10]
Kapkic et. al. 2024. CausalBench Demo Video. https://drive.google.com/drive/folders/1ckfKqci1sj8u7G6QAOb02YBSA3zqShxO?usp=sharing.
[11]
Kapkic et. al. 2024. CausalBench Website. www.causalbench.org.
[12]
Arunim Garg and Vijay Mago. 2021. Role of machine learning in medical research: A survey. Comput. Sci. Rev. 40, C (may 2021), 17 pages. https://doi.org/10.1016/j.cosrev.2021.100370
[13]
S Hettich and S. D Bay. 1999. The UCI KDD Archive. http://kdd.ics.uci.edu
[14]
M. Hino, E. Benami, and N. Brooks. 2018. Machine learning for environmental monitoring. Nature Sustainability 1, 10 (01 Oct 2018), 583--588. https://doi.org/10.1038/s41893-018-0142--9
[15]
Michael Höfler. 2005. The Bradford Hill considerations on causality: a counter-factual perspective. Emerging Themes in Epidemiology 2, 1 (03 Nov 2005), 11. https://doi.org/10.1186/1742--7622--2--11
[16]
Alex Krizhevsky. 2012. Learning Multiple Layers of Features from Tiny Images. University of Toronto (05 2012).
[17]
Mao-Lin Li, K. Selçuk Candan, and Maria Luisa Sapino. 2023. CTT: Causally Informed Tensor Train Decomposition. In IEEE Big Data. 1180--1187.
[18]
Ana Rita Nogueira, Andrea Pugnana, Salvatore Ruggieri, Dino Pedreschi, and João Gama. 2022. Methods and tools for causal discovery and causal inference. WIREs Data Mining and Knowledge Discovery 12, 2 (2022), e1449. https://doi.org/10.1002/widm.1449 arXiv:https://wires.onlinelibrary.wiley.com/doi/pdf/10.1002/widm.1449
[19]
Mattia Prosperi, Yi Guo, Matt Sperrin, James S. Koopman, Jae S. Min, Xing He, Shannan Rich, Mo Wang, Iain E. Buchan, and Jiang Bian. 2020. Causal inference and counterfactual prediction in machine learning for actionable healthcare. Nature Machine Intelligence 2, 7 (01 Jul 2020), 369--375. https://doi.org/10.1038/s42256-020-0197-y
[20]
Amit Sharma, Jake M. Hofman, and Duncan J. Watts. 2015. Estimating the Causal Impact of Recommendation Systems from Observational Data. In Proceedings of the Sixteenth ACM Conference on Economics and Computation (Portland, Oregon, USA) (EC '15). Association for Computing Machinery, New York, NY, USA, 453--470. https://doi.org/10.1145/2764468.2764488
[21]
Paras Sheth, Ruocheng Guo, Lu Cheng, Huan Liu, and K. Selçuk Candan. 2023. Causal Disentanglement for Implicit Recommendations with Network Information. ACM Trans. Knowl. Discov. Data 17, 7 (2023), 94:1--94:18.
[22]
Paras Sheth, Ruocheng Guo, Kaize Ding, Lu Cheng, K. Selçuk Candan, and Huan Liu. 2022. Causal Disentanglement with Network Information for Debiased Recommendations. In SISAP. 265--273.
[23]
Paras Sheth, Ujun Jeong, Ruocheng Guo, Huan Liu, and K. Selçuk Candan. 2021. CauseBox: A Causal Inference Toolbox for Benchmarking Treatment Effect Estimators with Machine Learning Methods. In CIKM. 4789--4793.
[24]
Paras Sheth, Ting Liu, Durmus Doner, Qi Deng, Yuhang Wei, Rebecca Muenich, John Sabo, K. Selçuk Candan, and Huan Liu. 2022. Causal Discovery for Feature Selection in Physical Process-Based Hydrological Systems. In 2022 IEEE International Conference on Big Data (Big Data). 5568--5577. https://doi.org/10.1109/BigData55660.2022.10020794
[25]
Paras Sheth, Raha Moraffah, Tharindu S. Kumarage, Aman Chadha, and Huan Liu. 2024. Causality Guided Disentanglement for Cross-Platform Hate Speech Detection. In WSDM. 626--635.
[26]
Paras Sheth, Ahmadreza Mosallanezhad, Kaize Ding, Reepal Shah, John Sabo, Huan Liu, and K. Selçuk Candan. 2023. STREAMS: Towards Spatio-Temporal Causal Discovery with Reinforcement Learning for Streamflow Rate Prediction. In CIKM. 4815--4821.
[27]
Paras Sheth, Reepal Shah, John Sabo, K. Selçuk Candan, and Huan Liu. 2022. STCD: A Spatio-Temporal Causal Discovery Framework for Hydrological Systems. In IEEE Big Data. 5578--5583.
[28]
Herbert A. Simon. 1977. Spurious Correlation: A Causal Interpretation. Springer Netherlands, Dordrecht, 93--106. https://doi.org/10.1007/978--94-010--9521--1_7
[29]
M. Stone. 2018. Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society: Series B (Methodological) 36, 2 (12 2018), 111--133. https://doi.org/10.1111/j.2517--6161.1974.tb00994.x arXiv:https://academic.oup.com/jrsssb/article-pdf/36/2/111/49096683/jrsssb_36_2_111.pdf
[30]
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. 2014. OpenML: networked science in machine learning. SIGKDD Explor. Newsl. 15, 2 (jun 2014), 49--60. https://doi.org/10.1145/2641190.2641198
[31]
Song Wei, Ronald Moore, Hanyu Zhang, Yao Xie, and Rishikesan Kamaleswaran. 2023. Transfer Causal Learning: Causal Effect Estimation with Knowledge Transfer. In ICML 3rd Workshop on Interpretable Machine Learning in Healthcare (IMLH). https://openreview.net/forum?id=V3GGYh8CKq
[32]
Timothy L. Wiemken and Robert R. Kelley. 2020. Machine Learning in Epidemiology and Health Outcomes Research. Annual Review of Public Health 41, Volume 41, 2020 (2020), 21--36. https://doi.org/10.1146/annurev-publhealth-040119-094437
[33]
Alessio Zanga, Elif Ozkirimli, and Fabio Stella. 2022. A Survey on Causal Discovery: Theory and Practice. International Journal of Approximate Reasoning 151 (2022), 101--129. https://doi.org/10.1016/j.ijar.2022.09.004
Index Terms
- Introducing CausalBench: A Flexible Benchmark Framework for Causal Analysis and Machine Learning
Recommendations
Socially Responsible Machine Learning: A Causal Perspective
KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data MiningThe evergrowing reliance of humans and society on machine learning methods has raised concerns about their trustworthiness and liability. As a response to these concerns, Socially Responsible Machine Learning (SRML) aims at developing fair, transparent, ...
Variable-Agnostic Causal Exploration for Reinforcement Learning
Machine Learning and Knowledge Discovery in Databases. Research TrackAbstractModern reinforcement learning (RL) struggles to capture real-world cause-and-effect dynamics, leading to inefficient exploration due to extensive trial-and-error actions. While recent efforts to improve agent exploration have leveraged causal ...
Comments
Information & Contributors
Information
Published In
October 2024
5705 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 October 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
- NSF
Conference
CIKM '24
Sponsor:
CIKM '24: The 33rd ACM International Conference on Information and Knowledge Management
October 21 - 25, 2024
ID, Boise, USA
Acceptance Rates
Overall Acceptance Rate 1,861 of 8,427 submissions, 22%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 85Total Downloads
- Downloads (Last 12 months)85
- Downloads (Last 6 weeks)13
Reflects downloads up to 08 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in