Investigating the Influence of Legal Case Retrieval Systems on Users' Decision Process
Authors: 
Beining Wang, Ruizhe Zhang, Yueyue Wu, 
Qingyao Ai, 
Min Zhang, 
Yiqun LiuAuthors Info & Claims
Beining Wang, Ruizhe Zhang, Yueyue Wu, Qingyao Ai, Min Zhang, Yiqun LiuAuthors Info & ClaimsPages 169 - 175
Abstract
Given a specific query case, legal case retrieval systems aim to retrieve a set of case documents relevant to the case at hand. Previous studies on user behavior analysis have shown that information retrieval (IR) systems can significantly influence users’ decisions by presenting results in varying orders and formats. However, whether such influence exists in legal case retrieval remains largely unknown. This study presents the first investigation into the influence of legal case retrieval systems on the decision-making process of legal users. We conducted an online user study involving more than ninety participants, and our findings suggest that the result distribution of legal case retrieval systems indeed affects users’ judgements on the sentences in cases. Notably, when users are presented with biased results that involve harsher sentences, they tend to impose harsher sentences on the current case as well. This research highlights the importance of optimizing the unbiasedness of legal case retrieval systems.
References
[1]
Faiz Ahamad. 2020. Impact of Online Job Search and Job Reviews on Job Decision. In Proceedings of the 13th International Conference on Web Search and Data Mining (Houston, TX, USA) (WSDM ’20). Association for Computing Machinery, New York, NY, USA, 909–910. https://doi.org/10.1145/3336191.3372184
[2]
Leif Azzopardi. 2021. Cognitive Biases in Search: A Review and Reflection of Cognitive Biases in Information Retrieval. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval (Canberra ACT, Australia) (CHIIR ’21). Association for Computing Machinery, New York, NY, USA, 27–37. https://doi.org/10.1145/3406522.3446023
[3]
Nick Bansback, Linda C. Li, Larry Lynd, and Stirling Bryan. 2014. Exploiting order effects to improve the quality of decisions. Patient Education and Counseling 96, 2 (2014), 197–203. https://doi.org/10.1016/j.pec.2014.05.021
[4]
Ransome Epie Bawack, Samuel Fosso Wamba, Kevin Daniel André Carillo, and Shahriar Akter. 2022. Artificial intelligence in E-Commerce: a bibliometric study and literature review. Electronic Markets 32, 1 (March 2022), 297–338. https://doi.org/10.1007/s12525-022-00537-
[5]
Ransome Epie Bawack, Samuel Fosso Wamba, Kevin Daniel André Carillo, and Shahriar Akter. 2022. Artificial intelligence in E-Commerce: a bibliometric study and literature review. Electronic Markets 32, 1 (March 2022), 297–338. https://doi.org/10.1007/s12525-022-00537-
[6]
BehimehrSara and JamaliHamid R.2020. Cognitive Biases and Their Effects on Information Behaviour of Graduate Students in Their Research Projects. Journal of Information Science Theory and Practice 8, 2 (06 2020), 18–31.
[7]
Dan Cosley, Shyong K. Lam, Istvan Albert, Joseph A. Konstan, and John Riedl. 2003. Is Seeing Believing? How Recommender System Interfaces Affect Users’ Opinions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA) (CHI ’03). Association for Computing Machinery, New York, NY, USA, 585–592. https://doi.org/10.1145/642611.642713
[8]
Robert Epstein and Ronald E. Robertson. 2015. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences 112, 33 (2015), E4512–E4521. https://doi.org/10.1073/pnas.1419828112 arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.1419828112
[9]
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9
[10]
Markus Kattenbeck and David Elsweiler. 2019. Understanding credibility judgements for web search snippets. Aslib J. Inf. Manag. 71 (2019), 368–391.
[11]
Silvia Knobloch-Westerwick, Benjamin K. Johnson, and Axel Westerwick. 2015. Confirmation Bias in Online Searches: Impacts of Selective Exposure Before an Election on Political Attitude Strength and Shifts. Journal of Computer-Mediated Communication 20, 2 (2015), 171–187. https://doi.org/10.1111/jcc4.12105 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/jcc4.12105
[12]
Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh, Krishna P. Gummadi, and Karrie Karahalios. 2017. Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW ’17). Association for Computing Machinery, New York, NY, USA, 417–432. https://doi.org/10.1145/2998181.2998321
[13]
Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh, Krishna P. Gummadi, and Karrie Karahalios. 2019. Search bias quantification: investigating political bias in social media and web search. Information retrieval (Boston) 22, 1-2 (2019), 188–227.
[14]
Annie Y S Lau and Enrico W Coiera. 2007. Do People Experience Cognitive Biases while Searching for Information?Journal of the American Medical Informatics Association 14, 5 (2007), 599–608.
[15]
Haitao Li, Yunqiu Shao, Yueyue Wu, Qingyao Ai, Yixiao Ma, and Yiqun Liu. 2023. LeCaRDv2. https://github.com/THUIR/LeCaRDv2
[16]
J. Z. Liu, K. Lars, and S. Holger. 2021. Precedents and Chinese Judges: An Experiment. The American Journal of Comparative Law1 (2021), 1.
[17]
Xitong Liu, Hui Fang, and Deng Cai. 2015. Towards Less Biased Web Search. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval (Northampton, Massachusetts, USA) (ICTIR ’15). Association for Computing Machinery, New York, NY, USA, 373–376. https://doi.org/10.1145/2808194.2809476
[18]
Yixiao Ma, Qingyao Ai, Yueyue Wu, Yunqiu Shao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2022. Incorporating Retrieval Information into the Truncation of Ranking Lists for Better Legal Search. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (Eds.). ACM, 438–448. https://doi.org/10.1145/3477495.3531998
[19]
Yixiao Ma, Yunqiu Shao, Yueyue Wu, Yiqun Liu, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2021. LeCaRD: A Legal Case Retrieval Dataset for Chinese Law System. In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (Eds.). ACM, 2342–2348. https://doi.org/10.1145/3404835.3463250
[20]
Alamir Novin and Eric Meyers. 2017. Making Sense of Conflicting Science Information: Exploring Bias in the Search Engine Result Page. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (Oslo, Norway) (CHIIR ’17). Association for Computing Machinery, New York, NY, USA, 175–184. https://doi.org/10.1145/3020165.3020185
[21]
BHAVIK Pathak, ROBERT GARFINKEL, RAM D. GOPAL, RAJKUMAR VENKATESAN, and FANG YIN. 2010. Empirical Analysis of the Impact of Recommender Systems on Sales. Journal of Management Information Systems 27, 2 (2010), 159–188. http://www.jstor.org/stable/29780174
[22]
Evaggelia Pitoura, Panayiotis Tsaparas, Giorgos Flouris, Irini Fundulaki, Panagiotis Papadakos, Serge Abiteboul, and Gerhard Weikum. 2018. On Measuring Bias in Online Information. SIGMOD Rec. 46, 4 (feb 2018), 16–21. https://doi.org/10.1145/3186549.3186553
[23]
Frances A. Pogacar, Amira Ghenai, Mark D. Smucker, and Charles L.A. Clarke. 2017. The Positive and Negative Influence of Search Results on People’s Decisions about the Efficacy of Medical Treatments. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (Amsterdam, The Netherlands) (ICTIR ’17). Association for Computing Machinery, New York, NY, USA, 209–216. https://doi.org/10.1145/3121050.3121074
[24]
Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Weizhi Ma, and Shaoping Ma. 2020. BERT-PLI: Modeling Paragraph-Level Interactions for Legal Case Retrieval. In Twenty-Ninth International Joint Conference on Artificial Intelligence and Seventeenth Pacific Rim International Conference on Artificial Intelligence IJCAI-PRICAI-20.
[25]
Yunqiu Shao, Yueyue Wu, Yiqun Liu, Jiaxin Mao, M. Zhang, and Shaoping Ma. 2021. Investigating User Behavior in Legal Case Retrieval. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (2021).
[26]
Adrian F. Ward. 2021. People mistake the internet’s knowledge for their own. Proceedings of the National Academy of Sciences 118, 43 (2021), e2105061118. https://doi.org/10.1073/pnas.2105061118 arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.2105061118
[27]
Luyan Xu, Mengdie Zhuang, and Ujwal Gadiraju. 2021. How Do User Opinions Influence Their Interaction With Web Search Results?. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (Utrecht, Netherlands) (UMAP ’21). Association for Computing Machinery, New York, NY, USA, 240–244. https://doi.org/10.1145/3450613.3456824
[28]
Ziwei Zhu, Jianling Wang, and James Caverlee. 2020. Measuring and Mitigating Item Under-Recommendation Bias in Personalized Ranking Systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20). Association for Computing Machinery, New York, NY, USA, 449–458. https://doi.org/10.1145/3397271.3401177
Index Terms
- Investigating the Influence of Legal Case Retrieval Systems on Users' Decision Process
Recommendations
An Intent Taxonomy of Legal Case Retrieval
Legal case retrieval is a special Information Retrieval (IR) task focusing on legal case documents. Depending on the downstream tasks of the retrieved case documents, users’ information needs in legal case retrieval could be significantly different from ...
Incorporating Structural Information into Legal Case Retrieval
Legal case retrieval has received increasing attention in recent years. However, compared to ad hoc retrieval tasks, legal case retrieval has its unique challenges. First, case documents are rather lengthy and contain complex legal structures. Therefore, ...
Understanding Relevance Judgments in Legal Case Retrieval
Legal case retrieval, which aims to retrieve relevant cases given a query case, has drawn increasing research attention in recent years. While much research has worked on developing automatic retrieval models, how to characterize relevance in this ...
Comments
Information & Contributors
Information
Published In
November 2023
324 pages
ISBN:9798400704086
DOI:10.1145/3624918
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 November 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SIGIR-AP '23
Sponsor:
SIGIR-AP '23: Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region
November 26 - 28, 2023
Beijing, China
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 149Total Downloads
- Downloads (Last 12 months)149
- Downloads (Last 6 weeks)19
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in