research-article

Open access

AI and Holistic Review: Informing Human Reading in College Admissions

Authors:

anthony lising antonio,

Benjamin W. Domingue,

Ben Gebre-Medhin,

Mitchell L. StevensAuthors Info & Claims

AIES '20: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society

Pages 200 - 206

https://doi.org/10.1145/3375627.3375871

Published: 07 February 2020 Publication History

Abstract

College admissions in the United States is carried out by a human-centered method of evaluation known as holistic review, which typically involves reading original narrative essays submitted by each applicant. The legitimacy and fairness of holistic review, which gives human readers significant discretion over determining each applicant's fitness for admission, has been repeatedly challenged in courtrooms and the public sphere. Using a unique corpus of 283,676 application essays submitted to a large, selective, state university system between 2015 and 2016, we assess the extent to which applicant demographic characteristics can be inferred from application essays. We find a relatively interpretable classifier (logistic regression) was able to predict gender and household income with high levels of accuracy. Findings suggest that data auditing might be useful in informing holistic review, and perhaps other evaluative systems, by checking potential bias in human or computational readings.

References

[1]

Attali, Y., and Burstein, J. 2006. Automated essay scoring with e-rater® v. 2. The Journal of Technology, Learning and Assessment 4(3).

[2]

Bastedo, M. N.; Bowman, N. A.; Glasener, K. M.; and Kelly, J. L. 2018. What are we talking about when we talk about holistic review? selective college admissions and its effects on low-ses students. The Journal of Higher Education 89(5):782--805.

[3]

Bernstein, B. B. 2003. Class, codes and control: Applied studies towards a sociology of language, volume 2. Psychology Press.

[4]

Blei, D. M., and Lafferty, J. D. 2006. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, 113--120. ACM.

[5]

Bourdieu, P. 1973. Cultural reproduction and social reproduction. London: Tavistock 178.

[6]

Boyd, D., and Crawford, K. 2012. Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon. Information, communication & society 15(5):662--679.

[7]

Buolamwini, J., and Gebru, T. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, 77--91.

[8]

Celis, L. E.; Mehrotra, A.; and Vishnoi, N. K. 2019. Toward controlling discrimination in online ad auctions. In International Conference on Machine Learning.

[9]

Choudhary, R., and Gianey, H. K. 2017. Comprehensive review on supervised machine learning algorithms. In 2017 International Conference on Machine Learning and Data Science (MLDS), 37--43. IEEE.

[10]

Corbett-Davies, S., and Goel, S. 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023.

[11]

Cortes, C.; Mohri, M.; and Rostamizadeh, A. 2009. L 2 regularization for learning kernels. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, 109--116. AUAI Press.

[12]

Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

[13]

Dixon-Román, E. J.; Everson, H. T.; and McArdle, J. J. 2013. Race, poverty and sat scores: Modeling the influences of family income on black and white high school students' sat performance. Teachers College Record 115(4):1--33.

[14]

Dressel, J., and Farid, H. 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances 4(1):eaao5580.

[15]

Eckert, P., and McConnell-Ginet, S. 2013. Language and gender. Cambridge University Press.

[16]

Espenshade, T. J., and Radford, A. W. 2013. No longer separate, not yet equal: Race and class in elite college admission and campus life. Princeton University Press.

[17]

Espinosa, L. L.; Orfield, G.; and Gaertner, M. N. 2015. Race, class, and college access: Achieving diversity in a shifting legal landscape.

[18]

Fesler, L.; Dee, T.; Baker, R.; and Evans, B. 2019. CEPA Working Paper No. 19-04.

[19]

Fisher v. University of Texas, U. . 2016.

[20]

Garg, N.; Schiebinger, L.; Jurafsky, D.; and Zou, J. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences 115(16):E3635--E3644.

[21]

Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J.W.;Wallach, H.; Daumeé III, H.; and Crawford, K. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010.

[22]

Gratz v. Bollinger, 539 U.S. 244, . 2003.

[23]

Green, B., and Chen, Y. 2019. The principles and limits of algorithm-in-theloop decision making. Proceedings of the ACM on Human-Computer Interaction 3(CSCW):50.

[24]

Grutter v. Bollinger, 539 U.S. 306, . 2003.

[25]

Hartocollis, A. 2018. Harvard rated asian-american applicants lower on personality traits, suit says. New York Times (June 15, 2018).

[26]

Jurafsky, D., and Martin, J. H. 2014. Speech and language processing. Pearson.

[27]

Karabel, J. 2006. The chosen: The hidden history of admission and exclusion at Harvard, Yale, and Princeton. Houghton Mifflin Harcourt.

[28]

Kessler, M. D.; Yerges-Armstrong, L.; Taub, M. A.; Shetty, A. C.; Maloney, K.; Jeng, L. J. B.; Ruczinski, I.; Levin, A. M.; Williams, L. K.; Beaty, T. H.; et al. 2016. Challenges and disparities in the application of personalized genomic medicine to populations with african ancestry. Nature communications 7:12521.

[29]

Kiritchenko, S., and Mohammad, S. M. 2018. Examining gender and race bias in two hundred sentiment analysis systems. arXiv preprint arXiv:1805.04508.

[30]

Koppel, M.; Argamon, S.; and Shimoni, A. R. 2002. Automatically categorizing written texts by author gender. Literary and linguistic computing 17(4):401--412.

[31]

Lawton, D. 2006. Social class language and education. Routledge.

[32]

Loper, E., and Bird, S. 2002. Nltk: the natural language toolkit. arXiv preprint cs/0205028.

[33]

Lowry, S., and Macpherson, G. 1988. A blot on the profession. British medical journal (Clinical research ed.) 296(6623):657.

[34]

Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, 3111--3119.

[35]

Moody, J. 2019. 10 colleges that received the most applications.

[36]

Pennebaker, J.W.; Chung, C. K.; Frazee, J.; Lavergne, G. M.; and Beaver, D. I. 2014. When small words foretell academic success: The case of college admissions essays. PloS one 9(12):e115844.

[37]

Rivera, L. A. 2016. Pedigree: How elite students get elite jobs. Princeton University Press.

[38]

Rothstein, J. M. 2004. College performance predictions and the sat. Journal of Econometrics 121(1--2):297--317.

[39]

Rudinger, R.; May, C.; and Van Durme, B. 2017. Social bias in elicited natural language inferences. In Proceedings of the First ACLWorkshop on Ethics in Natural Language Processing, 74--79. Valencia, Spain: Association for Computational Linguistics.

[40]

Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15(1):1929--1958.

[41]

Stevens, M. L. 2009. Creating a class. Harvard University Press.

[42]

Students for Fair Admissions, I. v. P., and Fellows of Harvard College, C. A. N. .-c.-.-A. 2019.

[43]

Synnott, M. 2017. The half-opened door: Discrimination and admissions at Harvard, Yale, and Princeton, 1900--1970. Routledge.

[44]

Wechsler, H. S. 2017. The qualified student: A history of selective college admission in America. Routledge.

[45]

Zafar, M. B.; Valera, I.; Rodriguez, M. G.; and Gummadi, K. P. 2015. Fairness constraints: Mechanisms for fair classification. arXiv preprint arXiv:1507.05259.

Cited By

Chan SLo NWong A(2024)Enhancing university level English proficiency with generative AI: Empirical insights into automated feedback and learning outcomesContemporary Educational Technology10.30935/cedtech/1560716:4(ep541)Online publication date: 2024
https://doi.org/10.30935/cedtech/15607
Alvero ALee JRegla-Vargas AKizilcec RJoachims TAntonio A(2024)Large language models, social demography, and hegemony: comparing authorship in human and synthetic textJournal of Big Data10.1186/s40537-024-00986-711:1Online publication date: 27-Sep-2024
https://doi.org/10.1186/s40537-024-00986-7
Lee JHarvey EZhou JGarg NJoachims TKizilcec R(2024)Ending Affirmative Action Harms Diversity Without Improving Academic MeritProceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization10.1145/3689904.3694706(1-17)Online publication date: 29-Oct-2024
https://dl.acm.org/doi/10.1145/3689904.3694706
Show More Cited By

Index Terms

AI and Holistic Review: Informing Human Reading in College Admissions
1. Applied computing
  1. Education
  2. Law, social and behavioral sciences
    1. Sociology
2. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Classification and regression trees

Recommendations

Bias in peer review

Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look ...
People Perceive Algorithmic Assessments as Less Fair and Trustworthy Than Identical Human Assessments
CSCW

Algorithmic risk assessments are being deployed in an increasingly broad spectrum of domains including banking, medicine, and law enforcement. However, there is widespread concern about their fairness and trustworthiness, and people are also known to ...
Disambiguating Algorithmic Bias: From Neutrality to Justice
AIES '23: Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society

As algorithms have become ubiquitous in consequential domains, societal concerns about the potential for discriminatory outcomes have prompted urgent calls to address algorithmic bias. In response, a rich literature across computer science, law, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AIES '20: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society

February 2020

439 pages

ISBN:9781450371100

DOI:10.1145/3375627

General Chairs:
Annette Markham
Aarhus University | Loyola University
,
Julia Powles
University of Western Australia
,
Toby Walsh
TU Berlin | University of New South Wales | Data61
,
Anne L. Washington
New York University

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 February 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

AIES '20

Sponsor:

SIGAI

AIES '20: AAAI/ACM Conference on AI, Ethics, and Society

February 7 - 9, 2020

NY, New York, USA

Acceptance Rates

Overall Acceptance Rate 61 of 162 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
2,794
Total Downloads

Downloads (Last 12 months)755
Downloads (Last 6 weeks)61

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chan SLo NWong A(2024)Enhancing university level English proficiency with generative AI: Empirical insights into automated feedback and learning outcomesContemporary Educational Technology10.30935/cedtech/1560716:4(ep541)Online publication date: 2024
https://doi.org/10.30935/cedtech/15607
Alvero ALee JRegla-Vargas AKizilcec RJoachims TAntonio A(2024)Large language models, social demography, and hegemony: comparing authorship in human and synthetic textJournal of Big Data10.1186/s40537-024-00986-711:1Online publication date: 27-Sep-2024
https://doi.org/10.1186/s40537-024-00986-7
Lee JHarvey EZhou JGarg NJoachims TKizilcec R(2024)Ending Affirmative Action Harms Diversity Without Improving Academic MeritProceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization10.1145/3689904.3694706(1-17)Online publication date: 29-Oct-2024
https://dl.acm.org/doi/10.1145/3689904.3694706
Yung VColyvas J(2024)Munging the Ghosts in the MachineThe Oxford Handbook of the Sociology of Machine Learning10.1093/oxfordhb/9780197653609.013.3Online publication date: 22-May-2024
https://doi.org/10.1093/oxfordhb/9780197653609.013.3
Chan SLo NWong A(2024)Leveraging generative AI for enhancing university-level English writing: comparative insights on automated feedback and student engagementCogent Education10.1080/2331186X.2024.244018212:1Online publication date: 20-Dec-2024
https://doi.org/10.1080/2331186X.2024.2440182
Butt M(2024)The role of letters of recommendation in perpetuating or challenging the social stratification of American secondary schools: a quantitative analysis of admission officer assessments in highly selective college admissionDiscover Education10.1007/s44217-024-00175-x3:1Online publication date: 8-Jul-2024
https://doi.org/10.1007/s44217-024-00175-x
Van Busum KFang S(2024)Bias analysis of AI models for undergraduate student admissionsNeural Computing and Applications10.1007/s00521-024-10762-6Online publication date: 5-Dec-2024
https://doi.org/10.1007/s00521-024-10762-6
Lee HKizilcec RJoachims TSpikol DViberg OMartínez-Monés AGuo P(2023)Evaluating a Learned Admission-Prediction Model as a Replacement for Standardized Tests in College AdmissionsProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3593382(195-203)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3593382
Van Busum KFang SHong JLanperne MPark JCerny TShahriar H(2023)Analysis of AI Models for Student Admissions: A Case StudyProceedings of the 38th ACM/SIGAPP Symposium on Applied Computing10.1145/3555776.3577743(17-22)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3555776.3577743
Priyadarshini AMartinez-Neda BGago-Masague S(2023)Admission Prediction in Undergraduate Applications: an Interpretable Deep Learning Approach2023 Fifth International Conference on Transdisciplinary AI (TransAI)10.1109/TransAI60598.2023.00040(135-140)Online publication date: 25-Sep-2023
https://doi.org/10.1109/TransAI60598.2023.00040
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents