research-article

A Survey on Bias and Fairness in Machine Learning

Authors:

Ninareh Mehrabi,

Fred Morstatter,

Nripsuta Saxena,

Kristina Lerman,

Aram GalstyanAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 54, Issue 6

Article No.: 115, Pages 1 - 35

https://doi.org/10.1145/3457607

Published: 13 July 2021 Publication History

Abstract

With the widespread use of artificial intelligence (AI) systems and applications in our everyday lives, accounting for fairness has gained significant importance in designing and engineering of such systems. AI systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that these decisions do not reflect discriminatory behavior toward certain groups or populations. More recently some work has been developed in traditional machine learning and deep learning that address such challenges in different subdomains. With the commercialization of these systems, researchers are becoming more aware of the biases that these applications can contain and are attempting to address them. In this survey, we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and ways they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.

References

[1]

Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. 2019. Fair regression: Quantitative definitions and reduction-based algorithms. In Proceedings of the International Conference on Machine Learning. 120–129.

[2]

Sina Aghaei, Mohammad Javad Azizi, and Phebe Vayanos. 2019. Learning optimal and fair decision trees for non-discriminative decision-making. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 1418–1426.

Digital Library

[3]

Nazanin Alipourfard, Peter G. Fennell, and Kristina Lerman. 2018. Can you trust the trend? Discovering Simpson’s paradoxes in social data. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining. ACM, 19–27.

[4]

Nazanin Alipourfard, Peter G. Fennell, and Kristina Lerman. 2018. Using Simpson’s paradox to discover interesting patterns in behavioral data. In Proceedings of the 12th International AAAI Conference on Web and Social Media.

[5]

Alexander Amini, Ava P. Soleimany, Wilko Schwarting, Sangeeta N. Bhatia, and Daniela Rus. 2019. Uncovering and mitigating algorithmic bias through learned latent structure. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (AIES’19). Association for Computing Machinery, New York, NY, USA, 289–295. https://doi.org/10.1145/3306618.3314243

Digital Library

[6]

Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2019. Machine bias: There’s software used across the country to predict future criminals. and it’s biased against blacks. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.

[7]

A. Asuncion and D. J. Newman. 2007. UCI Machine Learning Repository. Retrieved from http://www.ics.uci.edu/$∖sim$mlearn/{MLR}epository.html.

[8]

Arturs Backurs, Piotr Indyk, Krzysztof Onak, Baruch Schieber, Ali Vakilian, and Tal Wagner. 2019. Scalable fair clustering. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.), Vol. 97. PMLR, 405–413. Retrieved from http://proceedings.mlr.press/v97/backurs19a.html.

[9]

Ricardo Baeza-Yates. 2018. Bias on the web. Commun. ACM 61, 6 (May 2018), 54–61.

Digital Library

[10]

Samuel Barbosa, Dan Cosley, Amit Sharma, and Roberto M Cesar Jr. 2016. Averaging gone wrong: Using time-aware analyses to better understand behavior. In Proceedings of the 25th International Conference on World Wide Web. 829–841.

Digital Library

[11]

Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic et al. 2018. AI fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943 (2018).

[12]

Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Trans. Assoc. Comput. Ling. 6 (2018), 587–604.

[13]

Misha Benjamin, Paul Gagnon, Negar Rostamzadeh, Chris Pal, Yoshua Bengio, and Alex Shee. 2019. Towards standardization of data licenses: The Montreal data license.

[14]

Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. 2017. A Convex Framework for Fair Regression. arxiv:cs.LG/1706.02409 (2017).

[15]

Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. 2021. Fairness in Criminal Justice Risk Assessments: The State of the Art. Sociological Methods & Research 50, 1 (2021), 3–44. https://doi.org/10.1177/0049124118782533

[16]

Peter J. Bickel, Eugene A. Hammel, and J. William O’Connell. 1975. Sex bias in graduate admissions: Data from Berkeley. Science 187, 4175 (1975), 398–404.

[17]

R. D. P. Binns. 2018. Fairness in machine learning: Lessons from political philosophy. J. Mach. Learn. Res. (2018).

[18]

Colin R. Blyth. 1972. On Simpson’s paradox and the sure-thing principle. J. Amer. Statist. Assoc. 67, 338 (1972), 364–366.

[19]

Miranda Bogen and Aaron Rieke. 2018. Help Wanted: An Examination of Hiring Algorithms, Equity and Bias. Technical Report. Upturn.

[20]

Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 4349–4357.

[21]

Shikha Bordia and Samuel Bowman. 2019. Identifying and reducing gender bias in word-level language models. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop. 7–15.

[22]

Avishek Bose and William Hamilton. 2019. Compositional fairness constraints for graph embeddings. In Proceedings of the International Conference on Machine Learning. 715–724.

[23]

Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. 2019. Understanding the origins of bias in word embeddings. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.), Vol. 97. PMLR, 803–811. Retrieved from http://proceedings.mlr.press/v97/brunet19a.html.

[24]

Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research), Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, 77–91. Retrieved from http://proceedings.mlr.press/v81/buolamwini18a.html.

[25]

Toon Calders and Sicco Verwer. 2010. Three naive Bayes approaches for discrimination-free classification. Data Mining Knowl. Discov. 21, 2 (2010), 277–292.

Digital Library

[26]

Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183–186.

[27]

Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R. Varshney. 2017. Optimized pre-processing for discrimination prevention. In Proceedings of the International Conference on Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 3992–4001. Retrieved from http://papers.nips.cc/paper/6988-optimized-pre-processing-for-discrimination-prevention.pdf.

[28]

Manel Capdevila, Marta Ferrer, and Eulália Luque. 2005. La reincidencia en el delito en la justicia de menores. Centro de Estudios jurídicos y formación especializada, Generalitat de Catalunya. Documento no publicado (2005).

[29]

Allison J. B. Chaney, Brandon M. Stewart, and Barbara E. Engelhardt. 2018. How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. In Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 224–232.

[30]

Jiahao Chen, Nathan Kallus, Xiaojie Mao, Geoffry Svacha, and Madeleine Udell. 2019. Fairness under unawareness: Assessing disparity when protected class is unobserved. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 339–348.

Digital Library

[31]

Xingyu Chen, Brandon Fain, Liang Lyu, and Kamesh Munagala. 2019. Proportionally fair clustering. In Proceedings of the International Conference on Machine Learning. 1032–1041.

[32]

S. Chiappa. 2019. Path-specific counterfactual fairness. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence. 7801–7808.

[33]

S. Chiappa and W. S. Isaac. 2019. A causal Bayesian networks viewpoint on fairness. In Privacy and Identity Management. Fairness, Accountability, and Transparency in the Age of Big Data. Privacy and Identity 2018. IFIP Advances in Information and Communication Technology, E. Kosta, J. Pierson, D. Slamanig, S. Fischer-Hübner, S. Krenn (Eds.,) Vol. 547. Springer, Cham.

[34]

Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data 5, 2 (2017), 153–163.

[35]

Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research), Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, 134–148. Retrieved from http://proceedings.mlr.press/v81/chouldechova18a.html.

[36]

Alexandra Chouldechova and Aaron Roth. 2018. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810 (2018).

[37]

John S. Chuang, Olivier Rivoire, and Stanislas Leibler. 2009. Simpson’s paradox in a synthetic microbial system. Science 323, 5911 (2009), 272–275.

[38]

Kevin A. Clarke. 2005. The phantom menace: Omitted variable bias in econometric research. Conflict Manag. Peace Sci. 22, 4 (2005), 341–352.

[39]

Lee Cohen, Zachary C. Lipton, and Yishay Mansour. 2019. Efficient candidate screening under multiple tests and implications for fairness. arxiv:cs.LG/1905.11361 (2019).

[40]

United States. Equal Employment Opportunity Commission. 1987. EEOC Compliance Manual. U.S. Equal Employment Opportunity Commission, Washington, D.C.

[41]

Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 797–806.

Digital Library

[42]

Elliot Creager, David Madras, Joern-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. 2019. Flexibly fair representation learning by disentanglement. In Proceedings of the International Conference on Machine Learning. 1436–1445.

[43]

Brian d’Alessandro, Cathy O’Neil, and Tom LaGatta. 2017. Conscientious classification: A data scientist’s guide to discrimination-aware classification. Big Data 5, 2 (2017), 120–134.

[44]

David Danks and Alex John London. 2017. Algorithmic bias in autonomous systems. In Proceedings of the International Joint Conference on Artificial Intelligence. 4691–4697.

[45]

Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso. 2011. Extraneous factors in judicial decisions. Proc. Nat. Acad. Sci. 108, 17 (2011), 6889–6892.

[46]

Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Sci. Adv. 4, 1 (2018).

[47]

Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. Retrieved from http://archive.ics.uci.edu/ml.

[48]

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (ITCS’12). ACM, New York, NY, 214–226.

Digital Library

[49]

Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. 2018. Decoupled classifiers for group-fair and efficient machine learning. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research), Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, 119–133. Retrieved from http://proceedings.mlr.press/v81/dwork18a.html.

[50]

Golnoosh Farnadi, Behrouz Babaki, and Lise Getoor. 2018. Fairness in relational domains. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES’18). ACM, New York, NY, 108–114.

Digital Library

[51]

Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’15). Association for Computing Machinery, New York, NY, 259–268.

Digital Library

[52]

Joel Escudé Font and Marta R. Costa-Jussà. 2019. Equalizing gender biases in neural machine translation with word embeddings techniques. arXiv preprint arXiv:1901.03116 (2019).

[53]

Batya Friedman and Helen Nissenbaum. 1996. Bias in computer systems. ACM Trans. Inf. Syst. 14, 3 (July 1996), 330–347.

Digital Library

[54]

Anna Fry, Thomas J. Littlejohns, Cathie Sudlow, Nicola Doherty, Ligia Adamska, Tim Sprosen, Rory Collins, and Naomi E. Allen. 2017. Comparison of sociodemographic and health-related characteristics of UK biobank participants with those of the general population. Amer. J. Epidem. 186, 9 (06 2017), 1026–1034.

[55]

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets.

[56]

C. E. Gehlke and Katherine Biehl. 1934. Certain effects of grouping upon the size of the correlation coefficient in census tract material. J. Amer. Statist. Assoc. 29, 185A (1934), 169–170.

[57]

Naman Goel, Mohammad Yaghini, and Boi Faltings. 2018. Non-discriminatory machine learning through convex fairness criteria. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence.

Digital Library

[58]

Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. arXiv preprint arXiv:1903.03862 (2019).

[59]

Sandra González-Bailón, Ning Wang, Alejandro Rivero, Javier Borge-Holthoefer, and Yamir Moreno. 2014. Assessing the bias in samples of large online networks. Soc. Netw. 38 (2014), 16–27.

[60]

Susan T. Gooden. 2015. Race and Social Equity: A Nervous Area of Government. Routledge.

[61]

Nina Grgic-Hlaca, Muhammad Bilal Zafar, Krishna P. Gummadi, and Adrian Weller. 2016. The case for process fairness in learning: Feature selection for fair decision making. In Proceedings of the NIPS Symposium on Machine Learning and the Law, Vol. 1. 2.

[62]

S. Hajian and J. Domingo-Ferrer. 2013. A methodology for direct and indirect discrimination prevention in data mining. IEEE Trans. Knowl. Data Eng. 25, 7 (July 2013), 1445–1459.

Digital Library

[63]

Moritz Hardt, Eric Price, Nati Srebro et al. 2016. Equality of opportunity in supervised learning. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 3315–3323.

[64]

Eszter Hargittai. 2007. Whose space? Differences among users and non-users of social network sites. J. Comput.-mediat. Commun. 13, 1 (10 2007), 276–297.

[65]

Yuzi He, Keith Burghardt, and Kristina Lerman. 2020. A geometric solution to fair representations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 279–285.

Digital Library

[66]

Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677 (2018).

[67]

Ayanna Howard and Jason Borenstein. 2018. The ugly truth about ourselves and our robot creations: The problem of bias and social inequity. Sci. Eng. Ethics 24, 5 (2018), 1521–1536.

[68]

Gary B. Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition, Erik Learned-Miller and Andras Ferencz and Frédéric Jurie, Marseille, France. https://hal.inria.fr/inria-00321923.

[69]

Lingxiao Huang and Nisheeth Vishnoi. 2019. Stable and fair classification. In Proceedings of the International Conference on Machine Learning. 2879–2890.

[70]

Ben Hutchinson and Margaret Mitchell. 2019. 50 Years of test (Un) fairness: Lessons for machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 49–58.

Digital Library

[71]

L. Introna and H. Nissenbaum. 2000. Defining the Web: The politics of search engines. Computer 33, 1 (Jan. 2000), 54–62.

Digital Library

[72]

Ayush Jaiswal, Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan. 2018. Unsupervised Adversarial Invariance. arxiv:cs.LG/1809.10083 (2018).

[73]

Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. 2020. Wasserstein fair classification. In Uncertainty in Artificial Intelligence. PMLR, 862–872.

[74]

F. Kamiran and T. Calders. 2009. Classifying without discriminating. In Proceedings of the 2nd International Conference on Computer, Control and Communication. 1–6.

[75]

Faisal Kamiran and Toon Calders. 2010. Classification with no discrimination by preferential sampling. In Proceedings of the 19th Machine Learning Conference. Citeseer, 1–6.

[76]

Faisal Kamiran and Toon Calders. 2012. Data preprocessing techniques for classification without discrimination. Knowl. Inf. Syst. 33, 1 (01 Oct. 2012), 1–33.

[77]

Faisal Kamiran and Indrė Žliobaitė. 2013. Explainable and Non-explainable Discrimination in Classification. Springer Berlin, 155–170.

[78]

Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2012. Fairness-aware classifier with prejudice remover regularizer. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 35–50.

[79]

Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In Proceedings of the International Conference on Machine Learning. 2569–2577.

[80]

Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2019. An empirical study of rich subgroup fairness for machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 100–109.

Digital Library

[81]

Rogier Kievit, Willem Eduard Frankenhuis, Lourens Waldorp, and Denny Borsboom. 2013. Simpson’s paradox in psychological science: A practical guide. Front. Psychol. 4 (2013), 513.

[82]

Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. 2017. Avoiding discrimination through causal reasoning. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 656–666.

[83]

Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807 (2016).

[84]

Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the MT Summit, Vol. 5. 79–86.

[85]

Emmanouil Krasanakis, Eleftherios Spyromitros-Xioufis, Symeon Papadopoulos, and Yiannis Kompatsiaris. 2018. Adaptive sensitive reweighting to mitigate bias in fairness-aware classification. In Proceedings of the World Wide Web Conference (WWW’18). International World Wide Web Conferences Steering Committee, 853–862.

Digital Library

[86]

Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit et al. 2017. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Retrieved from https://github.com/openimages.

[87]

Matt J. Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. In Proceedings of the International Conference on Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4066–4076. Retrieved from http://papers.nips.cc/paper/6995-counterfactual-fairness.pdf.

[88]

Anja Lambrecht and Catherine Tucker. 2019. Algorithmic bias? an empirical study of apparent gender-based discrimi-nation in the display of stem career ads. Management Science 65, 7 (2019), 2966–2981.

[89]

J. Larson, S. Mattu, L. Kirchner, and J. Angwin. 2016. Compas analysis. Retrieved from https://github.com/propublica/compas-analysis.

[90]

Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 335–340.

Digital Library

[91]

Kristina Lerman. 2018. Computational social scientist beware: Simpson’s paradox in behavioral data. J. Comput. Soc. Sci. 1, 1 (2018), 49–58.

[92]

Kristina Lerman and Tad Hogg. 2014. Leveraging position bias to improve peer recommendation. PLoS One 9, 6 (2014), e98914. Retrieved from http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.009891%4.

[93]

Zachary C. Lipton, Alexandra Chouldechova, and Julian McAuley. 2017. Does mitigating ML’s disparate impact require disparate treatment?stat 1050 (2017), 19.

[94]

Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed impact of fair machine learning. In Proceedings of the 35th International Conference on Machine Learning.

[95]

Joshua R. Loftus, Chris Russell, Matt J. Kusner, and Ricardo Silva. 2018. Causal reasoning for algorithmic fairness. arXiv preprint arXiv:1805.05859 (2018).

[96]

Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. 2016. The variational fair autoencoder. stat 1050 (2016), 4.

[97]

Arjun K. Manrai, Birgit H. Funke, Heidi L. Rehm, Morten S. Olesen, Bradley A. Maron, Peter Szolovits, David M. Margulies, Joseph Loscalzo, and Isaac S. Kohane. 2016. Genetic misdiagnoses and the potential for health disparities. New Eng. J. Med. 375, 7 (2016), 655–665.

[98]

Ray Marshall. 1974. The economics of racial discrimination: A survey. J. Econ. Lit. 12, 3 (1974), 849–871.

[99]

Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. arXiv preprint arXiv:1903.10561 (2019).

[100]

Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2019. Man is to person as woman is to location: Measuring gender bias in named entity recognition. arXiv preprint arXiv:1910.10872 (2019).

[101]

Ninareh Mehrabi, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2019. Debiasing community detection: The importance of lowly-connected nodes. arXiv preprint arXiv:1903.08136 (2019).

[102]

Aditya Krishna Menon and Robert C. Williamson. 2018. The cost of fairness in binary classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research), Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, 107–118. Retrieved from http://proceedings.mlr.press/v81/menon18a.html.

[103]

Michele Merler, Nalini Ratha, Rogerio S. Feris, and John R. Smith. 2019. Diversity in faces. arXiv preprint arXiv:1901.10436 (2019).

[104]

Hannah Jean Miller, Jacob Thebault-Spieker, Shuo Chang, Isaac Johnson, Loren Terveen, and Brent Hecht. 2016. “Blissfully Happy” or “Ready to Fight”: Varying interpretations of emoji. In Proceedings of the 10th International AAAI Conference on Web and Social Media.

[105]

I. Minchev, G. Matijevic, D. W. Hogg, G. Guiglion, M. Steinmetz, F. Anders, C. Chiappini, M. Martig, A. Queiroz, and C. Scannapieco. 2019. Yule-Simpson’s paradox in galactic archaeology. arXiv preprint arXiv:1902.01421 (2019).

[106]

Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT’19). ACM, New York, NY, 220–229.

Digital Library

[107]

Fred Morstatter, Jürgen Pfeffer, Huan Liu, and Kathleen M. Carley. 2013. Is the sample good enough? Comparing data from Twitter’s streaming API with Twitter’s Firehose. In Proceedings of the 7th International AAAI Conference on Weblogs and Social Media. AAAI Press.

[108]

Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. 2018. Invariant representations without adversarial training. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 9084–9093.

[109]

Amitabha Mukerjee, Rita Biswas, Kalyanmoy Deb, and Amrit P. Mathur. 2002. Multi–objective evolutionary algorithms for the risk–return trade–off in bank loan management. Int. Trans. Oper. Res. 9, 5 (2002), 583–597.

[110]

David B. Mustard. 2003. Reexamining criminal behavior: The importance of omitted variable bias. Rev. Econ. Statist. 85, 1 (2003), 205–211.

[111]

Razieh Nabi, Daniel Malinsky, and Ilya Shpitser. 2018. Learning optimal fair policies. arXiv preprint arXiv:1809.02244 (2018).

[112]

Razieh Nabi and Ilya Shpitser. 2018. Fair inference on outcomes. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence.

[113]

Azadeh Nematzadeh, Giovanni Luca Ciampaglia, Filippo Menczer, and Alessandro Flammini. 2017. How algorithmic popularity bias hinders or promotes quality. arXiv preprint arXiv:1707.00574 (2017).

[114]

Dong-Phuong Nguyen, Rilana Gravel, Rudolf Berend Trieschnigg, and Theo Meder. 2013. “How old do you think I am?”: A study of language and age in Twitter. In Proceedings of the 7th International AAAI Conference on Weblogs and Social Media (ICWSM ’13). AAAI Press, 439–448.

[115]

Anne O’Keeffe and Michael McCarthy. 2010. The Routledge Handbook of Corpus Linguistics. Routledge.

[116]

Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kıcıman. 2019. Social data: Biases, methodological pitfalls, and ethical boundaries. Frontiers in Big Data 2 (2019), 13.

[117]

Cathy O’Neil. 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown Publishing Group, New York, NY.

Digital Library

[118]

Luca Oneto, Michele Doninini, Amon Elders, and Massimiliano Pontil. 2019. Taking advantage of multitask learning for fair classification. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 227–237.

Digital Library

[119]

Osonde A. Osoba and William Welser IV. 2017. An Intelligence in Our Image: The Risks of Bias and Errors in Artificial Intelligence. Rand Corporation.

[120]

Edmund S. Phelps. 1972. The statistical theory of racism and sexism. Amer. Econ. Rev. 62, 4 (1972), 659–661.

[121]

Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q. Weinberger. 2017. On fairness and calibration. In Proceedings of the International Conference on Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 5680–5689. Retrieved from http://papers.nips.cc/paper/7151-on-fairness-and-calibration.pdf.

[122]

Marcelo O. R. Prates, Pedro H. Avelar, and Luis C. Lamb. 2019. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications (2019), 1–19.

[123]

Bilal Qureshi, Faisal Kamiran, Asim Karim, and Salvatore Ruggieri. 2016. Causal discrimination discovery through propensity score analysis. arXiv preprint arXiv:1608.03735 (2016).

[124]

Inioluwa Deborah Raji and Joy Buolamwini. 2019. Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (AIES’19). Association for Computing Machinery, New York, NY, USA, 429–435. https://doi.org/10.1145/3306618.3314244

Digital Library

[125]

M. Redmond. 2011. Communities and crime unnormalized dataset. UCI Machine Learning Repository : Retrieved from http://www.ics.uci.edu/mlearn/MLRepository.html.

[126]

Willy E. Rice. 1996. Race, gender, redlining, and the discriminatory access to loans, credit, and insurance: An historical and empirical analysis of consumers who sued lenders and insurers in federal and state courts, 1950–1995. San Diego L. Rev. 33 (1996), 583.

[127]

Stephanie K. Riegg. 2008. Causal inference and omitted variable bias in financial aid research: Assessing solutions. Rev. Higher Educ. 31, 3 (2008), 329–354.

[128]

Lauren A. Rivera. 2012. Hiring as cultural matching: The case of elite professional service firms. Amer. Sociol. Rev. 77, 6 (2012), 999–1022.

[129]

Andrea Romei and Salvatore Ruggieri. 2014. A multidisciplinary survey on discrimination analysis. The Knowledge Engineering Review 29, 5 (2014), 582–638.

[130]

Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics, 8–14.

[131]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein et al. 2015. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 3 (2015), 211–252.

Digital Library

[132]

Pedro Saleiro, Benedict Kuester, Abby Stevens, Ari Anisfeld, Loren Hinkson, Jesse London, and Rayid Ghani. 2018. Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577 (2018).

[133]

Samira Samadi, Uthaipon Tantipongpipat, Jamie Morgenstern, Mohit Singh, and Santosh Vempala. 2018. The price of fair PCA: One extra dimension. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS’18). Curran Associates Inc., 10999–11010. Retrieved from http://dl.acm.org/citation.cfm?id=3327546.3327755.

[134]

Nripsuta Ani Saxena. 2019. Perceptions of fairness. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES’19). ACM, New York, NY, 537–538.

Digital Library

[135]

Nripsuta Ani Saxena, Karen Huang, Evan DeFilippis, Goran Radanovic, David C. Parkes, and Yang Liu. 2019. How do fairness definitions fare?: Examining public attitudes towards algorithmic definitions of fairness. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. ACM, 99–106.

Digital Library

[136]

Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as treatments: Debiasing learning and evaluation. In Proceedings of the International Conference on Machine Learning. 1670–1679.

[137]

Andrew D. Selbst, Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 59–68.

Digital Library

[138]

Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley. 2017. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. stat 1050 (2017), 22.

[139]

Richard Shaw and Manuel Corpas. [n.d.]. Further bias in personal genomics? ([n. d.]).

[140]

Harini Suresh and John V. Guttag. 2019. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002 (2019).

[141]

Songül Tolan, Marius Miron, Emilia Gómez, and Carlos Castillo. 2019. Why machine learning may lead to unfairness: evidence from risk assessment for juvenile justice in catalonia. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law (ICAIL’19). Association for Computing Machinery, New York, NY, USA, 83–92. https://doi.org/10.1145/3322640.3326705

Digital Library

[142]

Zeynep Tufekci. 2014. Big questions for social media big data: Representativeness, validity and other methodological pitfalls. In Proceedings of the 8th International AAAI Conference on Weblogs and Social Media.

[143]

Berk Ustun, Yang Liu, and David Parkes. 2019. Fairness without harm: Decoupled classifiers with preference guarantees. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.), Vol. 97. PMLR, 6373–6382. Retrieved from http://proceedings.mlr.press/v97/ustun19a.html.

[144]

Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 3003–3008.

[145]

Sahil Verma and Julia Rubin. 2018. Fairness definitions explained. In Proceedings of the IEEE/ACM International Workshop on Software Fairness (FairWare’18). IEEE, 1–7.

Digital Library

[146]

Selwyn Vickers, Mona Fouad, and Moon S. Chen Jr. 2014. Enhancing minority participation in clinical trials (EMPaCT): Laying the groundwork for improving minority clinical trial accrual. Cancer 120 (2014), vi–vii.

[147]

Ting Wang and Dashun Wang. 2014. Why Amazon’s ratings might mislead you: The story of herding effects. Big Data 2, 4 (2014), 196–204.

[148]

Steven L. Willborn. 1984. The disparate impact model of discrimination: Theory and limits. Amer. UL Rev. 34 (1984), 799.

[149]

Christo Wilson, Bryce Boe, Alessandra Sala, Krishna P. N. Puttaswamy, and Ben Y. Zhao. 2009. User interactions in social networks and their implications. In Proceedings of the 4th ACM European Conference on Computer Systems. ACM, 205–218.

[150]

Blake Woodworth, Suriya Gunasekar, Mesrob I. Ohannessian, and Nathan Srebro. 2017. Learning non-discriminatory predictors. arXiv preprint arXiv:1702.06081 (2017).

[151]

Yongkai Wu, Lu Zhang, and Xintao Wu. 2018. Fairness-aware Classification: Criterion, Convexity, and Bounds. arxiv:cs.LG/1809.04737 (2018).

[152]

Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. 2018. FairGAN: Fairness-aware generative adversarial networks. In Proceedings of the IEEE International Conference on Big Data (Big Data’18). IEEE, 570–575.

[153]

Irene Y. Chen, Peter Szolovits, and Marzyeh Ghassemi. 2019. Can AI help reduce disparities in general medical and mental health care?AMA J. Ethics 21 (02 2019), E167–179.

[154]

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. 2017. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th International Conference on World Wide Web. 1171–1180.

[155]

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. 2015. Fairness constraints: Mechanisms for fair classification. arXiv preprint arXiv:1507.05259 (2015).

[156]

Lu Zhang and Xintao Wu. 2017. Anti-discrimination learning: A causal modeling-based framework. Int. J. Data Sci. Analyt. 4, 1 (01 Aug. 2017), 1–16.

[157]

Lu Zhang, Yongkai Wu, and Xintao Wu. 2016. On discrimination discovery using causal networks. In Social, Cultural, and Behavioral Modeling, Kevin S. Xu, David Reitter, Dongwon Lee, and Nathaniel Osgood (Eds.). Springer International Publishing, Cham, 83–93.

[158]

Lu Zhang, Yongkai Wu, and Xintao Wu. 2016. Situation testing-based discrimination discovery: A causal inference approach. In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI’16). AAAI Press, 2718–2724. Retrieved from http://dl.acm.org/citation.cfm?id=3060832.3061001.

[159]

Lu Zhang, Yongkai Wu, and Xintao Wu. 2017. Achieving non-discrimination in data release. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1335–1344.

Digital Library

[160]

Lu Zhang, Yongkai Wu, and Xintao Wu. 2017. A causal framework for discovering and removing direct and indirect discrimination. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI’17). 3929–3935.

[161]

L. Zhang, Y. Wu, and X. Wu. 2018. Causal modeling-based discrimination discovery and removal: Criteria, bounds, and algorithms. IEEE Trans. Knowl. Data Eng. (2018), 1–1.

[162]

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 629–634.

[163]

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.

[164]

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. arxiv:cs.CL/1804.06876 (2018).

[165]

Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018. Learning gender-neutral word embeddings. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 4847–4853.

[166]

James Zou and Londa Schiebinger. 2018. AI can be sexist and racist it’s time to make it fair. Nature Publishing Group.

Cited By

Basubrin O(2025)Current Status and Future of Artificial Intelligence in MedicineCureus10.7759/cureus.77561Online publication date: 16-Jan-2025
https://doi.org/10.7759/cureus.77561
Navarro G(2025)Fair and Ethical Resume Screening: Enhancing ATS with JustScreen the ResumeScreeningAppJournal of Information Technology, Cybersecurity, and Artificial Intelligence10.70715/jitcai.2024.v2.i1.0012:1(1-7)Online publication date: 15-Jan-2025
https://doi.org/10.70715/jitcai.2024.v2.i1.001
Hutzschenreuter TLämmermann T(2025)What Is Your AI Strategy? Systematically Integrating Self-Learning Technologies into Your Business StrategyAcademy of Management Perspectives10.5465/amp.2023.0243Online publication date: 7-Jan-2025
https://doi.org/10.5465/amp.2023.0243
Show More Cited By

Index Terms

A Survey on Bias and Fairness in Machine Learning
1. Computing methodologies
  1. Artificial intelligence

Recommendations

Deep learning, reinforcement learning, and world models
Abstract
Deep learning (DL) and reinforcement learning (RL) methods seem to be a part of indispensable factors to achieve human-level or super-human AI systems. On the other hand, both DL and RL have strong connections with our brain functions ...
Artificial learning companionusing machine learning and natural language processing
Abstract
Artificial Intelligence, also referred to as AI, is one of the most rapidly evolving branches of Computer Science. The two branches of AI which empowers it to understand and interact with humans are Machine Learning (ML) and Natural Language ...
Representation Learning: A Review and New Perspectives

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 54, Issue 6

Invited Tutorial

July 2022

799 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/3475936

Editor:
Albert Zomaya
University of Sydney, Australia

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2021

Accepted: 01 March 2021

Revised: 01 January 2021

Received: 01 September 2019

Published in CSUR Volume 54, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Defense Advanced Research Projects Agency (DARPA)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1,663
Total Citations
View Citations
35,741
Total Downloads

Downloads (Last 12 months)10,166
Downloads (Last 6 weeks)901

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Basubrin O(2025)Current Status and Future of Artificial Intelligence in MedicineCureus10.7759/cureus.77561Online publication date: 16-Jan-2025
https://doi.org/10.7759/cureus.77561
Navarro G(2025)Fair and Ethical Resume Screening: Enhancing ATS with JustScreen the ResumeScreeningAppJournal of Information Technology, Cybersecurity, and Artificial Intelligence10.70715/jitcai.2024.v2.i1.0012:1(1-7)Online publication date: 15-Jan-2025
https://doi.org/10.70715/jitcai.2024.v2.i1.001
Hutzschenreuter TLämmermann T(2025)What Is Your AI Strategy? Systematically Integrating Self-Learning Technologies into Your Business StrategyAcademy of Management Perspectives10.5465/amp.2023.0243Online publication date: 7-Jan-2025
https://doi.org/10.5465/amp.2023.0243
Tarabah N(2025)Enhancing ADHD Support Through a Human-AI Collaboration FrameworkTransforming Neuropsychology and Cognitive Psychology With AI and Machine Learning10.4018/979-8-3693-9341-3.ch016(395-432)Online publication date: 17-Jan-2025
https://doi.org/10.4018/979-8-3693-9341-3.ch016
Andrae S(2025)Fairness and Bias in Machine Learning Models for Credit DecisionsMachine Learning and Modeling Techniques in Financial Data Science10.4018/979-8-3693-8186-1.ch001(1-24)Online publication date: 31-Jan-2025
https://doi.org/10.4018/979-8-3693-8186-1.ch001
Volosevici D(2025)Complexities and Legal Challenges of Fostering Diversity in the WorkplaceDimensions of Diversity, Equity, Inclusion, and Belonging in Business10.4018/979-8-3693-3876-6.ch014(359-386)Online publication date: 10-Jan-2025
https://doi.org/10.4018/979-8-3693-3876-6.ch014
Santos J(2025)Emerging Paradigms in Non-Profit GovernanceNew Trends for the Governance of Non-Profit Organizations10.4018/979-8-3693-3723-3.ch001(1-86)Online publication date: 10-Jan-2025
https://doi.org/10.4018/979-8-3693-3723-3.ch001
Khan SShi TDonato-Woodger SChu C(2025)Mitigating Digital Ageism in Skin Lesion Detection with Adversarial LearningAlgorithms10.3390/a1802005518:2(55)Online publication date: 21-Jan-2025
https://doi.org/10.3390/a18020055
Orynbay LBekmanova GYergesh BOmarbekova ASairanbekova ASharipbay A(2025)The role of cognitive computing in NLPFrontiers in Computer Science10.3389/fcomp.2024.14865816Online publication date: 10-Jan-2025
https://doi.org/10.3389/fcomp.2024.1486581
Sanjay Kumar Gorai Anurag Sarangi Shekhar Pradhan (2025)Deep Learning for Image Classification: Methods, Challenges, and Future DirectionsInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology10.32628/CSEIT251111011:1(484-496)Online publication date: 13-Jan-2025
https://doi.org/10.32628/CSEIT2511110
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents