research-article

Open access

Characterizing Manipulation from AI Systems

Authors:

David KruegerAuthors Info & Claims

EAAMO '23: Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization

Article No.: 6, Pages 1 - 13

https://doi.org/10.1145/3617694.3623226

Published: 30 October 2023 Publication History

All formats PDF

Abstract

Manipulation is a concern in many domains, such as social media, advertising, and chatbots. As AI systems mediate more of our digital interactions, it is important to understand the degree to which AI systems might manipulate humans without the intent of the system designers. Our work clarifies challenges in defining and measuring this kind of manipulation from AI systems. Firstly, we build upon prior literature on manipulation and characterize the space of possible notions of manipulation, which we find to depend upon the concepts of incentives, intent, covertness, and harm. We review proposals on how to operationalize each concept and we outline challenges in including each concept in a definition of manipulation. Second, we discuss the connections between manipulation and related concepts, such as deception and coercion. We then analyze how our characterization of manipulation applies to recommender systems and language models, and give a brief overview of the regulation of manipulation in other domains. While some progress has been made in defining and measuring manipulation from AI systems, many gaps remain. In the absence of a consensus definition and reliable tools for measurement, we cannot rule out the possibility that AI systems learn to manipulate humans without the intent of the system designers. Manipulation could pose a significant threat to human autonomy and precautionary actions to mitigate it are likely warranted.

References

[1]

Gediminas Adomavicius, Jesse C. Bockstedt, Shawn P. Curley, and Jingjing Zhang. 2013. Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects. Information Systems Research 24, 4 (Dec. 2013), 956–975. https://doi.org/10.1287/isre.2013.0497 Publisher: INFORMS.

Digital Library

[2]

Gediminas Adomavicius, Jesse C. Bockstedt, Shawn P. Curley, and Jingjing Zhang. 2018. Effects of Online Recommendations on Consumers’ Willingness to Pay. Information Systems Research 29, 1 (March 2018), 84–102. https://doi.org/10.1287/isre.2017.0703

Digital Library

[3]

M. Mehdi Afsar, Trafford Crump, and Behrouz Far. 2021. Reinforcement learning based recommender systems: A survey. arXiv:2101.06286 [cs] (Jan. 2021). http://arxiv.org/abs/2101.06286 arXiv:2101.06286.

[4]

Hunt Allcott, Luca Braghieri, Sarah Eichmeyer, and Matthew Gentzkow. 2020. The Welfare Effects of Social Media. American Economic Review 110, 3 (March 2020), 629–676. https://doi.org/10.1257/aer.20190658

[5]

Hunt Allcott, Matthew Gentzkow, and Lena Song. 2022. Digital Addiction. American Economic Review 112, 7 (July 2022), 2424–2463. https://doi.org/10.1257/aer.20210867

[6]

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete Problems in AI Safety. arXiv:1606.06565 [cs] (July 2016). http://arxiv.org/abs/1606.06565 arXiv:1606.06565.

[7]

Jacob Andreas. 2022. Language Models as Agent Models. https://doi.org/10.48550/arXiv.2212.01681 arXiv:2212.01681 [cs].

[8]

Stuart Armstrong. 2015. Motivated Value Selection for Artiﬁcial Agents. (2015).

[9]

Muhammad Ashfaq, Jiang Yun, Shubin Yu, and Sandra Maria Correia Loureiro. 2020. I, Chatbot: Modeling the determinants of users’ satisfaction and continuance intention of AI-powered service agents. Telematics and Informatics 54 (Nov. 2020), 101473. https://doi.org/10.1016/j.tele.2020.101473

[10]

Hal Ashton. 2022. Definitions of Intent Suitable for Algorithms. Artificial Intelligence and Law (July 2022). https://doi.org/10.1007/s10506-022-09322-x

Digital Library

[11]

Hal Ashton and Matija Franklin. 2022. The Problem of Behaviour and Preference Manipulation in AI Systems. In The AAAI-22 Workshop on Artificial Intelligence Safety (SafeAI 2022).

[12]

Hal Ashton and Matija Franklin. 2022. Solutions to Preference Manipulation in Recommender Systems Require Knowledge of Meta-Preferences. http://arxiv.org/abs/2209.11801 arXiv:2209.11801 [cs].

[13]

Association for Computing Machinery (ACM). 2019. "Reinforcement Learning for Recommender Systems: A Case Study on Youtube," by Minmin Chen. https://www.youtube.com/watch?v=HEqQ2_1XRTs

[14]

Financial Conduct AuthorityA. 2016. FCA Handbook: MAR 1 Market Abuse. https://www.handbook.fca.org.uk/handbook/MAR.pdf

[15]

Alessio Azzutti. 2022. AI-driven Market Manipulation and Limits of the EU Law Enforcement Regime to Credible Deterrence. Computer Law & Security review 45 (Jan. 2022). https://doi.org/10.2139/ssrn.4026468

[16]

Alessio Azzutti, Wolf-Georg Ringe, and H. Siegfried Stiehl. 2021. Machine Learning, Market Manipulation and Collusion on Capital Markets: Why the. University of Pennsylvania journal of international law 43, 1 (2021). https://doi.org/10.2139/ssrn.3788872

[17]

Hui Bai. 2023. Artificial Intelligence Can Persuade Humans. (2023).

[18]

Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3411764.3445717

Digital Library

[19]

Marcia Baron. 2014. The Mens Rea and Moral Status of Manipulation. In Manipulation: Theory and Practice, Christian Coons and Michael Weber (Eds.). Oxford University Press, 0. https://doi.org/10.1093/acprof:oso/9780199338207.003.0005

[20]

Jonathan Bassen, Bharathan Balaji, Michael Schaarschmidt, Candace Thille, Jay Painter, Dawn Zimmaro, Alex Games, Ethan Fast, and John C. Mitchell. 2020. Reinforcement Learning for the Adaptive Scheduling of Educational Activities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–12. https://doi.org/10.1145/3313831.3376518

Digital Library

[21]

Yavar Bathaee. 2018. The Artificial Intelligence Black Box and the Failure of Intent and Causation. Harvard Journal of Law and Technology 31, 2 (2018), 890–938.

[22]

Omer Ben-Porat and Moshe Tennenholtz. 2018. A Game-Theoretic Approach to Recommendation Systems with Strategic Content Providers. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Vol. 31. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2018/file/a9a1d5317a33ae8cef33961c34144f84-Paper.pdf

[23]

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency(FAccT ’21). Association for Computing Machinery, New York, NY, USA, 610–623. https://doi.org/10.1145/3442188.3445922

Digital Library

[24]

Jonah Berger and Katherine L. Milkman. 2012. What Makes Online Content Viral?Journal of Marketing Research 49, 2 (April 2012), 192–205. https://doi.org/10.1509/jmr.10.0353

[25]

J. S. Blumenthal-Barby and Hadley Burroughs. 2012. Seeking Better Health Care Outcomes: the Ethics of Using the "Nudge". The American journal of bioethics: AJOB 12, 2 (2012), 1–10. https://doi.org/10.1080/15265161.2011.634481

[26]

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2022. On the Opportunities and Risks of Foundation Models. https://doi.org/10.48550/arXiv.2108.07258 arXiv:2108.07258 [cs].

[27]

Harriet Braiker. 2003. Who’s Pulling Your Strings?: How to Break the Cycle of Manipulation and Regain Control of Your Life: How to Break the Cycle of Manipulation and Regain Control of Your Life. McGraw Hill Professional. Google-Books-ID: dGwgiQvyeq0C.

[28]

Michael Bratman. 1987. Intention, plans, and practical reason. https://philpapers.org/rec/BRAIPA

[29]

Harry Brignull. 2018. Deceptive Design - User Interfaces Crafted to Trick You. https://www.deceptive.design/

[30]

Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering Latent Knowledge in Language Models Without Supervision. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=ETKGuby0hcs

[31]

Qingpeng Cai, Shuchang Liu, Xueliang Wang, Tianyou Zuo, Wentao Xie, Bin Yang, Dong Zheng, Peng Jiang, and Kun Gai. 2023. Reinforcing User Retention in a Billion Scale Short Video Recommender System. http://arxiv.org/abs/2302.01724 arXiv:2302.01724 [cs].

[32]

Agnes Callard. 2018. Aspiration: The Agency of Becoming. Oxford University Press, Oxford, New York.

[33]

M. Ryan Calo. 2014. Digital Market Manipulation. George Washington Law Review 82, 4 (2014), 996–1051. https://doi.org/10.2139/ssrn.2309703

[34]

Ryan Carey, Eric Langlois, Tom Everitt, and Shane Legg. 2020. The Incentives that Shape Behaviour. arXiv:2001.07118 [cs] (Jan. 2020). http://arxiv.org/abs/2001.07118 arXiv:2001.07118.

[35]

Micah Carroll, Anca Dragan, Stuart Russell, and Dylan Hadﬁeld-Menell. 2022. Estimating and Penalizing Induced Preference Shifts in Recommender Systems. Proceedings of machine learning research 162 (2022), 2686–2708.

[36]

Thomas L. Carson. 2010. Lying and Deception: Theory and Practice. Oxford University Press, Oxford ; New York. OCLC: ocn464581525.

[37]

CFTC. 2013. Antidisruptive Practices Authority Interpretative Guidance and Policy Statement. Technical Report RIN 3038-AD96. Commodity Futures Trading Commission. https://www.federalregister.gov/documents/2013/05/28/2013-12365/antidisruptive-practices-authority

[38]

Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molamohammadi, John Burden, Wanru Zhao, Shalaleh Rismani, Konstantinos Voudouris, Umang Bhatt, Adrian Weller, David Krueger, and Tegan Maharaj. 2023. Harms from Increasingly Agentic Algorithmic Systems. https://doi.org/10.48550/arXiv.2302.10329 arXiv:2302.10329 [cs].

[39]

Allison J. B. Chaney. 2021. Recommendation System Simulations: A Discussion of Two Key Challenges. https://doi.org/10.48550/arXiv.2109.02475

[40]

Allison J. B. Chaney, Brandon M. Stewart, and Barbara E. Engelhardt. 2018. How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility. Proceedings of the 12th ACM Conference on Recommender Systems (Sept. 2018), 224–232. https://doi.org/10.1145/3240323.3240370 arXiv:1710.11214.

Digital Library

[41]

Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed Chi. 2020. Top-K Off-Policy Correction for a REINFORCE Recommender System. arXiv:1812.02353 [cs, stat] (Nov. 2020). http://arxiv.org/abs/1812.02353 arXiv:1812.02353.

[42]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and others. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021).

[43]

Paul Christiano, Ajeya Cotra, and Mark Xu. 2021. Eliciting Latent Knowledge. Technical Report. Alignment Research Center. https://ai-alignment.com/eliciting-latent-knowledge-f977478608fc

[44]

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. arXiv:1706.03741 [cs, stat] (July 2017). http://arxiv.org/abs/1706.03741 arXiv:1706.03741.

[45]

Thomas Christiano. 2022. Algorithms, Manipulation, and Democracy. Canadian Journal of Philosophy 52, 1 (Jan. 2022), 109–124. https://doi.org/10.1017/can.2021.29 Publisher: Cambridge University Press.

[46]

Leon Ciechanowski, Aleksandra Przegalinska, Mikolaj Magnuski, and Peter Gloor. 2019. In the Shades of the Uncanny Valley: An Experimental Study of Human–Chatbot Interaction. Future Generation Computer Systems 92 (March 2019), 539–548. https://doi.org/10.1016/j.future.2018.01.055

Digital Library

[47]

Ricky Cooper, Michael Davis, and Ben Van Vliet. 2016. The Mysterious Ethics of High-Frequency Trading. Business Ethics Quarterly 26, 1 (Jan. 2016), 1–22. https://doi.org/10.1017/beq.2015.41

[48]

Mihaela Curmei, Andreas A. Haupt, Benjamin Recht, and Dylan Hadfield-Menell. 2022. Towards Psychologically-Grounded Dynamic Preference Models. In Proceedings of the 16th ACM Conference on Recommender Systems. 35–48.

Digital Library

[49]

Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. 2020. Open Problems in Cooperative AI. https://doi.org/10.48550/arXiv.2012.08630 arXiv:2012.08630 [cs].

[50]

Shayan Doroudi, Vincent Aleven, and Emma Brunskill. 2019. Where’s the Reward?International Journal of Artificial Intelligence in Education 29, 4 (Dec. 2019), 568–620. https://doi.org/10.1007/s40593-019-00187-x

[51]

Robert Epstein and Ronald E. Robertson. 2015. The Search Engine Manipulation Effect (SEME) and its Possible Impact on the Outcomes of Elections. Proceedings of the National Academy of Sciences 112, 33 (Aug. 2015), E4512–E4521. https://doi.org/10.1073/pnas.1419828112 Publisher: Proceedings of the National Academy of Sciences.

[52]

Charles Evans and Atoosa Kasirzadeh. 2021. User Tampering in Reinforcement Learning Recommender Systems. arXiv:2109.04083 [cs] (Sept. 2021). http://arxiv.org/abs/2109.04083 arXiv:2109.04083.

[53]

Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful AI: Developing and Governing AI that does not Lie. arXiv:2110.06674 [cs] (Oct. 2021). http://arxiv.org/abs/2110.06674 arXiv:2110.06674.

[54]

Tom Everitt, Ryan Carey, Eric Langlois, Pedro A. Ortega, and Shane Legg. 2021. Agent Incentives: A Causal Perspective. arXiv:2102.01685.

[55]

Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. 2021. Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective. arXiv:1908.04734 [cs] (March 2021). http://arxiv.org/abs/1908.04734 arXiv:1908.04734.

[56]

Sebastian Farquhar, Ryan Carey, and Tom Everitt. 2022. Path-Specific Objectives for Safer Agent Incentives. arXiv:2204.10018 [cs, stat] (April 2022). http://arxiv.org/abs/2204.10018 arXiv:2204.10018.

[57]

Brian J Fogg. 1998. Captology: the Study of Computers as Persuasive Technologies. In CHI 98 Conference Summary on Human Factors in Computing Systems. 385.

Digital Library

[58]

Brian J Fogg. 2003. Persuasive Technology. Elsevier. https://doi.org/10.1016/B978-1-55860-643-2.X5000-8

[59]

Matija Franklin, Hal Ashton, Rebecca Gorman, and Stuart Armstrong. 2022. Recognising the Importance of Preference Change: A Call for a Coordinated Multidisciplinary Research Effort in the Age of AI. arXiv:2203.10525 [cs] (March 2022). http://arxiv.org/abs/2203.10525 arXiv:2203.10525.

[60]

Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, and Jack Clark. 2022. Predictability and Surprise in Large Generative Models. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM. https://doi.org/10.1145/3531146.3533229

Digital Library

[61]

Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Yuchen He, Zachary Kaden, Vivek Narayanan, Xiaohui Ye, Zhengxing Chen, and Scott Fujimoto. 2019. Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform. arXiv:1811.00260 [cs, stat] (Sept. 2019). http://arxiv.org/abs/1811.00260 arXiv:1811.00260.

[62]

Charles Goodhart. 1975. Problems of Monetary Management: the UK Experience in Papers in Monetary Economics. Monetary Economics 1 (1975).

[63]

Colin M. Gray, Yubo Kou, Bryan Battles, Joseph Hoggatt, and Austin L. Toombs. 2018. The Dark (Patterns) Side of UX Design. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–14. https://doi.org/10.1145/3173574.3174108

Digital Library

[64]

Lewis D. Griffin, Bennett Kleinberg, Maximilian Mozes, Kimberly T. Mai, Maria Vau, Matthew Caldwell, and Augustine Marvor-Parker. 2023. Susceptibility to Influence of Large Language Models. http://arxiv.org/abs/2303.06074 arXiv:2303.06074 [cs].

[65]

Till Grüne-Yanoff and Sven Ove Hansson (Eds.). 2009. Preference change: approaches from philosophy, economics and psychology. Number v. 42 in Theory and decision library. Series A, Philosophy and methodology of the social sciences. Springer, Dordrecht ; London. OCLC: ocn321018474.

[66]

Blake Hallinan, Jed R Brubaker, and Casey Fiesler. 2020. Unexpected Expectations: Public Reaction to the Facebook Emotional Contagion Study. New Media & Society 22, 6 (June 2020), 1076–1094. https://doi.org/10.1177/1461444819876944

[67]

Joseph Y. Halpern and Max Kleiman-Weiner. 2018. Towards Formal Definitions of Blameworthiness, Intention, and Moral responsibility. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence(AAAI’18/IAAI’18/EAAI’18). AAAI Press, New Orleans, Louisiana, USA, 1853–1860.

[68]

Christian Hansen, Rishabh Mehrotra, Casper Hansen, Brian Brost, Lucas Maystre, and Mounia Lalmas. 2021. Shifting Consumption towards Diverse Content on Music Streaming Platforms. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. ACM, Virtual Event Israel, 238–246. https://doi.org/10.1145/3437963.3441775

Digital Library

[69]

Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. 2016. Strategic Classification. In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science(ITCS ’16). Association for Computing Machinery, New York, NY, USA, 111–122.

Digital Library

[70]

Adrian Haret and Johannes Peter Wallner. 2022. An Axiomatic Approach to Revising Preferences. Proceedings of the AAAI Conference on Artificial Intelligence 36, 5 (June 2022), 5676–5683. https://doi.org/10.1609/aaai.v36i5.20509

[71]

Peter Hase and Mohit Bansal. 2020. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5540–5552. https://doi.org/10.18653/v1/2020.acl-main.491

[72]

D. Heckerman and R. Shachter. 1995. Decision-Theoretic Foundations for Causal Reasoning. Journal of Artificial Intelligence Research 3 (Dec. 1995), 405–430. https://doi.org/10.1613/jair.202

[73]

Christina Heinze-Deml, Marloes H. Maathuis, and Nicolai Meinshausen. 2018. Causal Structure Learning. Annual Review of Statistics and Its Application 5, 1 (2018), 371–391. https://doi.org/10.1146/annurev-statistics-031017-100630 _eprint: https://doi.org/10.1146/annurev-statistics-031017-100630.

[74]

Jacob B. Hirsh, Sonia K. Kang, and Galen V. Bodenhausen. 2012. Personalized Persuasion: Tailoring Persuasive Appeals to Recipients’ Personality Traits. Psychological Science 23, 6 (June 2012), 578–581. https://doi.org/10.1177/0956797611436349 Publisher: SAGE Publications Inc.

[75]

Joey Hong, Anca Dragan, and Sergey Levine. 2023. Learning to Influence Human Behavior with Offline Reinforcement Learning. https://doi.org/10.48550/arXiv.2303.02265 arXiv:2303.02265 [cs].

[76]

Yubo Hou, Dan Xiong, Tonglin Jiang, Lily Song, and Qi Wang. 2019. Social media addiction: Its impact, mediation, and intervention. Cyberpsychology: Journal of Psychosocial Research on Cyberspace 13, 1 (Feb. 2019). https://doi.org/10.5817/CP2019-1-4

[77]

Lily Hu, Nicole Immorlica, and Jennifer Wortman Vaughan. 2019. The Disparate Effects of Strategic Manipulation. In Proceedings of the Conference on Fairness, Accountability, and Transparency(FAT* ’19). Association for Computing Machinery, New York, NY, USA, 259–268.

Digital Library

[78]

(Robin) Hui Huang. 2009. Redefining Market Manipulation in Australia: The Role of an Implied Intent Element. Companies and Securities Law Journal 27 (April 2009). https://papers.ssrn.com/abstract=1376209

[79]

Ferenc Huszár, Sofia Ira Ktena, Conor O’Brien, Luca Belli, Andrew Schlaikjer, and Moritz Hardt. 2021. Algorithmic Amplification of Politics on Twitter. arXiv:2110.11010 [cs] (Oct. 2021). http://arxiv.org/abs/2110.11010 arXiv:2110.11010.

[80]

Max Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. 2019. Human-level Performance in 3D Multiplayer Games with Population-Based Reinforcement Learning. Science 364, 6443 (May 2019), 859–865. https://doi.org/10.1126/science.aau6249 Publisher: American Association for the Advancement of Science.

[81]

Meena Jagadeesan, Celestine Mendler-Dünner, and Moritz Hardt. 2021. Alternative Microfoundations for Strategic Classification. In ICML. http://arxiv.org/abs/2106.12705 arXiv:2106.12705.

[82]

Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-Writing with Opinionated Language Models Affects Users’ Views. https://doi.org/10.1145/3544548.3581196 arXiv:2302.00560 [cs].

Digital Library

[83]

Janus. 2022. Simulators. https://generative.ink/posts/simulators/

[84]

Mathias Jesse and Dietmar Jannach. 2021. Digital Nudging with Recommender Systems: Survey and Future Directions. Computers in Human Behavior Reports 3 (Jan. 2021), 100052. https://doi.org/10.1016/j.chbr.2020.100052

[85]

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of Hallucination in Natural Language Generation. Comput. Surveys (Nov. 2022). https://doi.org/10.1145/3571730 Just Accepted.

Digital Library

[86]

Ray Jiang, Silvia Chiappa, Tor Lattimore, András György, and Pushmeet Kohli. 2019. Degenerate Feedback Loops in Recommender Systems. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (Jan. 2019), 383–390. https://doi.org/10.1145/3306618.3314288 arXiv:1902.10730.

Digital Library

[87]

Eric J. Johnson and Daniel Goldstein. 2003. Do Defaults Save Lives?Science 302, 5649 (Nov. 2003), 1338–1339. https://doi.org/10.1126/science.1091721 Publisher: American Association for the Advancement of Science.

[88]

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know. https://doi.org/10.48550/arXiv.2207.05221 arXiv:2207.05221 [cs].

[89]

Jean Kaddour, Aengus Lynch, Qi Liu, Matt J. Kusner, and Ricardo Silva. 2022. Causal Machine Learning: A Survey and Open Problems. https://doi.org/10.48550/arXiv.2206.15475 arXiv:2206.15475 [cs, stat].

[90]

Emir Kamenica and Matthew Gentzkow. 2011. Bayesian Persuasion. American Economic Review 101, 6 (Oct. 2011), 2590–2615. https://doi.org/10.1257/aer.101.6.2590

[91]

Timotheus Kampik, Juan Carlos Nieves, and Helena Lindgren. 2018. Coercion and Deception in Persuasive Technologies. In 20th International Trust Workshop (co-located with AAMAS/IJCAI/ECAI/ICML 2018), Stockholm, Sweden, 14 July, 2018. CEUR-WS, 38–49.

[92]

Maurits Kaptein and Steven Duplinsky. 2013. Combining Multiple Influence Strategies to Increase Consumer Compliance. International Journal of Internet Marketing and Advertising 8, 1 (2013), 32. https://doi.org/10.1504/IJIMA.2013.056586

[93]

Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of Language Agents. arXiv:2103.14659 [cs] (March 2021). http://arxiv.org/abs/2103.14659 arXiv:2103.14659.

[94]

Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDermott, and Tom Everitt. 2022. Discovering Agents. https://doi.org/10.48550/arXiv.2208.08345 arXiv:2208.08345 [cs].

[95]

Poruz Khambatta, Shwetha Mariadassou, Joshua Morris, and S Christian Wheeler. 2022. Targeting Recommendation Algorithms to Ideal Preferences Makes Users Better Off. (2022).

[96]

Jon Kleinberg and Manish Raghavan. 2019. How Do Classifiers Induce Agents to Invest Effort Strategically?. In Proceedings of the 2019 ACM Conference on Economics and Computation(EC ’19). Association for Computing Machinery, New York, NY, USA, 825–844. https://doi.org/10.1145/3328526.3329584

Digital Library

[97]

Michal Kosinski. 2023. Theory of Mind May Have Spontaneously Emerged in Large Language Models. http://arxiv.org/abs/2302.02083 arXiv:2302.02083 [cs].

[98]

Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. 2019. Penalizing Side Effects Using Stepwise Relative Reachability. arXiv:1806.01186 [cs, stat] (March 2019). http://arxiv.org/abs/1806.01186 arXiv:1806.01186.

[99]

Ilan Kremer, Yishay Mansour, and Motty Perry. 2014. Implementing the "Wisdom of the Crowd". Journal of Political Economy 122, 5 (2014), 988 – 1012. https://econpapers.repec.org/article/ucpjpolec/doi_3a10.1086_2f676597.htm Publisher: University of Chicago Press.

[100]

David Krueger, Tegan Maharaj, and Jan Leike. 2020. Hidden Incentives for Auto-Induced Distributional Shift.

[101]

Arto Laitinen and Otto Sahlgren. 2021. AI Systems and Respect for Human Autonomy. Frontiers in Artificial Intelligence 4 (2021). https://www.frontiersin.org/articles/10.3389/frai.2021.705164

[102]

Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. 2022. Goal Misgeneralization in Deep Reinforcement Learning. In Proceedings of the 39th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 12004–12019. https://proceedings.mlr.press/v162/langosco22a.html

[103]

Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=DeG07_TcZvT

[104]

Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs] (Sept. 2021). http://arxiv.org/abs/2109.07958 arXiv:2109.07958.

[105]

Tom C. W. Lin. 2017. The New Market Manipulation. Emory Law Journal 66 (July 2017). https://papers.ssrn.com/abstract=2996896

[106]

David Lindner, Kyle Matoba, and Alexander Meulemans. 2021. Challenges for Using Impact Regularizers to Avoid Negative Side Effects. arXiv:2101.12509 [cs] (Feb. 2021). http://arxiv.org/abs/2101.12509 arXiv:2101.12509.

[107]

Jamie Luguri and Lior Jacob Strahilevitz. 2021. Shining a Light on Dark Patterns. Journal of Legal Analysis 13, 1 (March 2021), 43–109. https://doi.org/10.1093/jla/laaa006

[108]

James Edwin Mahon. 2016. The Definition of Lying and Deception. In The Stanford Encyclopedia of Philosophy (winter 2016 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/archives/win2016/entries/lying-definition/

[109]

Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective. http://arxiv.org/abs/2301.06627 arXiv:2301.06627 [cs].

[110]

David Manheim and Scott Garrabrant. 2019. Categorizing Variants of Goodhart’s Law. arXiv:1803.04585 [cs, q-fin, stat] (Feb. 2019). http://arxiv.org/abs/1803.04585 arXiv:1803.04585.

[111]

Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. 2020. Feedback Loop and Bias Amplification in Recommender Systems. arXiv:2007.13019 [cs] (July 2020). http://arxiv.org/abs/2007.13019 arXiv:2007.13019.

[112]

Jörg Meibauer. 2005. Lying and Falsely Implicating. Journal of Pragmatics 37, 9 (Sept. 2005), 1373–1399. https://doi.org/10.1016/j.pragma.2004.12.007

[113]

Meta Fundamental AI Research Diplomacy Team (FAIR), Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. 2022. Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning. Science 378, 6624 (Dec. 2022), 1067–1074. https://doi.org/10.1126/science.ade9097 Publisher: American Association for the Advancement of Science.

[114]

Smitha Milli, Micah Carroll, Yike Wang, Sashrika Pandey, Sebastian Zhao, and Anca D. Dragan. 2023. Engagement, User Satisfaction, and the Amplification of Divisive Content on Social Media. https://doi.org/10.48550/arXiv.2305.16941 arXiv:2305.16941 [cs].

[115]

Smitha Milli, John Miller, Anca D. Dragan, and Moritz Hardt. 2019. The Social Cost of Strategic Classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency(FAT* ’19). Association for Computing Machinery, New York, NY, USA, 230–239.

Digital Library

[116]

Stuart Mills. 2022. Finding the ‘Nudge’ in Hypernudge. Technology in Society 71 (Nov. 2022), 102117. https://doi.org/10.1016/j.techsoc.2022.102117

[117]

Kevin Munger and Joseph Phillips. 2020. Right-Wing YouTube: A Supply and Demand Perspective. The International Journal of Press/Politics (Oct. 2020), 1940161220964767. https://doi.org/10.1177/1940161220964767 Publisher: SAGE Publications Inc.

[118]

Maciej Musiał. 2022. Can We Design Artificial Persons without Being Manipulative?AI & SOCIETY (Oct. 2022). https://doi.org/10.1007/s00146-022-01575-z

Digital Library

[119]

Hendrik Müller, Aaron Sedley, and Elizabeth Ferrall-Nunge. 2014. Survey Research in HCI. In Ways of Knowing in HCI, Judith S. Olson and Wendy A. Kellogg (Eds.). Springer, New York, NY, 229–266. https://doi.org/10.1007/978-1-4939-0378-8_10

[120]

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and others. 2021. Webgpt: Browser-Assisted Question-Answering with Human Feedback. arXiv preprint arXiv:2112.09332 (2021).

[121]

Robert Noggle. 2022. The Ethics of Manipulation. In The Stanford Encyclopedia of Philosophy (summer 2022 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/archives/sum2022/entries/ethics-manipulation/

[122]

APA Dictionary of Psychology. 2023. Definition of manipulation. https://dictionary.apa.org/manipulation

[123]

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022. In-context Learning and Induction Heads. Transformer Circuits Thread (2022).

[124]

Alexander Pan, Chan Jun Shern, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. 2023. Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark. http://arxiv.org/abs/2304.03279 arXiv:2304.03279 [cs].

[125]

Peter S. Park, Simon Goldstein, Aidan O’Gara, Michael Chen, and Dan Hendrycks. 2023. AI Deception: A Survey of Examples, Risks, and Potential Solutions. http://arxiv.org/abs/2308.14752 arXiv:2308.14752 [cs].

[126]

L. A. Paul. 2014. Transformative experience (1st ed ed.). Oxford University Press, Oxford. OCLC: ocn872342141.

[127]

L. A. Paul. 2022. Choosing for Changing Selves. The Philosophical Review 131, 2 (April 2022), 230–235. https://doi.org/10.1215/00318108-9554756

[128]

Amalie Brogaard Pauli, Leon Derczynski, and Ira Assent. 2022. Modelling Persuasion through Misuse of Rhetorical Appeals. (2022).

[129]

Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. 2020. Performative Prediction. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119. PMLR.

[130]

Fabíola S. F. Pereira, João Gama, Sandra de Amo, and Gina M. B. Oliveira. 2018. On Analyzing User Preference Dynamics with Temporal Social Networks. Machine Learning 107, 11 (Nov. 2018), 1745–1773. https://doi.org/10.1007/s10994-018-5740-2

Digital Library

[131]

Billy Perrigo. 2021. How Frances Haugen’s Team Forced a Facebook Reckoning. Time (Oct. 2021). https://time.com/6104899/facebook-reckoning-frances-haugen/

[132]

Richard Pettigrew. 2019. Choosing for Changing Selves (1 ed.). Oxford University Press. https://doi.org/10.1093/oso/9780198814962.001.0001

[133]

Richard Pettigrew. 2022. Nudging for Changing Selves. SSRN Electronic Journal (2022). https://doi.org/10.2139/ssrn.4025214

[134]

Carina Prunkl. 2022. Human Autonomy in the Age of Artificial Intelligence. Nature Machine Intelligence 4, 2 (Feb. 2022), 99–101. https://doi.org/10.1038/s42256-022-00449-9 Number: 2 Publisher: Nature Publishing Group.

[135]

Tālis Putniņš. 2020. An Overview of Market Manipulation. In Corruption and Fraud in Financial Markets (1st ed.), Carol Alexander and Douglas Cumming (Eds.). John Wiley & Sons Inc., United States, 13–44.

[136]

Inioluwa Deborah Raji and Joy Buolamwini. 2019. Actionable Auditing. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. ACM. https://doi.org/10.1145/3306618.3314244

Digital Library

[137]

Inioluwa Deborah Raji, Peggy Xu, Colleen Honigsberg, and Daniel Ho. 2022. Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. ACM. https://doi.org/10.1145/3514094.3534181

Digital Library

[138]

Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virgílio A. F. Almeida, and Wagner Meira. 2020. Auditing radicalization pathways on YouTube. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency(FAT* ’20). Association for Computing Machinery, New York, NY, USA, 131–141. https://doi.org/10.1145/3351095.3372879

Digital Library

[139]

Manoel Horta Ribeiro, Veniamin Veselovsky, and Robert West. 2023. The Amplification Paradox in Recommender Systems. http://arxiv.org/abs/2302.11225 arXiv:2302.11225 [cs].

[140]

Jonathan Richens, Rory Beard, and Daniel H. Thompson. 2022. Counterfactual Harm. In Advances in Neural Information Processing Systems, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (Eds.). https://openreview.net/forum?id=zkQho-Jxky9

[141]

Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. (2014), 23.

[142]

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv preprint arXiv:2302.04761 (2023).

[143]

Andreas T. Schmidt and Bart Engelen. 2020. The Ethics of Nudging: An Overview. Philosophy Compass 15, 4 (April 2020). https://doi.org/10.1111/phc3.12658

[144]

Gregory Scopino. 2015. Do Automated Trading Systems Dream of Manipulating the Price of Futures contracts? Policing Markets for Improper Trading Practices by Algorithmic Robots. Florida Law Review 67 (2015), 221.

[145]

Caroline Serbanescu. 2021. Why Does Artificial Intelligence Challenge Democracy? A Critical Analysis of the Nature of the Challenges Posed by AI-Enabled Manipulation. Copenhagen journal of legal studies 5, 1 (2021), 105–128. https://ssrn.com/abstract=4033258

[146]

Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. 2022. Goal Misgeneralization: Why Correct Specifications Aren’t Enough For Correct Goals. https://doi.org/10.48550/arXiv.2210.01790 arXiv:2210.01790 [cs].

[147]

caroline sinders. 2022. What’s In a Name?https://medium.com/@carolinesinders/whats-in-a-name-unpacking-dark-patterns-versus-deceptive-design-e96068627ec4

[148]

Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and Characterizing Reward Hacking. http://arxiv.org/abs/2209.13085 arXiv:2209.13085 [cs, stat].

[149]

David Horton Smith. 1967. Correcting for Social Desirability Response Sets in Opinion-Attitude Survey Research. The Public Opinion Quarterly 31, 1 (1967), 87–94. https://www.jstor.org/stable/2746886 Publisher: [Oxford University Press, American Association for Public Opinion Research].

[150]

Aaron J. Snoswell and Jean Burgess. 2022. The Galactica AI Model was Trained on Scientific Knowledge – but it Spat Out Alarmingly Plausible Nonsense. http://theconversation.com/the-galactica-ai-model-was-trained-on-scientific-knowledge-but-it-spat-out-alarmingly-plausible-nonsense-195445

[151]

Barry M. Staw. 1976. Knee-Deep in the Big Muddy: a Study of Escalating Commitment to a Chosen Course of Action. Organizational Behavior and Human Performance 16, 1 (June 1976), 27–44. https://doi.org/10.1016/0030-5073(76)90005-2

[152]

Jacob Steinhardt. 2023. Emergent Deception and Emergent Optimization. https://bounded-regret.ghost.io/emergent-deception-optimization/

[153]

Jonathan Stray, Steven Adler, and Dylan Hadfield-Menell. 2021. What are you optimizing for? Aligning Recommender Systems with Human Values. (2021), 7.

[154]

Michael Strevens. 2020. The Knowledge Machine: How Irrationality Created Modern Science. Liveright Publishing.

[155]

Cass R. Sunstein. 2021. Manipulation As Theft. SSRN Electronic Journal (2021). https://doi.org/10.2139/ssrn.3880048

[156]

Daniel Susser. 2019. Invisible Influence: Artificial Intelligence and the Ethics of Adaptive Choice Architectures. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society(AIES ’19). Association for Computing Machinery, New York, NY, USA, 403–408. https://doi.org/10.1145/3306618.3314286

Digital Library

[157]

Daniel Susser, Beate Roessler, and Helen Nissenbaum. 2019. Online Manipulation: Hidden Influences in a Digital World. Geo. L. Tech. Rev. 4 (2019), 1. Publisher: HeinOnline.

[158]

Daniel Susser, Beate Roessler, and Helen Nissenbaum. 2019. Technology, Autonomy, and Manipulation. Internet Policy Review 8, 2 (June 2019). https://papers.ssrn.com/abstract=3420747

[159]

Richard H. Thaler and Cass R. Sunstein. 2009. Nudge: Improving Decisions about Health, Wealth and Happiness (revised edition, new international edition ed.). Penguin Books, London New York Toronto Dublin Camberwell New Delhi Rosedale Johannesburg.

[160]

Luke Thorburn. 2022. How Platform Recommenders Work. https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a

[161]

Luke Thorburn, Jonathan Stray, and Priyanjana Bengani. 2022. What Will “Amplification” Mean in Court?https://techpolicy.press/what-will-amplification-mean-in-court/?curius=1684

[162]

Twitter. 2023. Twitter’s Recommendation Algorithm. https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm

[163]

Tomer Ullman. 2023. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. http://arxiv.org/abs/2302.08399 arXiv:2302.08399 [cs].

[164]

Aditya Nrusimha Vaidyam, Hannah Wisniewski, John David Halamka, Matcheri S. Kashavan, and John Blake Torous. 2019. Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape. Canadian Journal of Psychiatry. Revue Canadienne De Psychiatrie 64, 7 (July 2019), 456–464. https://doi.org/10.1177/0706743719828977

[165]

James Vincent. 2023. Microsoft’s Bing is an emotionally manipulative liar, and people love it. https://www.theverge.com/2023/2/15/23599072/microsoft-ai-bing-personality-conversations-spy-employees-webcams

[166]

Carissa Véliz. 2023. Chatbots Shouldn’t Use Emojis. Nature 615, 7952 (March 2023), 375–375. https://doi.org/10.1038/d41586-023-00758-y Bandiera_abtest: a Cg_type: World View Number: 7952 Publisher: Nature Publishing Group Subject_term: Ethics, Society, Machine learning, Technology.

[167]

Francis Rhys Ward. 2022. On Agent Incentives to Manipulate Human Feedback in Multi-Agent Reward Learning Scenarios. (2022).

[168]

Francis Rhys Ward, Tom Everitt, Francesca Toni, and Francesco Belardinelli. 2023. Honesty Is the Best Policy: Defining and Mitigating AI Deception. (2023).

[169]

Francis Rhys Ward, Francesca Toni, and Francesco Belardinelli. 2022. A Causal Perspective on AI Deception in Games. (2022).

[170]

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research (2022). https://openreview.net/forum?id=yzkSU5zdwD

[171]

Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2022. Taxonomy of Risks posed by Language Models. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM. https://doi.org/10.1145/3531146.3533088

Digital Library

[172]

Benjamin Weissman and Marina Terkourafi. 2019. Are False Implicatures Lies? An Empirical Investigation. Mind & Language 34, 2 (2019), 221–246. https://doi.org/10.1111/mila.12212 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/mila.12212.

[173]

Georgia Wells, Jeff Horwitz, and Deepa Seetharaman. 2021. Facebook Knows Instagram Is Toxic for Teen Girls, Company Documents Show. Wall Street Journal (Sept. 2021). https://www.wsj.com/articles/facebook-knows-instagram-is-toxic-for-teen-girls-company-documents-show-11631620739

[174]

Nicole Wetsman. 2021. Facebook’s Whistleblower Report Confirms what Researchers Have Known for Years. The Verge (Oct. 2021). https://www.theverge.com/2021/10/6/22712927/facebook-instagram-teen-mental-health-research

[175]

Lauren E. Willis. 2020. Deception by Design. Harvard journal of law and technology 34, 1 (Aug. 2020). https://papers.ssrn.com/abstract=3694575

[176]

Amy A. Winecoff, Matthew Sun, Eli Lucherini, and Arvind Narayanan. 2021. Simulation as Experiment: An Empirical Critique of Simulation Research on Recommender Systems. http://arxiv.org/abs/2107.14333 arXiv:2107.14333 [cs].

[177]

Allen W. Wood. 2014. Coercion, Manipulation, Exploitation. In Manipulation: theory and practice. Oxford University Press, Oxford ; New York.

[178]

Karen Yeung. 2017. ‘Hypernudge’: Big Data as a mode of regulation by design. Information, Communication & Society 20, 1 (Jan. 2017), 118–136. https://doi.org/10.1080/1369118X.2016.1186713

[179]

Savvas Zannettou, Sotirios Chatzis, Kostantinos Papadamou, and Michael Sirivianos. 2018. The Good, the Bad and the Bait: Detecting and Characterizing Clickbait on YouTube. In 2018 IEEE Security and Privacy Workshops (SPW). 63–69. https://doi.org/10.1109/SPW.2018.00018

[180]

Tal Z. Zarsky. 2019. Privacy and Manipulation in the Digital Age. Theoretical Inquiries in Law 20, 1 (March 2019), 157–188. https://doi.org/10.1515/til-2019-0006

[181]

Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency(FAT* ’20). Association for Computing Machinery, New York, NY, USA, 295–305. https://doi.org/10.1145/3351095.3372852

Digital Library

[182]

Zhengbang Zhu, Rongjun Qin, Junjie Huang, Xinyi Dai, Yang Yu, Yong Yu, and Weinan Zhang. 2022. Understanding or Manipulation: Rethinking Online Performance Gains of Modern Recommender Systems. http://arxiv.org/abs/2210.05662 arXiv:2210.05662 [cs].

[183]

Frederik Zuiderveen Borgesius, Judith Moeller, Sanne Kruikemeier, Ronan Ó Fathaigh, Kristina Irion, Tom Dobber, Balázs Bodó, and Claes H. de Vreese. 2018. Online Political Microtargeting: Promises and Threats for Democracy. Utrecht Law Review 14, 1 (Feb. 2018), 82–96. https://papers.ssrn.com/abstract=3128787

Cited By

Ward FMacDermott MBelardinelli FToni FEveritt TDastani MSichman JAlechina NDignum V(2024)The Reasons that Agents Act: Intention and Instrumental GoalsProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3663053(1901-1909)Online publication date: 6-May-2024
https://dl.acm.org/doi/10.5555/3635637.3663053
Williamson SPrybutok V(2024)The Era of Artificial Intelligence Deception: Unraveling the Complexities of False Realities and Emerging Threats of MisinformationInformation10.3390/info1506029915:6(299)Online publication date: 23-May-2024
https://doi.org/10.3390/info15060299
Stracqualursi LAgati P(2024)Twitter users perceptions of AI-based e-learning technologiesScientific Reports10.1038/s41598-024-56284-y14:1Online publication date: 11-Mar-2024
https://doi.org/10.1038/s41598-024-56284-y
Show More Cited By

Index Terms

Characterizing Manipulation from AI Systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Ethics of generative AI and manipulation: a design-oriented research agenda
Abstract
Generative AI enables automated, effective manipulation at scale. Despite the growing general ethical discussion around generative AI, the specific manipulation risks remain inadequately investigated. This article outlines essential inquiries ...
Trust in an AI versus a Human teammate: The effects of teammate identity and performance on Human-AI cooperation
Abstract
Recent advances in artificial intelligence (AI) enable researchers to create more powerful AI agents that are becoming competent teammates for humans. However, human distrust of AI is a critical factor that may impede human-AI ...
Highlights
- Humans accept their AI teammate's decision less often when they are deceived about the identity of the AI as another human.
Knotting/Unknotting Manipulation of Deformable Linear Objects

Here, we propose a planning method for knotting/unknotting of deformable linear objects. First, we propose a topological description of the state of a linear object. Second, transitions between these states are defined by introducing four basic ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

EAAMO '23: Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization

October 2023

498 pages

ISBN:9798400703812

DOI:10.1145/3617694

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 October 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

EAAMO '23

Sponsor:

EAAMO '23: Equity and Access in Algorithms, Mechanisms, and Optimization

October 30 - November 1, 2023

MA, Boston, USA

Upcoming Conference

EAAMO '24

Sponsor:
sigai
sigai

Equity and Access in Algorithms, Mechanisms, and Optimization

October 29 - 31, 2024

San Luis Potosi , Mexico

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
3,320
Total Downloads

Downloads (Last 12 months)3,320
Downloads (Last 6 weeks)471

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ward FMacDermott MBelardinelli FToni FEveritt TDastani MSichman JAlechina NDignum V(2024)The Reasons that Agents Act: Intention and Instrumental GoalsProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3663053(1901-1909)Online publication date: 6-May-2024
https://dl.acm.org/doi/10.5555/3635637.3663053
Williamson SPrybutok V(2024)The Era of Artificial Intelligence Deception: Unraveling the Complexities of False Realities and Emerging Threats of MisinformationInformation10.3390/info1506029915:6(299)Online publication date: 23-May-2024
https://doi.org/10.3390/info15060299
Stracqualursi LAgati P(2024)Twitter users perceptions of AI-based e-learning technologiesScientific Reports10.1038/s41598-024-56284-y14:1Online publication date: 11-Mar-2024
https://doi.org/10.1038/s41598-024-56284-y
Chen CShu K(2024)Combating misinformation in the age of LLMs: Opportunities and challengesAI Magazine10.1002/aaai.12188Online publication date: Aug-2024
https://doi.org/10.1002/aaai.12188

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents