-
The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models
Authors:
Bolei Ma,
Xinpeng Wang,
Tiancheng Hu,
Anna-Carolina Haensch,
Michael A. Hedderich,
Barbara Plank,
Frauke Kreuter
Abstract:
Recent advances in Large Language Models (LLMs) have sparked wide interest in validating and comprehending the human-like cognitive-behavioral traits LLMs may have. These cognitive-behavioral traits include typically Attitudes, Opinions, Values (AOV). However, measuring AOV embedded within LLMs remains opaque, and different evaluation methods may yield different results. This has led to a lack of…
▽ More
Recent advances in Large Language Models (LLMs) have sparked wide interest in validating and comprehending the human-like cognitive-behavioral traits LLMs may have. These cognitive-behavioral traits include typically Attitudes, Opinions, Values (AOV). However, measuring AOV embedded within LLMs remains opaque, and different evaluation methods may yield different results. This has led to a lack of clarity on how different studies are related to each other and how they can be interpreted. This paper aims to bridge this gap by providing an overview of recent works on the evaluation of AOV in LLMs. Moreover, we survey related approaches in different stages of the evaluation pipeline in these works. By doing so, we address the potential and challenges with respect to understanding the model, human-AI alignment, and downstream application in social sciences. Finally, we provide practical insights into evaluation methods, model enhancement, and interdisciplinary collaboration, thereby contributing to the evolving landscape of evaluating AOV in LLMs.
△ Less
Submitted 1 July, 2024; v1 submitted 16 June, 2024;
originally announced June 2024.
-
30 Years of Synthetic Data
Authors:
Joerg Drechsler,
Anna-Carolina Haensch
Abstract:
The idea to generate synthetic data as a tool for broadening access to sensitive microdata has been proposed for the first time three decades ago. While first applications of the idea emerged around the turn of the century, the approach really gained momentum over the last ten years, stimulated at least in parts by some recent developments in computer science. We consider the upcoming 30th jubilee…
▽ More
The idea to generate synthetic data as a tool for broadening access to sensitive microdata has been proposed for the first time three decades ago. While first applications of the idea emerged around the turn of the century, the approach really gained momentum over the last ten years, stimulated at least in parts by some recent developments in computer science. We consider the upcoming 30th jubilee of Rubin's seminal paper on synthetic data (Rubin, 1993) as an opportunity to look back at the historical developments, but also to offer a review of the diverse approaches and methodological underpinnings proposed over the years. We will also discuss the various strategies that have been suggested to measure the utility and remaining risk of disclosure of the generated data.
△ Less
Submitted 4 April, 2023;
originally announced April 2023.
-
Seeing ChatGPT Through Students' Eyes: An Analysis of TikTok Data
Authors:
Anna-Carolina Haensch,
Sarah Ball,
Markus Herklotz,
Frauke Kreuter
Abstract:
Advanced large language models like ChatGPT have gained considerable attention recently, including among students. However, while the debate on ChatGPT in academia is making waves, more understanding is needed among lecturers and teachers on how students use and perceive ChatGPT. To address this gap, we analyzed the content on ChatGPT available on TikTok in February 2023. TikTok is a rapidly growi…
▽ More
Advanced large language models like ChatGPT have gained considerable attention recently, including among students. However, while the debate on ChatGPT in academia is making waves, more understanding is needed among lecturers and teachers on how students use and perceive ChatGPT. To address this gap, we analyzed the content on ChatGPT available on TikTok in February 2023. TikTok is a rapidly growing social media platform popular among individuals under 30. Specifically, we analyzed the content of the 100 most popular videos in English tagged with #chatgpt, which collectively garnered over 250 million views. Most of the videos we studied promoted the use of ChatGPT for tasks like writing essays or code. In addition, many videos discussed AI detectors, with a focus on how other tools can help to transform ChatGPT output to fool these detectors. This also mirrors the discussion among educators on how to treat ChatGPT as lecturers and teachers in teaching and grading. What is, however, missing from the analyzed clips on TikTok are videos that discuss ChatGPT producing content that is nonsensical or unfaithful to the training data.
△ Less
Submitted 9 March, 2023;
originally announced March 2023.
-
A geospatial bounded confidence model including mega-influencers with an application to Covid-19 vaccine hesitancy
Authors:
Anna Haensch,
Natasa Dragovic,
Christoph Börgers,
Bruce Boghosian
Abstract:
We introduce a geospatial bounded confidence model with mega-influencers, inspired by Hegselmann and Krause. The inclusion of geography gives rise to large-scale geospatial patterns evolving out of random initial data; that is, spatial clusters of like-minded agents emerge regardless of initialization. Mega-influencers and stochasticity amplify this effect, and soften local consensus. As an applic…
▽ More
We introduce a geospatial bounded confidence model with mega-influencers, inspired by Hegselmann and Krause. The inclusion of geography gives rise to large-scale geospatial patterns evolving out of random initial data; that is, spatial clusters of like-minded agents emerge regardless of initialization. Mega-influencers and stochasticity amplify this effect, and soften local consensus. As an application, we consider national views on Covid-19 vaccines. For a certain set of parameters, our model yields results comparable to real survey results on vaccine hesitancy from late 2020.
△ Less
Submitted 14 October, 2022;
originally announced October 2022.
-
An Equity-Aware Recommender System for Curating Art Exhibits Based on Locally-Constrained Graph Matching
Authors:
Anna Haensch,
Dina Deitsch
Abstract:
Public art shapes our shared spaces. Public art should speak to community and context, and yet, recent work has demonstrated numerous instances of art in prominent institutions favoring outdated cultural norms and legacy communities. Motivated by this, we develop a novel recommender system to curate public art exhibits with built-in equity objectives and a local value-based allocation of constrain…
▽ More
Public art shapes our shared spaces. Public art should speak to community and context, and yet, recent work has demonstrated numerous instances of art in prominent institutions favoring outdated cultural norms and legacy communities. Motivated by this, we develop a novel recommender system to curate public art exhibits with built-in equity objectives and a local value-based allocation of constrained resources. We develop a cost matrix by drawing on Schelling's model of segregation. Using the cost matrix as an input, the scoring function is optimized via a projected gradient descent to obtain a soft assignment matrix. Our optimization program allocates artwork to public spaces in a way that de-prioritizes "in-group" preferences, by satisfying minimum representation and exposure criteria. We draw on existing literature to develop a fairness metric for our algorithmic output, and we assess the effectiveness of our approach and discuss its potential pitfalls from both a curatorial and equity standpoint.
△ Less
Submitted 10 October, 2023; v1 submitted 28 July, 2022;
originally announced July 2022.
-
Covid-19 vaccine hesitancy and mega-influencers
Authors:
Anna Haensch,
Natasa Dragovic,
Christoph Börgers,
Bruce Boghosian
Abstract:
Covid-19 vaccines are widely available in the United States, yet our Covid-19 vaccination rates have remained far below 100%. Not only that, but CDC data shows that even in places where vaccine acceptance was proportionally high at the outset of the Covid-19 vaccination effort, that willingness has not necessarily translated into high rates of vaccination over the subsequent months. We model how s…
▽ More
Covid-19 vaccines are widely available in the United States, yet our Covid-19 vaccination rates have remained far below 100%. Not only that, but CDC data shows that even in places where vaccine acceptance was proportionally high at the outset of the Covid-19 vaccination effort, that willingness has not necessarily translated into high rates of vaccination over the subsequent months. We model how such a shift could have arisen, using parameters in agreement with data from the state of Alabama. The simulations suggest that in Alabama, local interactions would have favored the emergence of tight consensus around the initial majority view, which was to accept the Covid-19 vaccine. Yet this is not what happened. We therefore add to our model the impact of mega-influencers such as mass media, the governor of the state, etc. Our simulations show that a single vaccine-hesitant mega-influencer, reaching a large fraction of the population, can indeed cause the consensus to shift radically, from acceptance to hesitancy. Surprisingly this is true even when the mega-influencer only reaches individuals who are already somewhat inclined to agree with them, and under the conservative assumption that individuals give no more weight to the mega-influencer than they would give to a single one of their friends or neighbors. Our simulations also suggest that a competing mega-influencer with the opposite view can shift the mean population opinion back, but cannot restore the tightness of consensus around that view. Our code and data are distributed in the ODyN (Opinion Dynamic Networks) library available at https://github.com/annahaensch/ODyN.
△ Less
Submitted 15 April, 2022; v1 submitted 18 January, 2022;
originally announced February 2022.
