Quantitative Biology > Quantitative Methods
[Submitted on 14 Jul 2024]
Title:AI Detectors are Poor Western Blot Classifiers: A Study of Accuracy and Predictive Values
View PDFAbstract:The recent rise of generative artificial intelligence (GenAI) capable of creating scientific images presents a challenge in the fight against academic fraud. This study evaluates the efficacy of three free web-based AI detectors in identifying AI-generated images of Western blots, which is a very common technique in biology. We tested these detectors on a collection of artificial Western blot images (n=48) that were created using ChatGPT 4 DALLE 3 and on authentic Western blots (n=48) that were sampled from articles published within four biology journals in 2015; this was before the rise of generative AI based on large language models. The results reveal that the sensitivity (0.9583 for Is It AI, 0.1875 for Hive Moderation, and 0.7083 for Illuminarty) and specificity (0.5417 for Is It AI, 0.8750 for Hive Moderation, and 0.4167 for Illuminarty) are very different. Positive predictive values (PPV) across various AI prevalence were low, for example reaching 0.1885 for Is It AI, 0.1429 for Hive Moderation, and 0.1189 for Illuminarty at an AI prevalence of 0.1. This highlights the difficulty in confidently determining image authenticity based on the output of a single detector. Reducing the size of Western blots from four to two lanes reduced test sensitivities and increased test specificities but did not markedly affect overall detector accuracies and also only slightly improved the PPV of one detector (Is It AI). These findings strongly argue against the use of free AI detectors to detect fake scientific images, and they demonstrate the urgent need for more robust detection tools that are specifically trained on scientific content such as Western blot images.
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.