Abstract

INTRODUCTION

With the increased consumption of social networks such as Facebook, Twitter, and Instagram, Internet and social networking platforms are being used by researchers for study recruitment and data collection in HIV and other health-related research.^1–5 These platforms are beneficial due to the cost-effectiveness of online advertisements and the volume of potential participants/users who may interact with the advertisement.^2,4,6–8 Further, internet-based research expands the geographic reach of recruitment efforts, reduces participation burdens (e.g., financial, travel, time commitment), and improves ability to reach and provide increased anonymity to participants that have been marginalized (e.g., people of color or LGBTQ+ people) and/or those experiencing stigma (e.g., due to gender identity or HIV serostatus).^1,2,6,9,10

Despite its benefits, online recruitment is a complex and indirect form of communication involving third parties that collect, aggregate, and store participant data.² Other challenges of online research include recruiting ineligible individuals (fraud), “bots” (computer software designed to complete automated tasks), and people entering duplicate responses.^2,5,6,10,11 Therefore, strategies are needed to ensure that data are collected from individuals who are eligible for research participation.

There are a number of available automated data safety mechanisms (e.g., Completely Automated Public Turing Test to tell Computers and Humans Apart (reCAPTCHA), bot detection, anti-ballot stuffing) in survey software but studies show that human participants and sophisticated bots can bypass them.^5,8,12 Other mechanisms from prior survey studies have included questions requiring a meaningful open-ended text responses. In one study, by reviewing responses to questions like “How has coronavirus (COVID-19) affected your life?”, the authors identified a considerable number of bots (13.3%) with exact duplicate text responses.¹⁰ Another study recommended open-ended responses with “trap questions” that could not be true for anyone (e.g., “I was born in the 18^th century).¹²

Other strategies involve rigorously examining survey data for duplicate, inconsistent, or suspicious responses. For example, geolocation data can identify respondents outside of the recruitment area, names and email addresses can be reviewed to identify duplicates or unusual formats, and phone numbers can be reviewed to identify inactive numbers or duplicates.^4–6,10 As part of a national online survey of young adults living with HIV in the United States, we describe a series of measures we used to ensure that only truly eligible human respondents were enrolled in our study.

METHODS

RESULTS

The results of our eligibility verification measures are detailed in Figure 1. We received a total of 1308 eligibility surveys, of which four (.3%) were incomplete, 50 (3.8%) indicated they were no longer interested before providing their contact information, and 48 (3.6%) were determined to be duplicates (i.e., surveys completed by the same person more than once). Duplicates occurred if a participant completed the survey twice in a short period of time or had already participated in the study once yet completed the screening survey again at a later date. Finally, 470 (35.9%) respondents were ineligible because they were under 18 or over 29 (n=142), or reported their HIV status as unknown or negative (n=240). Others opted not to provide documentation of their HIV status or age (n=88). Importantly, these are distinct from another group of responses that did not include verification documents as we discuss below. These were determined to be real human respondents (as opposed to bot responses) based on having responded directly to ads, and because they provided working phone numbers and email address. In three cases, the SC was able to reach the respondents via email or phone. These three participants indicated that they were not comfortable providing the requested documentation, while others did not respond to our requests.

Open in a separate window

Figure 1:

Results of Screening and Data Integrity Procedures

Of the remaining 739 potentially eligible responses, 413 (55.8%) were determined to be fraudulent or bot responses. These included 45 responses with uploaded images that did not confirm age (e.g., a selfie; n=5), HIV status (e.g., fake diagnosis letter, photo of non-antiretroviral medication; n=24), or both (n=16), as well as those who, according to geolocation data, were outside of the US (e.g., Zambia, Vietnam, Mexico; n=28). Because we opted not to use the “prevent ballot stuffing” Qualtrics feature, so as not to exclude partners or roommates using the same computer, it was possible for multiple surveys to be completed using the same IP address. Upon review by the SC, we identified 91 surveys that we determined to be “repeated bot responses,” a category that included some combination of IP address, name, email, or phone number repeated in multiple surveys over a short period of time (typically within a few minutes), and if none of those responses included complete screening information and verification documents. For example, on May 3, 2021, we received six responses from the same IP address within 33 seconds of each other. These responses all reported detectable HIV viral loads, contained invalid phone numbers, and all responded “no” when asked to upload age and status verification documents. These 91 “repeated bot responses” included only 37 IP addresses, 33 unique names, and 14 names that were repeated between 2–8 times within a few seconds or minutes of each other.

A final 246 surveys did not include age or HIV status verification documents which was due to, presumably, automated responses that were interrupted due to lack of document upload. In addition to not uploading verification documents, these survey responses also had inactive phone numbers, unusual names or email addresses, or a combination of these factors. A total of 326 (24.9% of all received eligibility surveys) unique survey links were sent to eligible participants, and 271 surveys were included in the final sample, giving us a survey completion rate of 83%.

DISCUSSION

We describe a new combination approach to reduce the threat of fraudulent participants and bot responses in online surveys. In anticipation of fraudulent responses and bots, we created a screening survey with various security measures in place. In addition to using built-in Qualtrics features (e.g., reCAPTCHA), we implemented a document-upload step and human review processes to evaluate screening data prior to sending the incentivized survey link.

Importantly, two of our most important recruitment integrity strategies were built into our study design by including individuals who were 18–29 year of age and living with HIV. We required potential participants to upload a photo ID showing their name, birthdate, and documentation of their HIV serostatus. Because of this document upload, bots were prevented from completing the survey and ineligible human respondents who uploaded fraudulent documents (e.g., photoshopped ID, faked letters of diagnosis) or did not respond when contacted, were excluded. Finally, study procedures included sending separate and unique Qualtrics survey links to each eligible participant which could not be shared with others or accessed by bots.

A number of verification strategies have been used in online-based studies such as having participants enter their names multiple times in the survey (e.g., for e-gift card receipt, emails, consent to be contacted for future research) which can be cross-referenced by the research team.⁶ Similar to other research,⁹ we reviewed names to identify possible duplicate responses and were able to identify screening surveys with repeat names, emails, and IP addresses (mostly completed within a short period of time). We also identified unusual names or name formats, which were often paired with an unusual and repeated pattern of email addresses (e.g., names followed by a series of random numbers or letters), which is an indication of bot email generation.^6,10,15–17 Lastly, geolocation data showed that the respondents were located outside of the United States. While built-in fraud detection settings are useful in detecting bots, they are grossly insufficient. Therefore, the review of names, email addresses, and phone numbers by the study team can be an effective check to ensure participant eligibility prior to enrollment in the incentivized study.

There are challenges to our approach worth noting. Requiring participants to upload ID and HIV status may lead to under sampling of some important groups (people who use drugs, incarcerated or undocumented) who may not have identification or may not be comfortable uploading these documents. Inconsistent access to broadband and slow internet speeds also present challenges. Lastly, human review of data is time-intensive and tedious, and there is inherent subjectivity when discerning the validity of self-reported data. Thus, we may have overlooked a fraudulent entry or denied a valid participant when evaluating screening survey data. Despite these potential limitations, our screening strategies were effective in identifying a large number of bot and fraudulent human responses. The use of technologies that may be useful in this process. For example, we may be able to use artificial intelligence in place of human review of images and documents. This can reduce the need for research coordinator time to review each image and it may be more precise in detecting fraud (e.g., detecting photoshopped images).

Innovative and multi-component fraud protection protocols are necessary to avoid threats to data security and improve data quality, but does require greater burdens to persons to pass these protections. Our approach and findings suggest that proactive fraud prevention can be built into the screening phase of the study to identify potentially fraudulent responses, prevent wasted resources related to data cleaning and unretrievable study incentives, and ultimately improve data quality. We encourage researchers to implement and share additional strategies in implementing valid online surveys and for journal editors and the peer review process to further scrutinize papers using online research methodology.

Funding:

REFERENCES