Duped by Bots: Why Some Are Better Than Others at Detecting Fake Social Media Personas
Abstract
Objective: We examine individuals' ability to detect social bots among Twitter personas, along with participant and persona features associated with that ability.
Background: Social media users need to distinguish bots from human users. We develop and demonstrate a methodology for assessing those abilities, with a simulated social media task.
Method: We analyze performance from a signal detection theory perspective, using a task that asked lay participants whether each of 50 Twitter personas was a human or social bot. We used the agreement of two machine learning models to estimate the probability of each persona being a bot. We estimated the probability of participants indicating that a persona was a bot with a generalized linear mixed-effects model using participant characteristics (social media experience, analytical reasoning, and political views) and stimulus characteristics (bot indicator score and political tone) as regressors.
Results: On average, participants had modest sensitivity (d') and a criterion that favored responding "human." Exploratory analyses found greater sensitivity for participants (a) with less self-reported social media experience, (b) greater analytical reasoning ability, and (c) who were evaluating personas with opposing political views. Some patterns varied with participants' political identity.
Conclusions: Individuals have limited ability to detect social bots, with greater aversion to mistaking bots for humans than vice versa. Greater social media experience and myside bias appeared to reduce performance, as did less analytical reasoning ability.
Application: These patterns suggest the need for interventions, especially when users feel most familiar with social media.
Recommended Citation
R. Kenny et al., "Duped by Bots: Why Some Are Better Than Others at Detecting Fake Social Media Personas," Human Factors, SAGE Publication, Feb 2022.
The definitive version is available at https://doi.org/10.1177/00187208211072642
Department(s)
Engineering Management and Systems Engineering
Keywords and Phrases
Analytical Reasoning; Myside Bias; Signal Detection Theory; Social Bots; Social Media
International Standard Serial Number (ISSN)
1547-8181; 0018-7208
Document Type
Article - Journal
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2022 Human Factors and Ergonomics Society, All rights reserved.
Publication Date
24 Feb 2022
Comments
Funding for this research was provided by the U.S. Army, Advanced Strategic Planning and Policy Program, Goodpaster Fellowship, and by the Swedish Foundation for Humanities and Social Science.