In summary
- Nearly 8,000 accounts on X use AI-generated profile photos to amplify political messages and Cryptocurrency schemes.
- 25.84% of the accounts with synthetic photos follow a pattern, having exactly 106 followers, which indicates possible coordination.
- Accounts with fake images tend to avoid direct interaction and focus on controversial topics, especially in English.
AI-generated profile photos are becoming a significant tool for coordinated manipulation on X, with researchers in Germany identifying nearly 8,000 accounts using synthetic faces primarily focused on amplifying political messages and cryptocurrency schemes.
“Recent advances in the field of generative artificial intelligence (AI) have blurred the lines between authentic and machine-generated content, making it nearly impossible for humans to distinguish between such media,” the study notes.
The research, conducted by teams from Ruhr University Bochum, the GESIS Leibniz Institute and the CISPA Helmholtz Center, found that more than half of these accounts were created in 2023, often in suspected mass creation events.
“A significant portion of the accounts were created en masse shortly before our data collection, which is a common pattern for accounts created for message amplification, disinformation campaigns, or similar disruptive activities,” the researchers explain.
This finding gains additional context from a recent analysis of the platform by the Center for Countering Digital Hate showing that X owner Elon Musk’s political posts favoring Donald Trump received 17.1 billion views—more than double than all American political campaign ads combined during the same period.
“At least 87 of Musk’s posts this year have promoted claims about the US election that fact-checkers have called false or misleading, racking up 2 billion views. None of these posts included a Community Note, the name X to user-generated data verifications,” the CCDH report says.
The use of generative AI—whether to generate fake images or text—was easy to detect as accounts with synthetic faces exhibited distinctive patterns that separated them from legitimate users. “Accounts with fake images have fewer followers (mean: 393.35, median: 60) compared to accounts with real images (mean: 5,086.38, median: 165).” The study also found that fake accounts tend to interact less with their ecosystem of followers and instead post messages without responding or interacting with other accounts.
The study also emphasized specific patterns that suggest coordinated activity: “We note that 1,996 accounts with fake images (25.84%) have exactly 106 followers. Our content analysis reveals that these accounts belong to a large group of fake accounts engaged in inauthentic behavior coordinated.”
The research team achieved remarkable precision in their detection methods, reporting almost 100% certainty in their findings.
The researchers also noted that many of the accounts do not last long, with more than half being suspended in less than a year.
Content analysis also revealed carefully orchestrated posting patterns in multiple languages. The study identified “large networks of accounts with fake images that were likely created automatically and that were involved in large-scale spam attacks.” English-language accounts focused heavily on controversial topics, with researchers finding that accounts preferred to address topics such as the war in Ukraine, the US election, and debates over COVID-19 and vaccination policies.
Outside of politics, many of these accounts also promoted cryptocurrency scams and sex-related content.
Looking ahead, the researchers plan to expand detection capabilities to identify AI images generated with other models based on different technologies—such as Diffusion models instead of Generative Adversarial Networks (GANs). They also want to improve their methodology to find more ways to identify what they categorize as “inauthentic behavior coordinated across social platforms.”
Edited by Josh Quittner and Sebastian Sinclair
Generally Intelligent Newsletter
A weekly AI journey narrated by Gen, a generative AI model.
For the Latest Crypto News, Follow ©KeynoteUSA on Twitter Or Google News.
Disclaimer: Please note that the information provided on this page is for News purposes only and should not be considered investment or trading advice. ©Crypto.keynoteusa.com strongly recommends that you conduct independent research and/or consult with a qualified professional before making any investment decisions.
