New Report Details Massive Mysterious Influence Campaign On Twitter

from the something's-happening dept

Over the weekend, the group released a fascinating report detailing what appears to be a massive influence campaign taking shape on Twitter. By way of disclosure, one of the three key authors of the report, Brett Horvath, is also one of the key people behind the election simulation game that we helped create and run, though I have nothing to do with this new report. The report is fascinating, and if you don’t feel like reading the whole thing, Bloomberg also has a write up.

The key to the report is that they have identified some truly fascinating patterns that they’ve spotted among a cluster of users on Twitter, who, at the very least appear to be acting in a manner that suggests some attempt to influence others. I should note that unlike other such reports that jump to conclusions, the authors of this report are very, very, very clear that they’re not saying these are “bots.” Nor are they saying these are Russian trolls. Nor are they saying that a single source is controlling them. Nor are they saying that everyone engaged in the activity they spotted is officially part of whatever is happening. They note it is entirely possible that some very real people are a part of what’s happening and might not even know it.

However, what they uncovered does appear strange and notable. It certainly looks like coordinated behavior, at least in part, and it appears to be designed to boost certain messages. The report specifically looks at statements on Twitter about voter fraud using the hashtag #voterfraud, but it appears that this “network” is targeting much more than that. What made the report’s authors take notice is that in analyzing instances of the use of the tag #voterfraud, they noticed that it appeared to have a “heartbeat.” That is, it would spike up and down on a semi-regular basis, based on nothing in particular. There wasn’t a specific news hook why this entire network would suddenly talk about #voterfraud, and they wouldn’t talk about it all the time. But… every month or so there would suddenly be a spike.

From there, they started digging into the accounts involved in this particular activity. And they found a very noticeable pattern:

We wanted to know how these accounts were coming onto Twitter and gaining mentions at such a high velocity ? what was leading accounts to gain influence, so quickly? So we took a sample set of accounts from a group of suspicious Voter Fraud accounts and started looking at their activity day-by-day, starting at day one. What we began to notice is a pattern for how the influence machine might be working, and how coordination could be happening.

Here’s the consistent network pattern we saw:

  • User signs up for an account.
  • User starts replying to multiple accounts?some known verified Twitter users and many other accounts that are also on our list of actors, or that fit a similar profile.
  • The replies tend to contain: text, memes, hashtags, and @mentions of other accounts, building on common themes.
  • At some point the pattern shifts from being all replies to original tweets. Those original tweets contain the same types of content as their replies do.
  • It appears that this pattern cycles and repeats when the next batch of new accounts come online. The next batch starts replying to the existing, newly influential accounts, and carry on with the same sequence of events for gaining influence.

The report highlights this pattern with a few example accounts, though the full study looked at (and continues to look at) many, many more. What you see over and over again are Twitter feeds of people who seem to do little other than constantly tweet pro-Trump memes and disinformation, and yet magically get thousands and tens of thousands of retweets, often coming out of nowhere. Here’s one example:

The gray line at the bottom is the number of tweets. The black line is the number of mentions from others. Notice how it goes from nothing to around 10,000 in no time? Sometimes the accounts are more or less dormant for a while, before suddenly becoming massively popular for no clear reason at all:

Again, as the report makes clear, these aren’t necessarily bots (though, they may be). They aren’t necessarily even aware that they’re a part of something. But the patterns seen over and over and over and over again are uncanny. And it certainly provides strong circumstantial evidence of some sort of influence operation — and it’s one that appears to continue to grow and grow.

As the report notes:

We don?t know why this activity is occurring, or who is behind it. However, the best we can do is look at the data around what?s actually happening. What we’ve discovered along the way is that there are overlapping patterns of behavior, demonstrating some form of coordination.

We think it’s possible that some of these accounts don’t realize that they’re coordinating or part of a larger influence network. For example, one of these sample accounts might genuinely care about Voter Fraud. A bad actor, coordinating large numbers of accounts could find this person?s tweets useful, then amplify those tweets through thousands of @mentions and replies.

By focusing on the hard data around coordination, we can better understand how public conservations are being distorted and how it affects society. Whatever your views are on Voter Fraud, these accounts and the accounts that amplify them are rapidly accelerating their activity in the lead-up to Election Day.

Similarly, of course, it’s not clear that this is actually having any impact on anyone’s views. But it’s at least worth looking at what happens when there is what appears to be massively coordinated activity, mostly focused around spreading disinformation regarding the election and more. The full methodology of the report is available on the site, as are the names of 200 of the accounts studied.

What’s fascinating, of course, is the sheer size of what’s happening, and the level of coordination necessary to make it happen. Twitter’s response to the report (as noted in the Bloomberg article) is pretty much what you’d expect Twitter to say:

?While we prohibit coordinated malicious behavior, and enforce accordingly, we?ve also seen real people who share the same views organize using Twitter,? the company?s statement said. ?This report effectively captures what often happens when hot button issues gain attention and traction in active groups.?

Indeed, that’s part of what’s so tricky here. Could this kind of thing happen organically? Well, certainly much of it can. Lots of people who share the same views on any particular subject often will see surges in conversations around those topics, including lots of retweets, mentions and replies. But the pattern here definitely looks different. When these things happen organically, they tend to have a fairly different rhythm, either a lot more sustained, or the spikes are much more spread out and explainable (e.g., there was some news event that tied to the topic). Similarly, it is hard to see how so many pseudononymous people, who no one else really knows, magically all jump up to thousands or tens of thousands of mentions with no clear explanation for their sudden and sustained fame.

But this is also why Twitter is put in an impossible position if it’s expected to spot all of this. Even with so much evidence, it’s still possible that what spotted was organically formed. It may seem unlikely, but how can you tell? And you can bet that there are some with less than virtuous intent, who are actively figuring out ways to increasingly make all of this activity look organic. Expecting that Twitter, or any company, can always magically determine what is and what is not “authentic” behavior online, is an impossible task. And the very fact that it might sweep up some perfectly innocuous accounts in the process also makes it troubling to expect that the platform should be in charge of sorting out who’s who and who’s real in these kinds of situations. But, then again, if these kinds of disinformation campaigns truly are having an impact on influencing the public, that too should be a concern. Either way, as the report highlights, there is still much work to be done in analyzing how social networks are being used to influence the public.

Filed Under: , , , , ,
Companies: twitter

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “New Report Details Massive Mysterious Influence Campaign On Twitter”

Subscribe: RSS Leave a comment
Anonymous Coward says:

Re: Re: Re:

I’m not a fan of internet voters, but someone who lives in the region, cares enough to vote, and doesn’t happen to be a US citizen? Hell, bring ’em on. At least they want to perform the duties of a citizen and participate in the process.

And yeah, I mean, hypothetically Mexico or China could send hundreds of thousands of people to live in a county just to try and swing the election that way, but… really?

Anonymous Coward says:

Re: Re: Re: Re:

hypothetically Mexico or China could send hundreds of thousands of people to live in a county just to try and swing the election that way, but… really?

Ah, but in China’s case, they don’t send hundreds of thousands of people expressly for this purpose; they just attempt to exploit the hundreds of thousands who already emigrated but still have ties (business, family, etc.) back to the mainland.

Recent example: — one of the mayoral candidates in Richmond appeared to be hand-picked by the Chinese government.

China appears to be doing this mostly at the local level; once they’ve got a foothold there’s nothing to prevent them moving it up to a state/province level though. And the techniques aren’t just limited to influencing Chinese ex-pats; they’ll also scoop up anyone wanting to make a quick buck or get some cheap flights.

Anonymous Coward says:

Re: Re: Re:2 Re:

Interesting! Thank you for the link.

On the other hand, it doesn’t report on the result/success rate/which candidate they were trying to influence, which makes it hard to see how big of a deal it was.

I think that should be considered, if only to determine whether the actual danger is affecting the results of elections (A few hundred votes seems unlikely to be telling, though it’s certainly alarming) or the reputation of free and fair elections and the integrity of the voting process. (Russia and China both have motivation to try and undermine the latter and dissillusion people with the system.)

Anonymous Coward says:

Re: Re: Re:2 Re:

I wonder if these Foxconn workers from China will be H1Bs. Also, will they be allowed to vote? /s

Some of the Chinese workers have said they do not want to live in Wisconsin. LOL

Foxconn Considers Bringing Chinese Workers to Wisconsin as U.S. Labor Market Tightens

Darkness Of Course (profile) says:

Re: Re: Re: Re:

Do you seriously not know what it takes to be a citizen, and a voter in your burg be it little or huge. Just being Chinese and having a P.O. Box will not do it. So shipping lots and lots of Chinese in to shift an election in the USA isn’t gonna fly. EVER!

In my middle sized burg: You must be an American citizen. A resident of the burg and at least 16 to register then 18 by the election. Otherwise they might as well stay in China.

Or in the magical thinking land of Da Trump and his Ettes.

Rekrul says:

Both Black Mirror and The Orville had episodes where a global net service could effect people’s lives by how many up or down votes they got. We’re rapidly approaching that point with Twitter and Facebook. They tell people who to vote for, who is good, whose life needs to be upended over some allegations, etc. Post the wrong thing or offend someone and they will make it their mission in life to destroy you and to get as many of their “followers” to do so as well.

Darkness Of Course (profile) says:

Well, what have we here?

Take a sample of the twitter users.

Interesting, no pictures of themselves, their family or their pets. No shots of sunsets, sunrises, the family car. The pictures are repeated hundreds of times, all post many, many pictures of their photo set.

Nothing but processed (photo shopped or done professionally) GOP/Trump-think images. And the ever believable, “I am not a racist” posts. Shortly after calling Dems commies, or being a proud nationalist.

Oddly enough they sound exactly like a racist, act like a racist, bitch like a racist. Me thinks the racist does protest too much.

Zof (profile) says:

Social Media Isn't Social Media Anymore

Comments Sections aren’t comments sections anymore. They are “idea farms”. See, I’ve been reading the marketing section on how easy it is to build your own following with an AI Cloud “community and advocacy hub”. There’s literally a product you can buy now, if you are lazy. Like, if you don’t want to hire your own Network Operations Center (NOC) and you want to outsource that, you can. It’s going to suck, but you can. This is that same kind of thing, but you allow an AI to do it, and sign forms that say you won’t sue if it messes up really bad somehow.

So yeah, you can make a fake PeTa if you want with one of these advocacy farms, but make it for anything. Like your lame candidacy for town fire whatever. And it will have fake wholesome AI bots pretending to be real people chatting up your website visitors and “forming meaningful relationships that lead to support you can use, at the push of a button”.

The reason this is relevant is, I noticed something about a week ago when I finally quit social media forever.

Nobody is talking to each other anymore.

They have somehow made made things so bad now, we all just talk to the Ether, waiting for replies (like I am now, Like we all are, Like we all always are) but never directlyi at each other.

We have literally been conditioned to just shout at each other in rooms like this, basking in the attention of a number incrementing on an approval indicator rather than expecting some meaningful conversation.

(because that’s more efficient and they make clicks and money faster. Man that boy that thought this up was a genius)

This entire system in place now, with AI controlled comments sections, and AI deciding who to ban and who to keep on Twitter (Oh, unless you think a human being is doing that. Handling the judgement call on hundreds of millions of… anymore. Noob. Still think PEOPLE at the IRS do your taxes? Do you child?

Most of our communication now is THEM to us. That was unheard of even say, 20 years ago. But now a good 90 percent of your TOTAL communication is between you and your interests rather than people. And they got your interests. They got that Game of Thrones. They got that hate boner waiting for you on Google News. Go justify being a hateful loser. The news media supports that now. They got those worldviews you should have this week on CNN. Whatever the wealthy 1 percenters on CNN think you should believe that week. For example, Maybe Warren Buffet has paid for the Media to say nice things about Apple again for that week. Since he had just invested a bunch of money into them, and didn’t want them mentioning that they are now just a third place loser in smartphones. They got that new sexy Doctor Who spouting those Buzzfeed talking points. Because that’s what the doctor does. Hang out on Earth and handle social justice issues.

Point is, that’s not you talking to me. That’s not us talking to each other. That’s us being nodes. And at some point they won’t even need us anymore. It will be most cost effective to handle us with a “pocket AI” that just decides what we spend money on to “keep the economy stable”.

If that isn’t already how things are. Kinds feels like it huh.

: )

This will probably be my last post here. I’ve suddenly decided this well is poisoned. I just spent six months following a careful plan to leave social media entirely and replace it with “actual media” where I have “friends” in “real life” again like I used to say, 20 years ago.

I figured out a good 40 percent of what I thought I “liked” was just me trying to appease imaginary judges

Example. “Actually, I kinda hate that TV show. A lot. I just felt I HAD to like it because of all the pressure from twitter”. I kept watching a terrible tv show because I was conditioned to believe I was a “bad person” if I did not like that show.

This was a year ago. Then, with a few months, boom, this huge controversy when absolutely terrible content from low rent talent was exposed to have hired the same PR astroturfing firms to make their shows about “sexism” or “racism” to promote them, and that’s how terrible content from people with zero talent was suddenly both in the news, and being shame-sold to me by twitter AI bots.

Again, a year ago…

bob says:

Re: Social Media Isn't Social Media Anymore

So I get what you are trying to do and say. And maybe your particular case requires stopping everything for a time to reset your perceptions and then slowly reconnect to life and maybe in the future reconnect through electronic media. At least you recognize your own situation.

Yes, it would be nice to have more conversations about topics here on techdirt but conversations take 2 people. I won’t for a second assume anyone here is a friend. Most are friendly but of course there are still problem children that try to cause problems. But I also like that I can state my point of view and debate the merits of it with knowledgeable people.

Maybe don’t read the comments if they get to you. You can read full articles from the main page.

Anonymous Coward says:

Re: Re: Social Media Isn't Social Media Anymore

“…of course there are still problem children who try to cause problem”

Ah. You mean, like, “disagree”? So, when you refer to “having conversations with knowledgeable people” you mean “having others reinforce my virtuous rightness through conversation”.

Sure. That’s one way to describe it. That’s also, minus the problem children, an echo chamber.

Ninja (profile) says:

Re: Social Media Isn't Social Media Anymore

I don’t see this specific site as some sort of social media like Twitter or Facebook.

Anyway, I do agree with quite a bit of your post. I’ve stopped using most social media as means for interaction a while ago. Nowadays I mostly use instant messengers with people I know personally and I prefer to meet and talk in person than via phones. It’s actually healthy for your mind I found out.

Mike Masnick (profile) says:

Re: decide

You need to decide whether you support the 1st amendment and quit dicking around.

How is anything I’ve written "dicking around"? Especially regarding my support for the 1st Amendment?

You don’t get to go hunting online for people who oppose you politically and start raising red flags about it if you want a 1st amendment.

Where the hell have I done that? Literally today I posted something arguing about people suing Trump — who I oppose not for political reasons, but because he’s a fucking disaster — noting that their arguments go against the 1st Amendment and could do real damage. I’ve written about many people I disagree with politically and supported their 1st Amendment rights.

Nothing in this post goes anywhere near arguing against anyone’s 1st amendment rights.

I’m good either way you go. Just a warning.


Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...