Creator Of Botometer Goes On Media Tour To Explain Why Elon Musk’s Claims About Bots (Using Botometer) Are Meaningless
from the convince-me-musk-is-not-a-bot dept
As you may recall, in his response to Twitter’s lawsuit trying to force him to fulfill the terms of the purchase agreement he made, Elon Musk relied on the findings of a tool called Botometer to argue that there were more bots on Twitter than Twitter was claiming. Again, I have to remind everyone, as much as Musk keeps insisting this case is about bots and spam, the actual case has nothing at all to do with bots or spam, and if you think it does, you’ve been lied to.
However, Musk is doing everything he can to confuse people, including the judge, into believing the case is about bots and spam. And, in doing so, he feels that he needs to convince people that there are a lot of bots and spam on the platform (he claims more than Twitter admits to, but he — and many others — are totally misrepresenting what number Twitter is actually reporting). In his attempt to confuse everyone, in the lawsuit, Musk cited results from Botometer — which made tons of experts in the space laugh. Botometer is a fun toy to play around with, but no one takes it seriously as an actual tool to determine how many bots are on Twitter.
Including Botometer’s creator.
In the wake of Musk putting all of his eggs in Botometer’s basket, it appears that a creator of Botometer has been making the media rounds pointing out that Musk is a fool.
A couple weeks ago, grad student Kai-Cheng Yang from Indiana University was interviewed by Yahoo Finance, and helped debunk some of Musk’s talking points. First, he notes that what Twitter is reporting to the SEC as bots in its monetized daily active users is a different thing than Botometer tries to analyze. There is some overlap, but they’re looking at different things:
I think Twitter has made it clear that they are focusing on spam and the false accounts. And my understanding of their definition about spam accounts is that those kind of accounts would send repeatedly all different kinds of information, trying to promote some website or some product or some cryptocurrency to people, kind of annoying, right? But you can achieve those kind of goals through bots, of course. But also you can have real people control those accounts. In my opinion spam accounts has an overlap with social bots, which is what we detect. But also it’s not entirely the same thing.
He also pours some cold water on an oft-cited claim from five years ago, made by the team behind Botometer, that it believed between 9% and 15% of Twitter was bots. As he notes, that was a long time ago — Botometer has changed a ton since then and measures things differently, and (perhaps more importantly) Twitter’s own approaches to dealing with bots and spam has changed drastically as well.
Yes. So our group did those kind of estimations back then. But I do want to mention that we’ve been upgrading our tools constantly. So the Botometer today is different from what we have before, right?
And also the situation on Twitter has been changing quickly, because Twitter also, they have been doing a lot trying to remove bots and other inauthentic accounts. So actually, I think they drive to the bad actors to change their behavior, to change their accounts. So I am not sure that estimation is still accurate today, unfortunately.
Finally, when asked about Musk’s use of their tool to calculate bots, Yang basically says that users of his tool can effectively tweak the threshold to churn out any number they want for bots, so without knowing that (which Musk hasn’t disclosed), you can’t tell whether or not his estimate is reasonable:
Our tool works, if you give it account, it will give you a score. If the score is higher, it means the account is bot-like. If the score is low, it means it’s human-like. But it’s a score. So in order to have a number of percentage of bots on Twitter, you have to choose a threshold, right?
And that’s, I don’t know how Elon Musk did it. And technically you can choose any threshold you want and to get any result you want. So that’s my understanding right now. Elon Musk didn’t make it clear how they choose this threshold to me.
This is all sorts of funny, because throughout this whole thing Musk keeps complaining that Twitter hasn’t given him the relevant thresholds on how it scores spam (except, as noted at the bottom of this article, it has), and now the creator of the tool that Musk is relying on says that Musk is actually hiding the most important part of his own analysis.
The very next day, Yang also did an interview with CNN where he was even more aggressive in highlighting how absurd it is that Musk is relying on his tool.
“To be honest, you know, Elon Musk is really rich, right? I had assumed he would spend money on hiring people to build some sophisticated tool or methods by himself,” Yang told CNN Business Monday. Instead, Musk opted to use the Indiana University team’s free, publicly available tool.
He also reminded CNN that spam and bots are not the same thing, and all his tool does is try to determine automated accounts — many of which are legit and not spam.
He further delved into the important threshold setting question that basically allows anyone using his tool, Musk or not, to falsely imply something by fiddling with the threshold:
“It’s tempting to set some arbitrary threshold score and consider everything above that number a bot and everything below a human, but we do not recommend this approach,”
And then, last week, he did yet another interview, this time with the BBC, this time being even more aggressive in saying that Musk’s use of his tool is nonsense.
Using the tool, Mr Musk’s team estimated that 33% of “visible accounts” on the social media platform were “false or spam accounts”.
However, Botometer creator and maintainer, Kaicheng Yang, said the figure “doesn’t mean anything”.
As Yang explained, again, Musk doesn’t say what threshold he used, and that allows him to say whatever he wants.
“In order to estimate the prevalence [of bots] you need to choose a threshold to cut the score,” says Mr Yang.
“If you change the threshold from a three to a two then you will get more bots and less human. So how to choose this threshold is key to the answer of how many bots there are on the platform.”
Mr Yang says Mr Musk’s countersuit does not explain what threshold it used to reach its 33% number.
“It [the countersuit] doesn’t make the details clear, so he [Mr Musk] has the freedom to do whatever he wants. So the number to me, it doesn’t mean anything,” he said.
The BBC also spoke to Michael Kearney, who created another bot-measuring tool, Tweet Bot or Not, who pointed out the same thing Yang keeps trying to explain:
“Depending on how you define a bot, you could have anywhere from less than 1% to 20%,” he says.
“I think a strict definition would be a fairly low number. You have to factor in things like bot accounts that do exist, tweet at much higher volumes,” he said.
Of course, the reality remains that this case isn’t actually about bots and spam. Musk is leaning heavily on convincing his adoring fanbase it is, and as such, a tool like Botometer serves the job. He needs propaganda, not facts, and thus any tool will do, no matter how misused.