Twitter Testing Secret Filter To Stop Abuse: Is That A Good Thing Or An Attack On Free Speech?

from the this-isn't-easy dept

For quite some time now, it’s become popular for people to point to some (often horrible) abuse on Twitter and demand that Twitter “do something” about it. It’s rather easy to find tons of articles calling for Twitter to do more.

And apparently Twitter is listening. As Sarah Jeong notes in a great article over at The Verge, it appears that Twitter recently deployed a feature to block some very abusive tweets, going further than its past tactics of pulling abusive tweets and killing accounts. In this case, it blocked certain tweets from being sent altogether. It appeared to be a filter that combined certain keywords with an @ symbol towards a person receiving a lot of abuse. Of course, as with many such things, the abusers just sought ways around the filter.

For a while, at least, Berger didn?t receive any tweets containing anti-Semitic slurs, including relatively innocuous words like “rat.” If an account attempted to @-mention her in a tweet containing certain slurs, it would receive an error message, and the tweet would not be allowed to send. Frustrated by their inability to tweet at Berger, the harassers began to find novel ways to defeat the filter, like using dashes between the letters of slurs, or pictures to evade the text filters. One white supremacist site documented various ways to evade Twitter?s censorship, urging others to “keep this rolling, no matter what.”

The assumption, from those with at least some understanding of what happened, is that it was done via Twitter’s spam filter:

A source familiar with the incident told us, “Things were used that were definitely abnormal.”

A former engineer at Twitter, speaking on the condition of anonymity, agreed, saying, “There?s no system expressly designed to censor communication between individuals. ? It?s not normal, what they?re doing.”

He and another former Twitter employee speculated that the censorship might have been repurposed from anti-spam tools?in particular, BotMaker, which is described here in an engineering blog post by Twitter. BotMaker can, according to Twitter “deny any Tweets” that match certain conditions. A tweet that runs afoul of BotMaker will simply be prevented from being sent out?an error message will pop up instead. The system is, according to a source, “really open-ended” and is frequently edited by contractors under wide-ranging conditions in order to effectively fight spam.

Trollish behavior and abusive behavior is definitely a concern. And making people feel welcome online on services like Twitter, without fear of harassment seems like a welcome goal. But there are also some pretty serious concerns about how this is all happening in a non-transparent manner, leaving it open to abuse in its own way:

What?s worrisome to free speech advocacy groups like the EFF about this incident is how quietly it happened. Others may see the bigger problem being the fact that it appears to have been done for the benefit of a single, high-profile user, rather than to fix Twitter?s larger harassment issues. The selective censorship doesn?t seem to reflect a change in Twitter abuse policies or how they handle abuse directed at the average user; aside from a vague public statement by Twitter that elides the specific details of the unprecedented move, and a few, mostly-unread complaints by white supremacists, the entire thing could have gone unnoticed.

Eva Galperin thinks incidents like these could be put in check by transparency reports documenting the application of the terms of services, similar to how Twitter already puts out transparency reports for government requests and DMCA notices. But while a transparency report might offer users better information as to how and why their tweets are removed, some still worry about the free-speech ramifications of what transpired. One source familiar with the matter said that the tools Twitter is testing “are extremely aggressive and could be preventing political speech down the road.” He added, “are these systems going to be used whenever politicians are upset about something?”

It opens up some pretty serious questions, and you enter into the same slippery slope that questions around legislating against harassing speech, or even things like revenge porn, start to raise some concerns. It’s not that anyone wants to support those activities. The harassment here is absolutely deplorable. But there is a serious concern about where it leads when an intermediary suddenly takes it upon itself to determine what is and what is not acceptable speech. Was it used for good reasons in this case? Probably. But will it always be done in that manner? That’s where it gets a lot trickier.

Filed Under: , ,
Companies: twitter

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Twitter Testing Secret Filter To Stop Abuse: Is That A Good Thing Or An Attack On Free Speech?”

Subscribe: RSS Leave a comment
Anonymous Coward says:

Wrong approach to the problem

No matter how worthy the goal, trying to suppress undesirable speech with automated filters is a poor use of resources. E-mail spam, and more recently general comment spam, have shown this repeatedly. If automated systems could reliably recognize good and bad speech, spam would not be a problem. Instead, most people hide behind spam filters and hope that the filter does not eat anything important.

Among other problems, what if someone wanted to tweet a supportive (if vacuous) statement like “@Berger is a good person. Don’t call her a rat.” or an informative (in certain contexts) statement like “@Berger, when you travel to #NewYorkCity, beware the rats. They’re mean and usually diseased.” Both of these are harmless to good statements, yet would likely be wrongly flagged by the filter because it happens to contain a word that has been twisted to be a slur.

Michael (profile) says:

Re: Re:

While this is not a first amendment issue, it is something that should be concerning for people that want free speech.

Their motivation is important. If they are doing this kind of filtering under pressure that if they do not they may be held liable, this kind of thing can have a chilling effect not only on the use of Twitter, but on other companies that may want to compete in the same space.

Proponents of free speech should always cheer on companies and services that allow free expression of unpopular ideas and frown upon companies that quell speech in any way.

Anonymous Coward says:

Re: Re: Re:2 Re:

reach as many people as Twitter
That’s the beauty of a free and open internet; all sites are equally accessible. Centralization and censorship only exist so long as people support it.

I’d look into supporting and/or hosting your own StatusNet or instance. They’re by no means perfect, but supporting and using software that guarantees rights is a much better long term option than asking entities to grant you rights that they have no obligation to. Especially as this censorship becomes more concealed and insidious.

art guerrilla (profile) says:

Re: Re: Re:2 Re:

twitter could go away tomorrow, and the only thing that would happen, is people would find other avenues to waste time…

i have NEVER (except excerpted in articles) seen a tweet (sheesh i feel like i’m losing IQ points just using this stupid lingo) or care to; my life is NOT poorer for that…

that there are a few REAL dissidents, occupy types, etc who have used it to good effect is besides the point, it is mostly useless…

Tony (profile) says:

Re: Re: Re:2 Re:

I personally have written such a platform – and I can tell you from experience that the main problem with ANY such platform is adoption.

If you can get enough people to start using your platform, it then becomes viable. The problem is, everybody is on Twitter, so nobody wants to go anywhere else because nobody is there.

When they don’t like what Twitter does, people just quit using it instead of pursuing alternatives. It’s the easy (lazy) approach – getting your friends to sign up for another platform is just too much work.

Rich Kulawiec (profile) says:

Content-based measures don't work and won't work

We — well, some of us anyway — learned that in the anti-spam world years ago. Any sufficiently-clueful adversary can defeat content-driven filters at will, either by (a) performing sufficient testing using their own copy of it (if open-source) or (b) performing sufficient testing to reverse-engineer it (if closed-source). Of course not all adversaries are that clueful, and not all of the ones who are have the motivation to tackle the problem, but that still leaves enough to defeat all such measures. (And some of the enterprising ones will sell the solution to others who are less motivated and/or less clueful, but willing to pay for a ready-made solution.)

Content-based measures do have their uses: properly deployed, they can yield interesting and useful information about possible sources of spam, phishing, typosquatting and other forms of abuse. But the best use for that information is to collect it, collate it, summarize and then bring it to the attention of (well-trained) humans, who will hopefully exercise due diligence and will use out-of-band data to augment and check their conclusions.

That’s not what most people want to hear: they want to hear that the problem can be completely solved via automation, because that’s cheap and easy. But it can’t, and attempts to do so inevitably devolve into a cyclical pattern of (new) attack and (new) defense, followed by (newer) attack and (newer) defense, followed by more of the same. We’ve seen this with anti-virus software. We’ve seen this with anti-spam software. And now we’re seeing with anti-whatever-it-is that Twitter wants to call this.

Marcus Ranum, in his brilliant rant “The Six Dumbest Ideas in Computer Security“, see for the whole thing, nailed this nearly a decade ago, as dumb idea #2: Enumerating Badness. He says in part: “One clear symptom that you have a case of “Enumerating Badness” is that you’ve got a system or software that needs signature updates on a regular basis […]”. Bingo.

Just Another Anonymous Troll says:

Pointless and dangerous

This is pretty darn stupid. It’s pretty much impossible to stop harassment with any sort of speech filter whatsoever. Inevitably, people can and will find a way around it, and then it will inconvenience your other users. There’s really no point, and call me paranoid but I can see the National Spying Agency serving them up a National Security Letter forcing them to censor anti-government tweets.

Anonymous Coward says:

Re: Why is this complicated? Just give spam control to users

Agreed. When it comes to social networking applications and communications, I reserve it for what an acquaintance of mine calls “the Three F’s”

Friends, Fun, Family.

Many applications do anyway, but I see no reason why I as a user can’t have the tools to restrict my own coversations to suit what I want from it.

Anonymous Coward says:

sticks and stones may break my bones ...

A big problem here is that this person is a member of Parliament. People who are thin-skinned and easily offended should not be government officials, period. It also should go without saying that people who get relentlessly trolled are usually troll feeders. Many of us learned as young children that anyone who couldn’t take teasing would end up getting a lot more of it. Apparently some adults never learned this simple fact of human nature. And the more extensive the text filtering mechanisms, the more determined trolls will be to crack them, and then rub it in everyone’s face.

But as is usually the case, people who are high up tend to get special privileges not afforded to regular people, and these special privileges have a tendency to increase over time.

“The harassment here is absolutely deplorable.”

And that’s the whole point. There have been numerous cases in the past in which bigoted speech (or misconduct in general) by anonymous people turned out to be a “false-flag operation” done to create sympathy for the victim and demonize the side thought to be responsible. Even in “legitimate” cases, an agent provocateur may be planted behind the scenes. We need to keep in mind that these childish antics are not always what they seem.

Anonymous Coward says:

Of course, there is genuine harassment on these services but this can and will be abused by ideologues to silence dissent. Feminists, religions, etc. have a history of false flagging videos off youtube, getting twitter accounts banned, doxxing, etc. and there was no abuse or harassment, just criticism and facts proving them wrong. Those groups are often the ones being abusive and yet their accounts stay active. Obvious agenda is obvious.

Groaker (profile) says:

not feasible

An inordinate number of people seem to feel that they have a right not to be offended. Interestingly enough many of these same people are incredibly offensive to others, believing that their upbringing and opinions are just the natural order of the universe.

There are some two million words in English, and I would defy any filter to be able to deal with them, never mind the implications of an enormous number of ways to interpret combinations.

Quite a few companies on the net have gone out of business attempting censorship

Insurgence (profile) says:

To me it is fairly simple. Their servers their rules. If you want free speech then setup your own system for publishing. Then you can control what is published. That is the only way to guarantee free speech. If twitter was going out and hampering your ability to say something elsewhere then I would say the company was violating your free speech, but they are not. So it is completely within their rights to block content they find unacceptable and not be transparent about it.

If you do not like it then use another service. And that is another reason why I do not consider it violation of anyons freedom of speech. It is not the only way to say something on the internet, just one of the most widely used. That does not justify concerns about free speech.

Anonymous Coward says:

Re: Re:

“To me it is fairly simple. Their servers their rules.”

Who said otherwise?

“So it is completely within their rights to block content they find unacceptable and not be transparent about it.”

Again, who said otherwise. Straw-man much?

“And that is another reason why I do not consider it violation of anyons freedom of speech.”

But if you’re trying to twist that into saying that no one’s speech is being restricted, then you’re just wrong. No one is saying that Twitter is breaking any laws. Just pointing out what how they are restricting speech. And restricted speech is not free speech.

Sheogorath (profile) says:

Freedom of speech upheld

The harassment here is absolutely deplorable, but there is a serious concern about where it leads when an intermediary suddenly takes it upon itself to determine what is and what is not acceptable speech.
Actually, Twitter didn’t decide what people are allowed to say, they simply upheld the harassment recipients’ right to freedom of speech by blocking stuff aimed directly at them. While people have a right to say what they like, that right ends when it interferes with the rights of others, and tweeting @[name] to put something directly into someone’s Twitter stream forces them to see it, thus interfering with their right to freedom of speech. Simples!

Pragmatic says:

Re: Freedom of speech upheld

Uh, I see where you’re coming from, but that’s not a freedom of speech issue, it’s a freedom from harassment issue. Freedom of speech =/= freedom to abuse other people.

The Open Tweet Filter recommended by NoahVail seems like the most reasonable response. They can’t bug you if they can’t get to you.

Add Your Comment

Your email address will not be published.

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...