When Twitter first launched what it called “Birdwatch,” I was hopeful that it would turn into a useful alternative approach to helping with trust & safety/content moderation questions, but I noted that there were many open questions, in particular with how it would deal with malicious actors seeking to game the system. When Elon took over Twitter, he really seemed to embrace Birdwatch, though he changed the name to the pointlessly boring “Community Notes.”
I still think the concept is a good one, and think it’s one of Elon’s few good moves. I think other social media sites should experiment with some similar ideas as well.
The problem, though, is that Elon seems to think that Community Notes is an effective replacement for a comprehensive trust & safety program. At the heart of so many of Elon’s decisions in firing the vast majority of the company’s trust & safety staff was that “Community Notes can handle it.”
As we’re in the midst of a series of major crises around the globe, where the flow of information has proven incredibly important, one thing we’re clearly learning is that Community Notes is not up to the task. Just to drive this point home, over the weekend Elon himself posted some fucking nonsense (as he’s prone to do) and many hours later Community Notes pointed out it was hogwash. Elon, as he’s done in the past when he’s been “Noted,” claimed he was happy it happened to himself… before claiming that his post was “obviously a joke meme” and that “there is more than a grain of truth to it.”
So, first of all, there isn’t “more than a grain of truth to it.” The whole thing is simply false. But, more importantly, if you look at the top replies to his “obviously a joke meme,” suggests that Elon’s biggest fans did not, even remotely, think that this was “obviously a joke meme,” but rather took it entirely seriously, cheering him on for “telling the truth.” Here’s just one of the top replies to his original tweet:
Also, it took quite some time for the note to appear on Elon’s account. And, look, content moderation at scale is impossible to do well and all that, but Community Notes seems like the exact wrong approach in situations like this one. Especially at a time when the accounts pushing out the most viewed news these days seem to be made up by a combination of grifters and idiots:
Online we have seen many users of X describe their experience of this crisis as different. Some of that may result from the more ambiguous nature of the larger conflict, especially as the news cycle moves from the unambiguous horror of the initial attack to concerns about Israel’s response. However, our investigation here suggests an additional factor: in Musk’s short tenure as owner of the platform, a new set of news elites has emerged. These elites post frequently, many sharing unvetted content and emotionally charged media. While sharing no single political ideology, many embrace a similar culture of rapid production of unlinked or ambiguously sourced content, embracing a “firehose of media” ethos that places the onus of verification on the end-user. This occurs in an environment that has been shorn of many of the “credibility signals” that served to ground users in the past — checkmarks that indicated notability, fact-checks distributed through Twitter Trends, and Twitter/X-based labeling of deceptive content. Even fundamental affordances of the web — such as simple sourcing through links — have been devalued by the platform, and, perhaps as a result, by the new elites that now direct its users’ attention.
Leaving aside the significant concern of taking away professional, trained trust & safety employees, and replacing them with random (often hand-picked) untrained volunteers, there are serious concerns coming to light about how Community Notes actually works in practice.
Multiple reports have come out lately highlighting the limitations of Community Notes on important breaking news in the midst of various conflicts around the world, where you have malicious actors seeking to deliberately spread misinformation. A report at Wired found that Community Notes is actually making some of the problems worse, rather than better.
On Saturday, the company wrote on its own platform that “notes across the platform are now being seen tens of millions of times per day, generating north of 85 million impressions in the last week.” It added that thousands of new contributors had been enrolled in the system. However, a WIRED investigation found that Community Notes appears to be not functioning as designed, may be vulnerable to coordinated manipulation by outside groups, and lacks transparency about how notes are approved. Sources also claim that it is filled with in-fighting and disinformation, and there appears to be no real oversight from the company itself.
“I understand why they do it, but it doesn’t do anything like what they say it does,” one Community Notes contributor tells WIRED. “It is prone to manipulation, and it is far too slow and cumbersome. It serves no purpose as far as I can see. I think it’s probably making the disinformation worse, to be honest.”
The report isn’t just based on random Community Notes users, but looking more closely at how the program works, and the ability for it to be gamed. Wired found that it wasn’t difficult to set up multiple accounts controlled by one person which all had access to Community Notes, meaning that you could manipulate support for a position with just a small group of users controlling multiple accounts.
It also points to earlier (pre-Elon) research that showed that (then) Birdwatch wasn’t used nearly as much for standard fact checking, but was used in political debates by users who disagreed politically with someone who had tweeted.
Back during the summer, the Poynter Institute had a good analysis of the limitations of Community Notes for dealing with real-time misinformation campaigns during crises. Specifically, the design of the current Community Notes has some, well, questionable assumptions built in. Apparently, it looks over your tweeting history and assigns you to a camp as being either “left” or “right” and then only allows a Community Note to go public if enough of the “left” people and the “right” people agree on a note.
“It has to have ideological consensus,” he said. “That means people on the left and people on the right have to agree that that note must be appended to that tweet.”
Essentially, it requires a “cross-ideological agreement on truth,” and in an increasingly partisan environment, achieving that consensus is almost impossible, he said.
Another complicating factor is the fact that a Twitter algorithm is looking at a user’s past behavior to determine their political leanings, Mahadevan said. Twitter waits until a similar number of people on the political right and left have agreed to attach a public Community Note to a tweet.
While that may work on issues where there isn’t any kind of culture war, it’s completely useless for culture war issues, where plenty of disinformation flows. Indeed, the Poynter report notes that a huge percentage of the highest rated Community Notes inside the Community Notes system are never seen by the public because they don’t have “cross-ideological agreement.”
The problem is that regular Twitter users might never see that note. Sixty percent of the most-rated notes are not public, meaning the Community Notes on “the tweets that most need a Community Note” aren’t public, Mahadevan said.
The setup with “cross-ideological” consensus basically seems almost perfectly designed to make sure that the absolute worst nonsense will never have Community Notes shown publicly.
Meanwhile, a report from NBC News also highlights how even when Community Notes is able to help debunk false information, it often comes way too late.
NBC News focused on two prominent pieces of Israel-Hamas misinformation that have already been debunked: a fake White House news release that was posted to X claiming the Biden administration had granted Israel $8 billion in emergency aid and false reports that St. Porphyrius Orthodox Church in Gaza was destroyed.
Only 8% of 120 posts related to those stories had published community notes, while 26% had unpublished notes from volunteers that had yet to be approved. About two-thirds of the top posts NBC News reviewed had no proposed or published Community Notes on them.
The findings echo what a Community Notes volunteer said was X’s lack of response to efforts to debunk misleading posts.
“All weekend we were furiously vetting, writing, and approving Community Notes on hundreds of posts which were demonstrably fake news,” Kim Picazio, a Community Notes volunteer, wrote on Instagram’s Threads. “It took 2+ days for the backroom to press whatever button to finally make all our warnings publicly viewable. By that time… You know the rest of that sentence.”
And when the Community Notes don’t show up until much later, a ton of nonsense can spread:
A post about the debunked White House news release published by a verified account had nearly 500,000 views and no proposed or appended note Tuesday afternoon.The Community Notes system also showed that a user tried to submit a fact-check Sunday on another post including the same known misinformation but that it had yet to be approved, saying, “Needs more ratings.” The post had accrued 80,000 views since Sunday.
In a search for St. Porphyrius Orthodox Church in Gaza, only five Community Notes had been applied to the top 42 posts echoing the debunked misinformation. Several posts from verified users with no notes repeated the claim and got over 100,000 views, while 13 Community Notes had been proposed on posts of the debunked claims but had not yet been approved for publishing.
During the first 5 days of the conflict, just 438 Community Notes (attached to 309 posts from 223 unique accounts) earned a “HELPFUL” rating and ended up being displayed publicly to users. Although it’s impossible to know what percentage of content about the war this represents, the fact that trending topics related to the conflict have routinely involved hundreds of thousands or even millions of posts suggests that a few hundred posts is just a drop in the bucket. The visible notes were generally attached to popular posts — the 309 posts in question earned a combined total of 2147081 likes, an average of 6948 likes per post. The majority of the posts that earned Community Notes (222 of 309 posts, 71.8%) came from paid X Premium/Twitter Blue subscribers, and the majority of the accounts posting them (147 of 223, 65.9%) are X Premium subscribers, who are potentially earning a share of X’s ad revenue based on the number of times their posts are seen and who therefore have a financial motive to never delete misleading content. (Overall, roughly 7% of posts that received Community Notes were deleted during the period studied, but there’s no reliable way of knowing how many of these posts were related to the Israel/Hamas war.)
Again, I really like the concept of Community Notes. I think it’s a very useful tool — and one example (of many) of trust & safety tools beyond simply “taking down” content. But it needs to be part of a wider strategy, not the only strategy. And, the program can’t be setup with such a huge blindspot for culture war issues.
But, that’s exactly how things currently work, and it’s a shame, in part because I fear it’s going to discourage others from creating their own versions of Community Notes.
As you hopefully know by now, earlier this week we launched our new game, Trust & Safety Tycoon. It’s a free, browser-based game (playable on desktop or mobile, though we recommend desktop for the best experience) that puts you in the shoes of the head of trust and safety at a rapidly growing social media platform, where you must face all the difficult decisions that entails. As we did for our past games Moderator Mayhem and Startup Trail, Mike, myself, and our game design partner Randy Lubin of Leveraged Play sat down after the release to record a podcast episode all about the game, our design process, and the reaction from players so far.
Elon Musk keeps insisting that stopping spam bots is a huge priority. After all, he said he’d either stop them or die trying.
And, apparently one way to try to stop spam is to stop allowing people to report spam. I’m guessing this is just a mistake on a site that has massive errors pretty much every day, but it’s still crazy. Apparently, exTwitter no longer lets you actually report spam. If you try, it automatically misclassifies it as something else, and then you get a notice that because it’s not that something else, it remains online.
Spam problems solved!
I first spotted this over on Bluesky from user Q.H. Stone, who noted that if you now try to report spam on ExTwitter, rather than saying “we received your report for spam,” it instead says (incorrectly) “we received your report for sensitive media.”
Then, one of the three remaining trust & safety hamsters on the wheel looks at the spam, looks at the report, and says “this isn’t sensitive media,” so nothing to do here:
While I hid the user account in this case, I will note that (of course) it is paying for Twitter Blue, Twitter Verified, Verified, X Premium, so perhaps that’s why it’s not getting removed for spam. But, rest assured, all that account is doing is spamming people with crypto spam. But, apparently, it’s okay, because it’s not “sensitive media.”
Today we’re super excited to launch Trust & Safety Tycoon, a video game simulating what it’s like to run a trust & safety team at a fictitious, rapidly scaling social media company called Yapper*. If you’ve ever wanted to see how you’d do as the head of trust & safety, now is your chance.
Over the last few years we’ve spent a lot of time and effort trying to educate people on the concept of “trust & safety” at an internet platform, as it’s something that many people have strong opinions about, but very little direct experience with themselves. It’s why we ran the content moderation case study series for a while. It’s why we’ve written thousands of articles about content moderation and trust & safety here on Techdirt. And it’s why earlier this year, we created the Moderator Mayhem mobile game, to give people a taste of what’s it’s like to be a front line content moderator.
And now we’re launching Trust & Safety Tycoon. This browser-based game (which works on both mobile and desktop browsers) is a different kind of trust & safety simulator from the last game. While that game was about being a front line content moderator, this game is about actually running a trust & safety team for a rapidly scaling social media startup. You have to set policies, deal with various dilemmas, face internal and external pressures, weigh tradeoffs, determine resource allocation and more, all while trying to keep your website from descending into a cesspit of hate, driving away users and advertisers.
As with Moderator Mayhem, our goal with Trust & Safety Tycoon is to help more people better understand the kinds of dilemmas, nuances, and challenges of handling trust & safety these days. Too many people have very strong feelings about how it should work, despite having little experience in how things actually work.
Trust & Safety Tycoon gives people a chance to try out their own ideas on how to manage a trust & safety program at a startup and see how well it works. There is no “right” answer with any of this, but not all decisions will lead to positive outcomes for you, or the company you’re working for. How will you handle concerns from your team, the CEO, or the media? How will you handle a crisis when your team is already overwhelmed? Will you cause an international incident that gets you called to answer questions from Congress?
This game was developed with support from the Hewlett Foundation, and in association with the Atlantic Council’s recent Taskforce for a Trustworthy Future Web, which we spoke about earlier this year when its report on scaling trust came out. If you didn’t want to read that (excellent) 150 page report, think of this game as a somewhat more fun way of thinking through many of the same ideas.
The Director of the Democracy & Tech initiative at the Atlantic Council, Rose Jackson, told us, “As more people tune into questions around online harms, tech company decisionmaking, and how to keep the internet safe, this game is an invaluable resource to help people understand the tradeoffs at play. Anyone who cares about what they see online will benefit from this fun and thought-provoking experience.”
“We’re excited to support this innovative approach to helping people understand the hard trade-offs involved in trust and safety—not just content moderation, but also building tools, running global operations, and finding the diverse group of talented people needed to do all these things” said Eli Sugarman, director of the Hewlett Foundation’s Cyber Initiative. “Mike Masnick and his team, and the group of experts who advised them, have done a great job of creating a fun and engaging game about the challenges involved in keeping people safe online.”
As with our previous games, this game was a collaboration between our Copia Gaming effort and Randy Lubin at Leveraged Play.
* Any similarities to any real social media companies are purely coincidental entirely on purpose.
The trust and safety conversation tends to focus on the huge platforms, and the millions of smaller websites (some still quite big!) get ignored. But those websites have trust and safety needs too, and they use a lot of different tools to meet them. Most of these tools are proprietary, but there’s a growing push to build more open source tooling for the purpose, which was discussed by Derek Slater in a recent Atlantic Council report. This week, Derek joins us on the podcast to talk about the problems that open source trust and safety tools can solve.
I continue to be fascinated in watching how the various decentralized protocol-based social media systems are evolving — in particular how they’re dealing with the challenges of content moderation. There was an interesting discussion a recently on nostr over whether or not moderation should be best handled by relays or clients*.
ActivityPub has, of course, continued to move forward with its systems of moderation handled at each instance level, combined with the threat of “defederation” being used to keep “bad” instances in line (or cut off from parts of the network). That’s worked surprisingly well in some cases, but is also facing a few challenges, as there have been complaints about some of the largest instances, and now that Meta is planning to release an ActivityPub-compatible offering, there’s a weird push to make some instances promise to defederate from any Meta offering immediately.
But, again, Bluesky may be where the most interesting discussions on decentralized trust & safety and moderation are happening. A few months ago, we wrote about their plans for decentralized composable moderation, and recently they released some thoughts on how you can handle moderation in a public commons.
The goal of Bluesky is to turn social media into a shared public commons. We don’t want to own people’s social graphs or communities. We want to be a tool that helps communities own and govern themselves.
The reason we focus on communities is that for an open commons to work, there needs to be some sort of structure that protects the people who participate. Safety can’t just be left up to each individual to deal with on their own. The burden this puts on people — especially those who are most vulnerable to online abuse and harassment — is too high. It also doesn’t mirror how things work in the real world: we form groups and communities so that we can help each other. The tooling we’re building for moderation tries to take into consideration how social spaces are formed and shaped through communities.
Somewhat importantly, they make it clear that they don’t have all the answers (no one does!), but it’s really interesting to see them discussing this openly, and publicly, and asking for thoughts and feedback as they move forward. To me, the thing that stands out is that the ideas that are presented obviously involved a lot of thought (to the point that I haven’t fully wrapped my head around some of the different proposals, some of which seem clever, while others may need a bit more baking before they fully make sense).
Historically, trust & moderation decisions come in two forms: formed on high in a centralized system in which little is discussed publicly, and people are left trying to sort through what’s actually happening, or in an entirely distributed manner in which things often spring up ad hoc out of need (see: Usenet killfiles), which often run into problems later on.
The Bluesky folks are trying to think about something that is a more hybrid approach, in which the system itself is design to enable communities to better manage things, not just one giant opaque centralized control bunker, and not putting all the weight on users which is unfair to many (especially the targets of abuse and harassment).
I think this kind of vision seems exactly the right one for an organization like Bluesky to have:
A company is an efficient structure for building out a cohesive vision of how things should work, but locking users into our systems would be antithetical to our mission. An open commons can’t be governed at the sole discretion of one global company. We offer services like professional moderators so that we can help protect people and provide a good experience, but we shouldn’t exert total control over everyone’s experience, for all time, with no alternative. Users should be able to walk away from us without walking away from their social lives.
The reason we’re building in decentralization is because we observed that business interests and the open web have a habit of coming into conflict. Third-party developers often get locked out. Moderation policies come into conflict with the diverse interests and needs of different groups of users. Ads push towards algorithms that optimize for engagement. It’s a systemic problem that keeps playing out as centralized social media companies rise and fall.
On Bluesky itself, the lead developer, Paul Frazee noted that they view the future company as a potential adversary, and are designing accordingly. That, alone, is a fascinating perspective to have on things, and one that certainly makes sense in the age of enshittification. And, unlike the way many companies that start on the open web, and later come into conflict with it, as they seek to pull up the ladder behind them to protect a moat, Bluesky is trying to design its systems in a way that protects the system from their own future attempts at enshittification:
Even when things are working correctly on social platforms, there are weird dynamics caused by people’s relationships being mediated by a single company. The Internet is pretty obviously real life in the sense that its management has real-world consequences. When these places control our identities and our ability to connect and to make money, having no way out from the founding company is a precarious situation. The power difference is daunting.
The goal of Bluesky is to rebuild social networking so that there’s not a lock-in to the founding company, which is us. We can try to provide a cohesive, enjoyable experience, but there’s always an exit. Users can move their accounts to other providers. Developers can run their own connected infrastructure. Creators can keep access to their audiences. We hope this helps break the cycle of social media companies coming into conflict with the open web.
Now, some users point to the complex onboarding of Mastodon, or the “WTF how does any of this work?” nature of nostr, and worry that any decentralized/federated system has to be confusing. And that user unfriendliness, in some weird way, acts as a moderation tool in its own right, by keeping communities somewhat smaller. But it also keeps communities… smaller. So Bluesky has a different vision. A surprisingly refreshing and honest one:
A great experience should be simple to use. It shouldn’t be overly complex, and there should be sensible defaults and well-run entry points. If things are going well, the average user shouldn’t have to notice what parts are decentralized, or how many layers have come together to determine what they see. However, if conflict arises, there should be easy levers for individuals and communities to pull so that they can reconfigure their experience.
A great experience should recognize that toxicity is not driven only by bad actors. Good intentions can create runaway social behaviors that then create needless conflict. The network should include ways to downregulate behaviors – not just amplify them.
A great experience should respect the burden that community management can place on people. Someone who sets out to help protect others can quickly find themselves responsible for a number of difficult choices. The tooling that’s provided should take into account ways to help avoid burnout.
A great experience should find a balance between creating friendly spaces and over-policing each other. The impulse to protect can sometimes degrade into nitpicking. We should drive towards norms that feel natural and easy to observe.
A great experience should reflect the diversity of views within the network. Decisions that are subjective should be configurable. Moderation should not force the network into a monoculture.
Finally, a great experience should remember that social networking can be pleasant one day and harsh the next. There should be ways to react to sudden events or shifts in your mood. Sometimes you need a way to be online but not be 100% available.
There is no perfect content moderation solution out there. There is no whiz bang simple technical solution to the messiness that is human beings. As I’ve said many times, so many trust & safety dilemmas are really societal problems that we think are new or need to be solved by internet companies because they’re appearing through screens over the internet.
And, of course, nothing that Bluesky is working on may turn out to work, or matter. It’s still a small operation, and some of these ideas are completely untested. But, at the very least, it is presenting some pretty thoughtful ideas in an open way, and trying to think through the real consequences of what it’s creating here. And that, alone, is incredibly refreshing.
* The creator of nostr apparently does not believe moderation should happen at the client level, but when I asked him how relay operators could express their moderation rules suggested it didn’t matter since relays weren’t moderating anyway. Of course, since then I’ve noticed that nostr is being overrun with cryptocurrency spam, so at some point people there are going to realize that something needs to be done.
We talk a lot about the concept of “trust & safety” at internet companies, but the entire concept is relatively new, and still very confusing to many, including some who work in the field!
Over the last few years we’ve seen a growing movement to organize and somewhat “professionalize” the space, with very important and useful new organizations like the Trust & Safety Professional Association (TSPA) and the Digital Trust & Safety Partnership, both of which are filling in important gaps and helping to make “trust & safety” a much clearer concept and profession.
But, still, there is plenty of confusion, especially for those with no experience at all, who (extremely incorrectly) think “trust & safety” is a euphemism for “censors” or something similar. In fact, the trust & safety role springs naturally from people trying to do exactly what the name says: make sure a web service is trustworthy and safe for users. Usually, this starts with dealing with spam, but quickly picks up on other areas of problematic content around abuse, harassment, child sexual abuse material, and copyright infringement.
In part because this space is somewhat new and only just forming as a concept, it was exciting to be a part of a task force, organized by the Atlantic Council, over the last few months to explore what it means to create a trustworthy future web. The task force brought together a wide variety of experts studying all different aspects of the trust & safety world, from industry, academia, the media, civil society and more.
On Wednesday, the task force released the output of the effort, a thorough paper: Scaling Trust on the Web. The full document clocks in at 150 pages, but the main report is really just the first 50 or so pages, with six annexes included that explore different specific areas, including things like building better tooling for trust and safety, exploring the gaming ecosystem in particular, and looking at trust & safety in a federated world (we just had a podcast on this topic).
There’s also an annex on “respecting children as rights holders,” which does an important job of flipping the terrible current narrative that seems to ignore that children have rights themselves regarding the use of the internet, while various politicians, often egged on by the media, keep pushing to take away the rights of kids to make use of the internet.
There’s a lot in the report, but I think it’s an important step forward for those who are actually looking to understand not just the concept of “trust & safety,” but literally what it means to build a “trustworthy” web that enables so much of the good things that made all of us excited to use the web in the first place. And given just how many hysterical, hyperbolic reports we’ve seen lately about the web being horrible or whatever, it’s nice to have a careful, thoughtful, forward-looking report that is grounded in reality.
A big shoutout and kudos goes to Kat Duffy at the Atlantic Council who wrangled this project, bringing together dozens of very opinionated experts, sometimes with very different views, and pulling together such a massive and detailed report. And, similarly, kudos to Eli Sugarman who similarly helped pull all this together and made it possible via the Hewlett Foundation and Schmidt Futures.
For what it’s worth, my role in all of this was to attend a few meetings where I expressed a bunch of thoughts, as well as reviewing some early drafts and providing a few comments as this went through the sausage making process. I was told from the beginning that I was never expected to sign off on the end product or even agree with anything in it, which was key to me participating in the first place (especially as I’ve seen a few other attempts at something somewhat similar go completely off the rails). But I’m quite happy with how this turned out and think this will be an incredibly useful document going forward in continuing to make the kind of web that we all should want.
I hope that people who are legitimately interested in how the web should operate will read through the report to get a better understanding of the opportunities and challenges ahead.
Finally, I will note that I’ve been diligently working on another project related to this release, but it’s not quite ready for the light of day just yet, so stay tuned…
Last week, when Elon Musk hosted conspiracy theory nonsense peddler RFK Jr. on a Twitter Spaces, he admitted that, despite firing somewhere around 85% of Twitter staff at the point he took the company over and just no longer paying rent or many other bills, he’s still struggling to get the company to break even. This is kind of incredible, given that pre-Musk, Twitter was profitable in 16 of the previous 20 quarters. Obviously, the fact that advertisers have abandoned the platform (mostly because of Elon Musk himself) hasn’t helped.
But, really, the most incredible thing is that he can’t make the site profitable even when he’s not paying the bills.
The latest bill that he’s stopped paying is, according to Platformer, the Google Cloud bill, which hosts a wide variety of important internal Twitter tools, including some of their trust & safety tools. The report notes that, as with basically all of Twitter’s contracts, Elon has been trying to renegotiate them downward, but it had at least kept paying the Google bill, in part because Google was one of the companies that had stuck around and continued advertising on Twitter:
Twitter has been trying to renegotiate its contract with Google since at least March, the Information reported that month. It had also delayed payments to Amazon Web Services, leading the company to threaten withholding advertising payments.
At the time, Twitter decided to pay its Google Cloud bills in light of the fact that Google was (as of February) the company’s second-largest advertiser. It also pays to license the full stream of tweets to show in search results.
It’s unclear what changed. But as Twitter continued to push Google to lower its cloud costs, at some point it stopped paying its cloud invoices — and is now planning to move off the platform altogether.
Not paying Amazon is also notable, though Twitter only started using AWS a couple years ago. But still, these decisions to stop paying bills also means that some of the company’s biggest remaining advertisers may start pulling their ads as well.
Also notable, though is how Twitter strongly relies on Google Cloud for many internal tools. Platformer discusses how a key tool for removing both CSAM and bots (two things that Elon has said were top priorities, though his actions indicate otherwise) are hosted on Google Cloud and there seems to be little to no effort for Twitter to replace them:
Twitter’s core spam detection tools, and the systems that it uses to find violent extremism and media containing gratuitous gore, all run on Google Cloud. So are all of the systems that log data used by the trust and safety team to investigate bad actors. Shutting down GCP could leave the trust and safety team without a mechanism to investigate bad behavior.
Twitter now has three weeks to migrate over that tooling. Anything that is not migrated in time risks being shut down.
And, of course, all this comes out just days after it was revealed that Twitter’s use of PhotoDNA to detect and block known child sexual abuse material was borked, allowing that material to get uploaded to the site.
It really feels like Elon Musk has now turned Twitter into a giant game of Jenga, seeing just how many blocks he can remove without the whole thing tumbling over. So far, the whole thing has remained standing, but that doesn’t mean it will continue to do so.
One of the most important tools for trust and safety efforts is the “block” feature, allowing a user to entirely block someone else from following them. Yes, on Twitter you can get around this by going into incognito mode, but overall, the feature is a very useful tool for those being harassed to limit access to their abusers. Indeed, one of the biggest criticisms early on of the (still in invite-only beta) Bluesky social media app was that it opened its doors to thousands of users before they had implemented a “block” feature (that has since been added). Lots of people argued that launching social media today without the “block” feature is malpractice.
Elon Musk, however, seems to be going in the other direction.
On Wednesday, in reply to someone on Twitter complaining about being blocked, Elon said that “blocking public posts makes no sense” and saying that “it needs to be deprecated in favor of a stronger form of mute.”
This also comes just weeks after Twitter’s adjusted API policy effectively killed one of the most useful 3rd party tools for users on Twitter to avoid harassment: BlockParty.
Block Party’s anti-harassment tools for Twitter will be on indefinite hiatus as of May 31. It’s been a privilege to help you set your boundaries for the last four years. Together, we blocked and muted millions of trolls.
We’re heartbroken that we won’t be able to help protect you from harassers and spammers on the platform, at least for now; we fought very hard to stay, and we’re so sorry that we couldn’t make it happen.
And, of course, all of this comes right after Twitter’s trust & safety boss (who wasn’t particularly experienced with trust & safety work) resigned. So it seems that, yet again, Musk is winging it, and making decisions based on what the worst of his fans want, rather than what actually is best for the ecosystem he manages.
The underlying assumption here from Musk is that the only reason to use “block” is if you don’t want to hear from someone. But that’s wrong. That’s what the mute button is for. Block is an anti-harassment tool to help people avoid having stalkers, abusers, harassers, and the like being able to follow your every word without at least some level of friction.
Considering that Musk himself was so concerned with “doxing” of his public information, you’d hope he’d recognize that, but again Musk seems to view the safety of everyone on Twitter as if it’s identical to his own experience, and his own threat model.
Of course, there’s also the simple fact that the block feature is costly in terms of Twitter compute power:
So, even though he was, himself, an aggressive blocker for a while, more recently he’s urged people to stop using the block feature, and removed everyone from his own block list a few months back. Of course, it appears that even he went back on that promise, because there are reports of him blocking new people since his grand unblocking.
Separately, there’s the fact that there was the infamous “BlockTheBlue” campaign that sought to get Twitter users to block anyone who was subscribed to Twitter Blue, which has really pissed off Musk. Perhaps to the point that this is also a reason he wants to get rid of the block button?
Either way, this is yet another example of a situation where there’s a lot of actual expertise out there, but Musk ignores it all based on (1) his own gut instincts, (2) the requests of the terribly disingenuous people he follows, and (3) a desire to decrease Twitter server costs.
And all while making the website significantly less safe for a large segment of users.
