Twitter Keeps Breaking In Dangerous Ways: Deleted Tweets Reappearing
from the things-keep-breaking dept
A whole bunch of media articles are noting that Twitter users who deleted tweets have noticed in recent weeks that the deleted tweets have magically returned. There seems to be little rhyme or reason for which deleted tweets have returned, but it’s definitely happening to many users. In some cases, people said they had deleted tens of thousands of tweets, only to find them all come back.
Twitter has said nothing, and people are generally guessing what happened. A former Twitter employee says that maybe some servers were moved between data centers, and that they “didn’t properly adjust the topology before reinserting them into the network, leading to stale data becoming revived.”
This is the kind of thing that happens when you kick nearly all of the institutional knowledge that held your newly owned website together out the door.
Anyway, for most people this isn’t that big of a deal, but there is real potential for harm. There are many reasons why people might delete old tweets, and some of them may be to protect themselves. There could be legal reasons to delete a tweet. Or reasons to protect against harassment.
Having such tweets come back to life (without notification) creates a real risk that actual harm could occur. This is the kind of thing that a good engineering team, working with a good trust & safety team would, you know, strive to prevent, in order to keep the users of a platform safe.
But, it’s been made abundantly clear that this is not something that Elon Musk cares about. Putting people in danger is fine, just so long as he continues to be the center of attention.
Filed Under: deleted tweets, elon musk, institutional knowledge, safety, trust & safety
Companies: twitter
Comments on “Twitter Keeps Breaking In Dangerous Ways: Deleted Tweets Reappearing”
The Musk-Twitter saga is the gift that keeps on giving.
The first two weeks were pretty insane, and the pace has slowed down, but we still get a few gems now and then.
With the DeSantis announcement debacle and the zombie tweets, this week is fruitful. I’m happy I never was on Twitter to begin with. Condolences to those users who have to fight a horde of zombie tweets now.
I wonder if this can be a legal liability for Musk?
Re:
I am pretty sure Musk’s biggest legal liability (by at least 2 orders of magnitude) is .. Musk. Everything else probably pales
Re: Re:
Yes, but that’s what keeps breaking things. Back when he fired basically everyone trying to tell him how the site actually worked, his fanbase kept trying to say that everything would be OK because the site didn’t immediately die.
But, that’s not how these things work. It’s like a plane – if the pilots jump with the only parachutes, the plane isn’t going to immediately crash. But, it has to at some point. Even if you have an Airplane scenario where someone else lands it safely, there will be questions.
I suspect this is what’s starting to happen here. Without the institutional knowledge and people who kept things afloat, and fended off legal and service questions, cracks appear. Nobody knows how to fix them, so they get bigger. Eventually, they have to be dealt with.
In theory, most of the problems faced right now can be traced back to Musk’s personal decisions, so there will be trouble. At least, if the US still dares to hold a billionaire accountable, but there’s still major problems even if that’s ignored.
Re: Re: Re:
The problem with holding Musk iliable is that if he’s sued and taken the court, he’ll just no-show. The only way the government is going to hold him liable is if they physically go cuff him and force him to do what they want him to do. Mind, they could do that, but I’m not holding my breath.
Re: Re: Re:2
You don’t avoid liability with a no-show.
If anything, this might make things worse. If you don’t present even a minimal defense, you’re very likely to get a ruling by default that grants everything the opposite party asks, particularly in civil cases.
(Note that sending a team of lawyers in your stead doesn’t count as a no-show.)
After this, you are supposed to pay what you legally owe or comply with court orders. And yes, that can mean a physical arrest if you don’t. It’s rarely done with powerful people… but it’s not completely unheard of.
Re: Re: Re:3
Ask Alex Jones about that. Although the consequences are very slow in coming because of bankruptcy proceedings and the fact that these cases are so insane, Jones had every chance to fight his side in court. But, he chose to make a mockery of the court, refuse to provide the documents requested (despite his claims that he did), switch lawyers every few minutes, and generally refuse to comply. So, he was found guilty by default, which is why he wasn’t able to present “I’m a fighter for free speech” in court against his victims, he’d already been found guilty and the trial was just to work out damages. Which he did so poorly and so clearly in contempt of the court that he’s got a $1.5 billion bill.
It’s depressing that justice is so slowly served, but in the long run he definitely didn’t get away with not showing up, not presenting documents and generally not complying.
Re: Re: Re:2
Due process is a thing, which is good if frustrating with people like Musk and Trump.
There’s also the fact that he seems to be a true believer. You can get him on some kind of charge if you can find a conspiracy to tank the company for profit, but if the evidence says “yes, he really is this immature and stupid”, those things aren’t actually illegal even if wealth hides him from the real world consequences.
Re: Re: Re:
Right. However, twitter is now Musk’s toy, without a bunch of pesky shareholders wanting dividends. So he can do pretty much what he wants with it.
There are some potential problems if he fails to pay creditors such as hosting companies and landlords. Generally these will be between twitter and the offended vendors, however, and any remedies had against twitter will affect the owner and not a bunch of shareholders.
Twitter users have very little real claim if the service is disrupted or if it fails entirely. Remember that the users are the product, not the customers.
That’s not a thing that happens with proper software and database design. The Internet Relay Chat people did a reasonable job of fixing that in 1993, by timestamping everything. Before that code was widespread, it was common to see things get “weird” if the network split apart and then rejoined after things had changed on one side.
The claim of “danger”, though, seems absurd. There may be “legal reasons” to delete a tweet, but it’s doubtful a court will expect someone to watch for a surprise resurrection or hold them responsible for Twitter’s mistakes. Protecting against harassment? I’m not sure what that means; maybe an explanation is in order. Deletion would, anyway, be a half-assed measure, what with things like the Internet Archive, retweets, and screenshots. It’s a best-effort thing (with Twitter’s recent efforts being unimpressive), not something people can rely on for life safety.
Re:
I’m assuming you haven’t been near a software stack for a system that handles massive amounts of data that are spread all over the globe. You can’t just plug’n’play the servers and expect everything to work without a hitch.
Re: Re:
“I’m assuming you haven’t been near a software stack for a system that handles massive amounts of data that are spread all over the globe”
To be fair, neither had Musk. That’s why he asked for printouts of code like it was the 90s.
“You can’t just plug’n’play the servers and expect everything to work without a hitch.”
Without knowledge of the architecture, you can’t even know if something will die immediately, or whether you just unplugged something that was there to mitigate a known issue that happens every 6 months, or recording data for something that’s needed for regular reports to some obscure agency.
There’s something to be said for the “chaos” principle adopted by Netflix and many other services within production, but you still have to understand the architecture – which Musk immediately announced he didn’t and fired the people trying to correct him.
Re: Re:
Without a hitch? No. But if the designers and implementers of a system believe it to be potentially dangerous, it’s their responsibility to make it fail-safe. The same goes if they think a failure will pose significant legal risk (granted, Twitter pre-dates GDPR). You don’t just stick a footnote in some document to tell people that things will go horribly wrong if they make some minor and easily predictable error. The potential reappearance of deleted data should be obvious to anyone designing a distributed database; if deletion is considered a desired feature, they need to design around that.
In other words, this is probably a result of design failures that were present long before Musk got involved. But now the people who’d been papering over those problems are gone.
Re: Re: Re:
Any solution that has been built over a decade and a half will have flaws and those who have been responsible for running it probably know of those flaws and how to mitigate them. In this instance, allegedly moving servers around without knowing all the little details are bound to produce problems.
TL;DR: Institutional knowledge is damn hard to replace, no matter how smart someone think they are or how smart the system was designed.
Re: Re: Re:2
The loss of institutional knowledge over time is precisely what the system needs to be designed for. It’s an obvious risk that “smart” people will predict.
That’s one reason why programmers write important comments directly in the source code, for example, rather than referring to external documentation or assuming later developers will have found it on their own. The storage of “documents” is often managed differently than “source code”, and the documents may go missing or be difficult to find in the future. And the future programmers will, upon finding that, decide they’re smart enough to reverse-engineer the code and skip the bureaucracy of a documentation search.
Re: Re: Re:3
So tell me, if someone fires a majority of the people having the institutional knowledge, can that be considered “loss of institutional knowledge over time”?
Smart people may try to make things fool-proof but they can never proof it against a CEO that’s a fool.
You’d be surprised how you run and care for systems today isn’t about source code and its documentation but configuration and how systems are abstracted in multiple layers and topologies which makes “reverse engineering code” a futile exercise in understanding how a system actually slots together and work.
Re: Re: Re:3
A lot of institutional knowledge id about interaction between in house and bought in systems, and also tool that have been written to solve issues. All it it may be documents, but without a good librarian it can be difficult to find. All the documentation may be on a computer system, but unless organized and indexed,
properly it is of very limited use.
Witch companies employ a librarians, or better yet at least two, (in case of accidents and illness), to manage institutional knowledge?
Re: Re: Re:3
“The loss of institutional knowledge over time is precisely what the system needs to be designed for.”
Institutional knowledge usually means that there’s a bunch of shortcuts, hacky solutions and technical debt built up over the time the company’s been operational, and usually go past any initial design.
It’s not secrets that people built into some grand design, it’s usually some undocumented behaviour that when people see it happen they go “yeah, ask Dave he’ll fix it in 30 secs”. The problems come when Dave and the people who knew that Dave knew how to fix it leave, like in a mass layoff situation. Then, you’re left with a poor guy who can sort of see what the problem is, doesn’t understand the known fix or the decisions that led to the architecture he’s looking at, but he’s the one who needs to fix it.
“That’s one reason why programmers write important comments directly in the source code”
That’s cool. Also, not something that’s guaranteed, and also not something that’s relevant when the problem is that there was an extra node added somewhere to counter for a known issue with infrastructure, but the new boss killed that node after firing everyone because he didn’t understand its purpose. Oh, and documentation is light because the old management demanded other work took priority, but it was OK because Dave was there.
Re: Re: Re:
“make it fail-safe.”
AHHHahahahahaha – you funny
“probably a result of design failures that were present long before Musk got involved”
Probably is a word used when one has little to no data or analysis in support of their opinion.
Re:
“I’m not sure what that means”
You should probably read up on the subject more before commenting, then. An example that comes to mind are the EU’s “right to be forgotten” laws, which would also affect archives on the net. I’m no fan of that law, but if Twitter were ordered to take something down, they did and then it goes back up, they’ll be in trouble.
Re: Re:
It would affect only archives subject to EU jurisdiction, and only the ones that specifically received a valid “right to be forgotten” request. I’m sure there are “dark” archives that aren’t known to the public. It would be foolish to think one could post something to Twitter and make it totally disappear, even if Twitter were competent. I believe Techdirt, for example, has refused requests from EU people to make unfavorable stories disappear.
Re: Re: Re:
THat Techdirt evaluates RTBF requests and chooses to reject some after evaluating their legal value and accepts the potential liability that entails has no effect on the liability Twitter incurs if a user deletes a tweet due to a RTBF request, or in response to a court order, or as part of a legal settlement or contract dispute and than Twitter restores those tweets.
This raises both direct liability (twitter having restored tweets it removed in response to legal process) and indirect liability (Twitter’s actions creating legal liability for users)
Even if reasonable (its not), your comparison focuses on an example, one posisble avenue of liability, and ignores the wider point on liability.
Re: Re: Re:2
I think everyone’s generally agreed that Musk is gonna lose a bunch of money from Twitter, one way or another. Even Musk tried to back out of it. The point was not about liability per se, it’s that one cannot reasonably expect Twitter deletion to work well. If you’re a nobody, maybe it’s good enough, but if your tweet attracts attention, it’s likely to be around forever, somewhere.
Re: Re: Re:3
“I think everyone’s generally agreed that Musk is gonna lose a bunch of money from Twitter, one way or another”
He already has, both with Twitter and his other stocks. The reason he’s nominated a new CEO might just be because he needs a scapegoat when it collapses under the weight of his decisions so that he can say “I was working on Tesla at the time”. It’s unlikely to make a profit now, since he’s burdened the company with more debt than it was making in its more profitable times (and despite some claims, it was making a profit for some of the years before he paid for it).
“The point was not about liability per se, it’s that one cannot reasonably expect Twitter deletion to work well”
Or anything of note on the internet, which is why the EU law has always been somewhat silly (though, understandable within normal EU rules if you don’t understand the internet)
Luckily that's not the case
It’s a good thing no-one ever deletes a tweet because of things like ‘court orders’, ‘TOS violations’, ‘I just realized that having pro-[insert non-heterosexual label here] posts is likely to go really poorly with the bigoted family I’m still living with’ or ‘oh shit I’ve got a stalker I need to delete anything that might provide hints to where I live fast’ otherwise this sort of glitch might have some seriously negative repercussions for users.
That's not how it works
Years ago, I attended a talk by a Twitter Engineer who discussed how they sharded their data set into different databases. I remember being blown away when he mapped out how a follow worked, with your account linked to the one you’re following (but not vice-versa). In order to scale the graph, they found the easiest thing was to shard across many, many MySQL DBs. A relational DB would never have been able to keep up with the number of writes-per-second (something like 15k IIRC).
Obviously, they’ve tweaked and expanded their infrastructure since. You can read what they had—at least in 2017—in a nice blog post by the VP of Infrastructure and Ops Mazdak Hashemi (no doubt long gone 🫡).
Simple solution is simply wrong.
This issue is what you’d imagine would happen if Mount Stupid Musk wanted to shut down a “extra” data center, so he—not knowing anything about anything—told his goons to pull the servers from one and plug them in at the other. How hard could it be, right?
Re:
Why? If there were a timestamped deletion record, the topology of the network wouldn’t matter. Even if the deleted comment’s text somehow reappeared in the database, the deletion time would be newer than it, so the system would know to re-delete it.
Re: Re:
It’s not as simple as that when we are talking about what is essentially a distributed system. What if moving a server and updating it’s configuration means the shard it’s running on think it’s the one holding the most current data and starts replicating that data to other shards?
You should really read up on the complexities of keeping distributed data in sync, it’s not as simple as checking a timestamp.
Re: Re: Re:
I’ve implemented such a system, though not at this scale. That’s why I think it’s obvious: I’ve seen how data can reappear in naive or unfinished implementations, and added code to inhibit it.
Why would a server “think” it has the most current data? The data would have a timestamp, and its deletion record would have a timestamp (both of which might be Lamport timestamps). It’d be nothing to do with when it was last moved or configured. I suppose things could get tricky if all the servers which knew of the deletion had gone offline before the old one, which didn’t know but still had the message, came back online. But assuming there was at least one up-to-date server online, the connecting servers should quickly learn enough about the sharding to know the servers in control of certain data are unavailable, and nobody should share such data till they’re back.
Re: Re: Re:2
And that’s why you don’t understand the problems that arises at that scale. Data is segmented and handled on different shards/nodes, there isn’t a simple way to compare things between shards/nodes determining that a specific entry has a different state elsewhere due to someone doing something wrong. Ever used Apache ZooKeeper for example and the problems you can cause by introducing faulty configuration for one node which breaks the consensus?
If it were as simple as you believe it is there wouldn’t be any companies specializing in “big data” solutions.
Uncensorship
Hey, this is just Musk living up to his free speech and no censorship ideals.
Or....
Or….
Someone fat fingered and update to a sql data base and accidentally turned a bunch of “Deleted” flags from 1 to 0.
Always execute a select with your where clause before executing and update or delete.
“Where TwitterID like ‘%Wizard%'”
Does Twitter have an undelete post function? I’m wondering why they need to keep deleted posts. Even if you need to keep a record that the post happened, you could save space by deleting the contents.
This comment has been flagged by the community. Click here to show it.
Shown The Door
This is the kind of thing that happens when you kick nearly all of the institutional corruption that held your newly owned website hostage out the door.
Re:
OK… you’re making less sense than normal.
I mean, I think I can piece together that you think that the guys who tried to explain to Musk how wrong he was about how the site actually operated on a system level were somehow involved in your imagined conspiracy against your white supremacist friends. But, even if you buy that, you’d still need their knowledge to change the filters without breaking things.
It also doesn’t explain everyone else fired. The ad buyers, human CSAM moderators, accountants, operations techs, etc. wouldn’t have had an effect one way or another on your pet peeve.
Re: Re:
No you misunderstood. He’s saying it’s supposed to break this way and those no-good Old Twitter engineers kept thwarting that but Musk took care of them!
Re: Re: Re:
The fun new storyline is the way the Twitter Spaces event with Ron DeSantis announcing his presidential run failed. The service which is advertised as scaling to millions failed at around 300k then returned at a max of around 150k. AOC’s gaming session on Twitch had several times that.
Anyway, the fun part is that it seems that it depended originally on third party licences. Said third party (Redis) stopped supplying Twitter because of an unpaid bill. It seems unclear whether this was deliberate (Mush said to stop paying the bill) or inadvertent (Musk fired the department responsible for keeping software licences up to date), but either way it seems the people were not as disposable as he thought.
Re:
Care to expand on that?
Re:
Held it hostage for what, Koby?
Re: Run away little coward before I make another song about you
Elmo’s never going to know your name, much less let you suck it.
This comment has been flagged by the community. Click here to show it.
MM is plumbing the depths of how deranged a blogger can sound while hyperbolically criticizing Twitter and Musk.
Mike, please provide specific examples of people “put…in danger” through their use of Twitter who were too dumb to simply log-off.
Re:
You just staked out the position that for example stalkers do not harm or murder their victims (and/or people in their vicinity). That is an exceptional claim to make so you bring the evidence. Then again that is exactly why you fob it of to others since you know you cannot prove a negative. And no you will not get a No True Scotsman or Excluded Middle exception here. It is an additional tool, not the only tool.
Reappearing Tweets are an additional tool in a stalkers arsenal to find someone who started hiding to not be found by said stalker. A disgruntled stalker since their victim dared to hide from the only person they (according to the stalker) should never hide from. When you sift those reports about stalker violence they always do more harm after their victim dared to hide and they find them again.
Then there is this “How dare you to be different”-shtick that the Republicans have going (something that Musk is a very enthusiastic supporter of). Get exposed while living in a small town/village/whatever in MAGA country is risking a beating and it is almost guaranteed that the local law enforcement will tell the victim that they are the reason that the beating happened so they either shut up or get booked for deliberate provocation.
Re:
How stupid are you? Even Musk feels Twitter can be dangerous since he went after the kid publishing his jet’s location in near real time.
This comment has been flagged by the community. Click here to show it.
MM is plumbing the depths of how der@nged a blogger can sound while hyperbolically criticizing Twitter and Musk.
MM, please provide specific examples of people “put…in danger” through their use of Twitter who were too foolish to simply log-off.
This comment has been flagged by the community. Click here to show it.
Re:
the dumb or foolish – in either case, that’s who MM is advocating for here!
Re:
It’s one thing to say “don’t feed the troll”.
It’s another to basically ignore that harassment, stalking, SWATting, assault and battery, and murder will happen and “logging off” isn’t going to prevent those from happening IRL.
Or that online speech follows what happens in the real world.
Re:
“der@nged”
lol what is this tiktok puritan bullshit
I just read that Musk’s Neuralink company has been granted authorization to start human trials. The way Twitter is going doesn’t really inspire confidence, but who knows, maybe the test subjects will get their implants and suddenly start remembering things they’d forgotten.
“Putting people in danger is fine, just so long as he continues to be the center of attention.”
Who just got a state law saying he’s not responsible if his rocket blows up and kills the crew??