Techdirt is off for the holiday! We'll be back with our regular posts tomorrow.
Moderating Eating Disorder Content Is Harder Than You Think
from the content-moderation-at-scale dept
Both troubled teens and government agencies are asking, “How thin is thin enough?” The teens are thinking about how thin they want to look, while the government is thinking about what’s too thin to post online.
The refrain is always the same: the platforms need to do more—never mind the difficult details. Platforms need to remove posts that encourage eating disorders—but leave space for people to talk about recovery. They need to increase moderation. But they’re not supposed to use AI, which is fallible, easily gamed, and potentially illegal. And they’re not supposed to use more human moderators, who will be traumatized by their experiences.
Content moderation is impossible to do well at scale. Moderation around eating disorder content is especially fraught. And there’s no way to eliminate ED content online without broadly shutting down entire parts of the Internet.
Newcomers to this topic might think it’s easy to tell whether a particular post qualifies as eating disorder content. Is there a picture of a starving girl or not? We think we know it when we see it. And this is true sometimes: plenty of ED content openly identifies as such, usually in order to cordon itself off from the rest of the Internet.
An account bio might read, “edtwt, minors dni”—shorthand for “eating disorder Twitter, minors do not interact.” The account might also include that they’re pro-recovery, post or do not post fatphobia, their favorite K-pop group, whether they go to the gym or not, what kind of eating disorder they have, and what kind of fashion they like.
Behind these accounts are individuals who are complex, imperfect, and hurting. And this content has a context. A post that’s so obviously eating disorder content on this account may not obviously eating disorder content posted elsewhere. To be “pro-recovery” is to want to (or be open to) recover from your eating disorder. The person behind an account may be in therapy or receiving outpatient treatment. Both accounts dedicated to eating disorders and accounts that do not solely focus on eating disorders but still have individuals struggling with them may discuss their experiences with the disorder and recovery. Platforms cannot simply ban discussions about eating disorders without sweeping away plenty of honest conversations about mental health, which might also drive the conversation towards less helpful corners of the internet.
And the platforms probably can’t ban fatphobia, at least not in a way that would take care of accounts that use fatphobia to encourage disordered behavior. If an ED account posts a video of a person with a fatphobic caption, platforms can delete the ED account, but the platform shouldn’t delete the original video just because it could be harmful in the wrong hands. The same goes for pictures of people who happen to be skinny: the distinction between lean and clearly anorexic is not as clear-cut as you might like to imagine. We could have everyone register their BMIs and daily caloric intake with their username and password, to be sure that no unhealthy daily outfit pictures slip past the mods, but short of that appalling dystopia, we’re out of luck.
This is all a bit hypothetical and ridiculous, because after all, platforms currently host plenty of self-identified ED content. The platforms could start with banning that.
And once platforms start banning openly disordered content, ED accounts would quickly stop being open about it. This isn’t speculative. These accounts have all sorts of alternative names and phrases invented specifically to evade content moderation. They can invent ED dog whistles much faster than platforms can ban them. Or these ED accounts can create a new account that doesn’t post eating disorder content at all—only borderline content: Kate Moss, particularly thin K-pop idols, girls who might have an eating disorder but might just be really into yoga. Sometimes you know it when you see it, but sometimes you don’t. When is posting a picture a pack of gum ED content and when is it just posting a picture of a pack of gum?
Proposals and lawsuits that aim to make platforms liable for the harmful eating disorder content are asking for the impossible. The best outcome they could hope for is that platforms crack down on the explicitly rule-breaking content and that ED content gets a bit more covert. Because it’s not going away. Companies have been removing ED content since 2001. Eating disorders are older than social media, and advocates who think platforms can moderate ED content out of existence understand neither eating disorders nor content moderation.
No one is saying platforms should do nothing. They should ban accounts and remove content that violates platform rules. And platforms should further develop their current tools. They should prompt users to reach out to helplines or to take breaks. They should allow users to widely mute certain topics and ads. But platforms will never be able to remove all eating disorder content. Nor will they be able to remove all users with eating disorders. Nor should they, even if they could.
Social media cannot solve eating disorders. They can and should aim to host better conversations in general, but we should stop expecting them to moderate mental health problems away.
Santana Boulton is a legal fellow with TechFreedom
Filed Under: content moderation, eating disorders
Comments on “Moderating Eating Disorder Content Is Harder Than You Think”
This comment has been flagged by the community. Click here to show it.
This may not be the right place to bring up my new diet- The Gaza Diet. You don’t have time to eat because you spend all day and night GETTING BOMBED.
Re:
Dude, TPC. Don’t bring up some other controversial subject that has no relevance to the current content, and frankly, that is a terrible and insensitive joke.
Re:
Username checks out.
Re: Re:
I see no evidence that this poster is a mighty hunter.
Matched against what standard? You cannot even get scientific consensus of what is a proper BMI and daily caloric intake.
Also, what is the lookback period? If I wasn’t feeling well and ate less then normal last week, am I banned until I feel better and back to normal eating? Do I need to weigh-in and run the BMI calc each time I want to login?
Re:
At least we don’t have calories locked to particular vendors. The HP diet would purely suck.
Re:
One of the many problems with using BMI as the be-all and end-all indicator for obesity etc is that a person can have a higher BMI than normal because they have built up a lot of muscles, this is something body-builders have pointed out for a very long time now.
A better indicator is body fat percentage, but that requires actual measurement.
Re: Re:
There is no relatively quick and easy number which can be arrived at where a naturally lean but healthy person is not on the unsafe side and someone with at least a marginal ed issue is on the technically safe side
Re: Re:
It’s a good point, and also relates to the idea that what could be an absurdly unhealthy diet for most people can be, in fact, well-tailored nutritionally for others. For example, see the diet of Dwayne “The Rock” Johnson: seven large meals a day, totaling around 6000 kilocalories (about 2 to 3 times what an average adult would need). It seems crazy, certain to lead to morbid obesity, till one sees the corresponding exercise plan on the same page: 3 to 4 strenuous hours a day, 6 days a week.
So, would it be responsible to describe such a diet, or post pictures of it, on social media? Or the workout routine? Following either of those could quickly send a “normal” person to hospital.
Re: Re:
As a former overweight-by-bmi powerlifter, this does not account for the majority of our overweight-to-obese population. It takes a lot of hard work to reach that point.
The military long ago figured out how to work around this. BMI is perfectly sufficient for population level. You can fall back to a tape test for the few that fail BMI due to muscle.
Re: Re:
Relatively few people with overweight BMI due to muscle are mistaken for being fat. And those that are likely are better equipped to discuss their health/exercise routines with a doctor than the average. Of bigger concern is that the math is normalized on a specific average height decades ago. The math doesn’t just break down at low body fat, it breaks down at achievable heights on both ends of the scale. The big problem with BMI is the big problem with Body fat measurements. They are largely meaningless for accurately assessing health and diagnosing issues.
A spot check of BMI and body fat tells doctors very little about your health. A trend line can in some cases aid diagnosis but can not identify specific issues (given most Americans see their doctor maybe 1/year, few have accurate trend lines) But most often, medical professionals at all levels have leaned on an assumption that BMI or Body fat percentage indicates current health. This has lead to national news coverage of serious health issues being diagnosed as simply being fat.
Obesity is a cause of Diabetes type 2, and excess joint strain. Thats it. Everything else is a symptom of a sedentary lifestyle or bad diet. Obesity and this issues it causes are mitigated by exercise and diet, and so yes, a treatment for obesity can mitigate the risk of major health issues associated with being sedentary or bad diet. For decades, addressing obesity was considered sufficient to address the underlying health issues.
But, and this is super important, a healthy diet and healthy exercise will not necessarily result in weight loss in any reasonable time frame for an obese person. Noticeable weight loss can take years, and in some cases will require more drastic dieting and other weight loss interventions after years of metabolic damage.
Even if simply commenting that the weight itself is an issue due to joint injury, medical professionals continue a stereotype that the weight is the problem. This leads to dissuading obese people in making the smaller changes in their life that can have excellent health outcomes without the loss of weight. These small steps often being key first steps to sustainable changes in lifestyle. On top of the already discussed issue that while focusing on wieght loss, doctors have been known to ignore patient concerns both increasing the likelyhood of the patient resorting to unhealthy, unsustainable weight loss and the likelyhood serious medical complications go unnoticed or even worse, unreported.
Therefore, In seeking to combat the issue of obesity, the AMA recommends to avoid considering weight as a measure of health, and instead focus on objective measures of metabolic health that can see progress at a much more granular level. Any eagle eye view can note that Obesity is a health issue, but Body fat percentage is only marginally better than BMI at measuring the health problems people are actually facing. All BFP can tell you is to be slightly more accurate at whether or not you have a chance at developing health complications 20 at some time in the future, and other measurements tell you that better.
Re:
Not to mention calories are a shit way of tracking. You know how calories are determined? By setting the material on fire and seeing how much it can heat up water. Rest assured, your body does not process nutrients this way. There are different metabolic pathways that process different macros. 1000 calories of fat != 1000 calories of protein != 1000 calories of carbs.
Too many carbs is the major culprit. They burn quick and clean, and your body is a dynamo and processing them and producing excess energy to be stored as fat. The sugar industry knew this half a century ago when they started gaslighting America into pointing the finger at fats. Bodybuilders and strongmen often do their bulks and cuts by keeping protein intake more or less even, and simply adjusting carb intake. Bro-science literally had this figured out well before the gaslighting campaign even started. And now we have the additional challenge of food deserts being served by McDonald’s and Dollar General.
There actually is quite a bit we would be able to do, but for the money being spent in politics to keep the landscape confusing to the everyman. To keep communities poor and dependent on insufficient food options. The concepts themselves are so simple even the dumbest fucking jock can make sense of it. Greed is the only reason this information can’t be consistently and accurately dispersed.
That’s really stretching the definition of “openly”, isn’t it? I’m probably not looking at “account bios”, and if I am and see a bunch of random-looking letter strings (for some reason written as acronyms despite their dubious pronounceability), I’m probably not going to spend time trying to decode them.
This post seems overly focused on the “thin” side of eating disorders. What about, for example, pictures of doughnuts, cheesecakes, and dinner buffets? The glorification of food and eating, especially in relation to fatty and/or high-calorie foods, could well be correlated with the opposite problem. And isn’t it more common to be unhealthily overweight than underweight?
Re:
And isn’t it more common to be unhealthily overweight than underweight?
wouldnt that depend on your definition of unhealthy?
there are probably more people overweight then underweight.
Re: Re:
Not really my definition. Rather, what’s the relative frequency of doctors saying “you’ll need to gain weight to be healthy” versus “you’ll need to lose weight to be healthy”?
This is not quite the same as just being overweight or underweight. A doctor could also say “you’re a little underweight/overweight, but seem healthy for now.”
I’m guessing that in terms of advice from doctors, “lose weight” is more common. Though as a sibling comment noted, malnutrition could be a more acute condition, so relative frequency is not all we should base our policy on.
Re:
People who starve themsleves, particularly given the methods by which they do, can lead to directly to death in relatively short order in ways that over-eating (and retaining food for digestion) absolutely do not. They psychology of thinness-obsessions also tend to the more directly harmful, and malnutrition doesn’t help with that either.
You’re talking about two (or more) entirely different phenomena.
i honestly wish it had not already been ruined, but “Eating Disorder (content)” could use a better shorthand.
If only we stopped pretending that parents were powerless in these things.
So much responsibility is abdicated to big tech so that parents can focus more on all of the things that don’t really matter but look good on an college application.
We had her in ballet from 5!!!
We never monitored how & what she ate, what she was talking about online, the pressure to maintain the proper figure, what she was doing with all of those laxatives. But by god we should be able to sue big tech for not doing enough to protect my baby!!!
Swap in wrestling & add the extreme ways they cut weight (that aren;t healthy).
The greatest trick the idiots ever pulled was making the world believe that corporations & websites are thinking how to harm kids. The platforms don’t think, they do the same things they do to adults to kids, the algorithm serves up more content & related content that you were looking at.
Sometimes it leads to eating disorders, sometimes thinking Trump is the messiah.
The computer doesn’t just put harmful things in front of people for kicks, it does better when it puts things directly related to & what other users have said is related to the theme.
But then I’m of an age where I was taught not everything on the internet is real, that people can & will lie to you, & that critical thinking was my best defense.
Now we hand 2K phones to kids with no instruction beyond don’t share nudes. (if that) They get to act shocked that the whole world wasn’t required to live up to a higher duty of care than the parents.
Re:
Because children with eating disorders never hide them from their parents, you mean? And if parents were unaware of their homosexual children being encouraged deep into the closet by ‘pro-family’ websites, would you still be so blasé? Ableist ignoramus.
Re: Re:
You are conflating two different things here. You are equating parents being powerless with them being shitty parents so their kids hide things from them.
Because children with eating disorders never hide them from their parents, you mean? And if parents were unaware of their homosexual children being encouraged deep into the closet by ‘pro-family’ websites, would you still be so blasé? Ableist ignoramus.