Oversight Board Overturning Instagram Takedown Of Ayahuasca Post Demonstrates The Impossibility Of Content Moderation

from the this-is-not-easy dept

Congress has been holding lots of “but think of the children online!” hearings over the past couple of months, and one prominent topic that comes up over and over again is the fact that people can find “drug” information online. Fears about kids and drugs goes back decades, but politicians love it, because it always works. And, of course, the media loves to run these overhyped stories. A quick search finds dozens of stories like the following in just the last month or so:

There are many more such stories, but you get the idea. The Instagram stories, in particular, were targeted by the recent Senate hearing which was focused on allowing Senators to grandstand on how bad Instagram supposedly is for kids. Instagram responded to the report by pointing out that it’s doing a ton of stuff to try to block such content:

“We removed 1.8 million pieces of content related to drug sales in the last quarter alone, and due to our improving detection technology, the prevalence of such content is about 0.05 percent of content viewed, or about 5 views per every 10,000. We’ll continue to improve in this area in our ongoing efforts to keep Instagram safe, particularly for our youngest community members.”

But, within a day of that report getting so much attention, you had the Oversight Board step in to demonstrate how much more difficult an issue this is than it sounds to grandstanding politicians or narrative-driving journalists. The Oversight Board overturned Instagram’s decision to take down a post that was promoting ayahuasca as medicine. Instagram took down the post (which had just started trending) because it was promoting it as medical treatment:

In July 2021, an Instagram account for a spiritual school based in Brazil posted a picture of a dark brown liquid in a jar and two bottles, described as ayahuasca in the accompanying text in Portuguese. Ayahuasca is a plant-based brew with psychoactive properties that has deeply-rooted religious and ceremonial uses among Indigenous and other groups in some South American countries, and related communities elsewhere. Ayahuasca contains plants which are sources of dimethyltryptamine (DMT), a substance that is prohibited under Schedule I of the 1971 UN Convention on Psychotropic Substances and the law of many countries, though there are relevant exceptions under international law for substances containing DMT such as ayahuasca, and exceptions under some national laws, including in Brazil, for other uses such as religious and Indigenous use.

The text states that ?AYAHUASCA IS FOR THOSE WHO HAVE THE COURAGE TO FACE THEMSELVES? and includes statements that ayahuasca is for those who want to ?correct themselves,? ?enlighten,? ?overcome fear? and ?break free.? It further states ayahuasca is a ?remedy? and ?can help you? if one has humility and respect. It ends with ?Ayahuasca, Ayahuasca!/Gratitude, Queen of the Forest!?

The content was viewed over 15,500 times and no users reported it. The post was flagged for review by Meta?s automated systems because it had received around 4,000 views and was ?trending.? Meta specified neither the image nor the text triggered automatic review. The post was subsequently reviewed by a human moderator and removed. Meta told the Board that it was removed for violating Facebook?s Community Standard on Regulated Goods, but later stated it removed the content for ?violating the Instagram Community Guidelines, which include a link to the Facebook Community Standard on Regulated Goods.? Meta notified the user that the post went against Instagram?s Community Guidelines, stating ?post removed for sale of illegal or regulated goods.? The messaging also noted that Meta removes ?posts promoting the use of hard drugs.?

So, there are a few interesting tidbits there worth pointing out. Contrary to the fear-mongering and grandstanding, this notes that Instagram has a system to proactively flag this kind of content, and that some of the priority of such reviews is based on whether or not the content is getting a lot of attention and going viral (these are both things that many critics of Facebook/Instagram/Meta insist the company does not do). And, if you just listened to the Senators at hearings, or just read all those news articles above, you might not realize th.

But, the more important thing that this case showcases, is just how difficult it is to make these kinds of calls at scale. What seems obvious to some is not so obvious, and there’s a lot of subjectivity involved. This can be seen in the way the two sides (the poster of the original content and the company) explained their thinking. The organization that made the original post about ayahuasca says that it was just an image of a regulated and legal ceremony:

The user stated in their appeal that they are certain the post does not violate Instagram?s Community Guidelines, as their page is informative and never encouraged or recommended the purchase or sale of any product prohibited by the Community Guidelines. They said that they took the photo at one of their ceremonies, which are regulated and legal. According to the user, the account aims to demystify the sacred ayahuasca drink. They said that there is a great lack of knowledge about ayahuasca. The user stated that it brings spiritual comfort to people and their ceremonies can improve societal wellbeing. They further state that they have posted the same content previously on their account and that post remains online.

Instagram, for its part, takes the more hardline “drugs bad, mmkay” line that Congress and the press seems to want them to take:

Meta stated that the content violated Facebook?s Community Standards because ?the user described ayahuasca with a heart emoji, referred to it as ?medicine,? and stated that it ?can help you.?? Following a question from the Board about whether the content was removed for violating Instagram?s Community Guidelines or Facebook?s Community Standards, Meta responded the content was removed ?for violating the Instagram Community Guidelines, which include a link to the Facebook Community Standards on Regulated Goods.? Specifically, the user violated Instagram?s prohibition on content ?buying or selling illegal or prescription drugs (even if legal in your region).? Meta also cited another line of the Community Guidelines which links to the Community Standard on Regulated Goods, which ?clarifies that Facebook prohibits content that ?[c]oordinates or promotes (by which we mean speaks positively about […] non-medical drugs.??

Referring to Meta?s values, the company stated that ?Safety? displaced ?Voice.? Meta noted that users are permitted to advocate for the legalization of non-medical drugs and to discuss the medical and scientific benefits of non-medical drugs, but that there is no religious or traditional use allowance. Meta argued that this rule strikes the correct balance between ?Voice? and ?Safety.?

Meta also stated that prohibiting this content follows human rights principles. It stated that it considered the right to freedom of expression under Article 19 ICCPR and the right to freedom of religion or belief under Article 18 ICCPR, and argued that its decision satisfied the conditions required to restrict these rights.

According to Meta, Facebook?s Community Standard is easily accessible and its non-public definition of non-medical drug as a ?substance that causes ?a marked change in consciousness?? is apparent. It further argued that the decision sought to protect public health. It stated that dimethyltryptamine (DMT), one of the active hallucinogenic substances in ayahuasca, poses significant safety risks, citing the 1971 UN Convention on Psychotropic Substances.

The final ruling by the Oversight Board overturn’s Instagram’s decision to take it down, citing that Instagram’s own community guidelines were somewhat confusing and focused on the buying and selling of illegal drugs (though it notes that the rules are confusing, because they said “illegal or prescription drugs (even if legal in your region)” which is, as the Board notes “confusing and contradictory.” There was also some level of conflict between Instagram’s rules and Facebook’s rules, and an unclear delineation of which one should apply where.

But where the impossibility of content moderation on topics like this really shines through is in the discussion about “voice” vs. “safety,” which was one of the key points that Instagram made in defending its decision. The Board says it values things differently than the company:

The Board concludes that Meta?s decision to remove the content was not consistent with the company?s values. In this case, as in many, Meta’s values of ?Voice,? ?Safety,? and ?Dignity? point in different directions. Meta’s decision to take down the post weighted ?Safety? over ?Voice.? The Board would balance the values differently, believing that the genuine but not particularly strong interests in ?Safety? are outweighed in this context by the value of ?Voice? and the importance of recognizing the ?Dignity? of those engaging in traditional or religious uses where there is historic evidence of such use, including by Indigenous and religious communities. Scientific research indicates that the use of ayahuasca in a controlled context in traditional and religious ceremonies is not linked to a serious risk of harm. Meta cited the European Court of Human Rights case of Franklin-Beentjes and Ceflu-Luz da Floresta v. The Netherlands (Case No. 28167/07, European Court of Human Rights, May 6, 2014) to demonstrate the risks of ayahuasca use. Other courts, however, have reached different conclusions ? for example, the United States Supreme Court, in considering the risk of harm from ?the circumscribed, sacramental use of hoasca? by members of an ayahuasca based religion, found that the government had not put forward sufficient evidence of harm from religious use, which was its burden, to justify the prohibition in these circumstances. Meta?s rationale does not appear to have taken into account controlled uses of ayahuasca which aim to mitigate health risks. In light of scientific research, Meta?s rationale did not demonstrate the danger of this post to the value of ?Safety? in a manner sufficient to displace ?Voice? and ?Dignity? to the extent to justify removal of the post.

There should be no way to look at that paragraph and not realize just how subjective all of this is. Every one of these involves judgment calls and people weigh different values in very different ways. Even people who work at the same company. Figuring out how to (1) write rules that take all this into account, (2) express those rules in AI moderation systems, (3) express those rules clearly to tens of thousands of human moderators across hundreds of languages and cultures, (4) and expect it to work at a scale of billions of users and millions of posts per day… is literally impossible. It cannot be done well.

That doesn’t mean that everyone should give up, but there should at least be some level of humility in recognizing the level of this task, and highlighting individual decisions that we disagree with is not evidence that people or the company “don’t care” about this issue, but rather evidence of just how impossible a task this truly is.

Filed Under: , , , ,
Companies: facebook, instagram, meta, oversight board

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Oversight Board Overturning Instagram Takedown Of Ayahuasca Post Demonstrates The Impossibility Of Content Moderation”

Subscribe: RSS Leave a comment
17 Comments
Ninja (profile) says:

As far as I know ayahuasca is also widely used in religious rituals. Shamanism if memory serves. Even though it is generally illegal (as in forbidden by law) here generally speaking you are free to produce and use in religious rituals. So Instagram has basically two options here: ban because it is illegal or allow because it is religious stuff? Also, if it’s totally fine in country X but not in country Y but the persons deciding whether to ban or not are from country Y, then what? And what if the ones evaluating are quite christian and decide to ban based on their religious beliefs? Can Instagram guarantee the neutrality of the moderation? So many difficult questions in this specific case but it’s true for many other issues.

I’d say that one way to help mitigate wrongful takedowns would be to offer better appealing tools and combine other methods of moderation considering aspects such as account activity, longevity, exposure and amount of users flagging before taking action and when determining the urgency of an appeal. Even then I’m fairly sure collateral damage would still happen. It is an herculean task after all.

Just a few thoughts not sorted in any particular order, my apologies if it came out in a messy way. Happy new year folks!

Ninja says:

As far as I know ayahuasca is also widely used in religious rituals. Shamanism if memory serves. Even though it is generally illegal (as in forbidden by law) here generally speaking you are free to produce and use in religious rituals. So Instagram has basically two options here: ban because it is illegal or allow because it is religious stuff? Also, if it’s totally fine in country X but not in country Y but the persons deciding whether to ban or not are from country Y, then what? And what if the ones evaluating are quite christian and decide to ban based on their religious beliefs? Can Instagram guarantee the neutrality of the moderation? So many difficult questions in this specific case but it’s true for many other issues.

I’d say that one way to help mitigate wrongful takedowns would be to offer better appealing tools and combine other methods of moderation considering aspects such as account activity, longevity, exposure and amount of users flagging before taking action and when determining the urgency of an appeal. Even then I’m fairly sure collateral damage would still happen. It is an herculean task after all.

Just a few thoughts not sorted in any particular order, my apologies if it came out in a messy way. Happy new year folks!

Anonymous Coward says:

An addressed issue with moderating the big sites like FaceBook and Twitter is that they are not a single community with common standards, but many overlapping communities with differing standards. The same image may or may not be offensive to the same person depending on context. I.e a breast feeding woman is acceptable within a community of mothers not so much on a dating site.

Anonymous Coward says:

It’s worth noting that allowing communities to discuss drugs, and allowing information about drugs to be posted, serves a powerful harm reduction role. Advice can be disseminated concerning potential hazards to avoid, how to respond to drug-related emergencies, and much more. I was an addict for years, and information from the internet had helped people I knew to know what is in a given pill via imprint lookup, check safe starting dosages for unfamiliar users to avoid overdose, check dose equivalencies when using a different drug than usual, and much more.

It’s also worth noting that these sources actually disclosed the health risks involved in the relevant drugs far better than any health class or DARE scare session – specific warnings about ketamine’s effects on the bladder, how to know which cough syrup formulations should never be used for DXM abuse because of the dangerous side effects from the high doses of other ingredients, methamphetamine’s neurotoxicity relative to standard amphetamine and the actual impact of sleep deprivation, cocaine’s cardiotoxicity and how it increases roughly forty-fold when consumed alongside alcohol by combining in the bloodstream to form cocaethylene, and how GABAergics like alcohol barbiturates and benzodiazepines create a dangerous withdrawal state and are prone to a "kindling" effect where each withdrawal experience can make the next more severe (sometimes to the point of seizure or death), among many, many more details.

I wouldn’t want my kids using hard drugs, but if they are gonna go anywhere near them (as some kids will) then I want them to have the ability to access relevant information about those drugs, however they will pay attention to it. If I hadn’t known how to react in a crisis, I have friends who would likely be dead. Having access to information about drug use saves lives, and even when the stakes aren’t that high, that information increases the quality of life of those using drugs by helping them learn to prevent as many forms of harm as possible. Even if all you care about is whether addicts get clean, they have to survive that long first.

If potential addicts know the actual effects of the drugs they might use, not portrayed as a scare tactic but as an honest assessment of what is involved and how it is dealt with, they might well change their minds. But even if they do use, they will be healthier and less prone to serious harm if they have access to relevant information about drugs.

Raymondjoype (user link) says:

Как сделать личный блог 7 простых ш?

In school sensual massage women will hold erotic 4hands massage. Similar swedish massage, as in principle, and relaxation, influences on some area human body, this give a chance male gain strength.
And while, french massage and not violates practically any prohibitions, for the reason it's not about sexual contact.
Dear gentlemen!
Sensitive touch rasprekrasnoy girls will flow through your body, dipping in depth boundless the ocean pleasure. In the quiet slip, donating your skin kisses, prelestress envelops the warmth of one's body. You will be surprised at, which sea bliss today it is possible to feel fromnude massage in Midtown.
Systematically visiting the four hands massage for clients, you guarantee himself excellent sexual relaxation.

<a href=https://618cef1e3fcae.site123.me/>Рейтинг блогов о сео продвижении мой топ лучших блоггеров</a>
[—-]

Add Your Comment

Your email address will not be published.

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...
Loading...