OpenAI’s New Scientific Writing And Collaboration Workspace ‘Prism’ Raises Fears Of Vibe-Coded Academic AI Slop
from the beyond-hallucitations dept
It is no secret that large language models (LLMs) are being used routinely to modify and even write scientific papers. That’s not necessarily a bad thing: LLMs can help produce clearer texts with stronger logic, not least when researchers are writing in a language that is not their mother tongue. More generally, a recent analysis in Nature magazine, reported by Science magazine, found that scientists embracing AI — of any kind — “consistently make the biggest professional strides”:
AI adopters have published three times more papers, received five times more citations, and reach leadership roles faster than their AI-free peers.
But there is also a downside:
Not only is AI-driven work prone to circling the same crowded problems, but it also leads to a less interconnected scientific literature, with fewer studies engaging with and building on one another.
Another issue with LLMs, that of “hallucinated citations,” or “HalluCitations,” is well known. More seriously, entire fake publications can be generated using AI, and sold by so-called “paper mills” to unscrupulous scientists who wish to bolster their list of publications to help their career. In the field of biomedical research alone, a recent study estimated that over 100,000 fake papers were published in 2023. Not all of those were generated using AI, but progress in LLMs has made the process of creating fake articles much simpler.
Fake publications generated using LLMs are often obvious because of their lack of sophistication and polish. But a new service from OpenAI, called Prism, is likely to eliminate such easy-to-spot signs, by adding AI support to every aspect of writing a scientific paper:
Prism is a free workspace for scientific writing and collaboration, with GPT‑5.2—our most advanced model for mathematical and scientific reasoning—integrated directly into the workflow.
It brings drafting, revision, collaboration, and preparation for publication into a single, cloud-based, LaTeX-native workspace. Rather than operating as a separate tool alongside the writing process, GPT‑5.2 works within the project itself—with access to the structure of the paper, equations, references, and surrounding context.
It includes a number of features that make creating complex — and fake — papers extremely easy:
- Search for and incorporate relevant literature (for example, from arXiv) in the context of the current manuscript, and revise text in light of newly identified related work
- Create, refactor, and reason over equations, citations, and figures, with AI that understands how those elements relate across the paper
- Turn whiteboard equations or diagrams directly into LaTeX, saving hours of time manipulating graphics pixel-by-pixel
There is even voice-based editing, allowing simple changes to be made without the need to write anything. But scientists are already worried that the power of OpenAI’s Prism will make a deteriorating situation worse. As an article on Ars Technica explains:
[Prism] has drawn immediate skepticism from researchers who fear the tool will accelerate the already overwhelming flood of low-quality papers into scientific journals. The launch coincides with growing alarm among publishers about what many are calling “AI slop” in academic publishing.
One field that is already plagued by AI slop is AI itself. An FT article on the topic points to an interesting attempt by the International Conference on Learning Representations (ICLR), a major gathering of researchers in the world of machine learning, to tackle this problem with punitive measures against authors and reviewers who violate the ICLR’s policies on LLM-generated material. For example:
Papers that make extensive usage of LLMs and do not disclose this usage will be desk rejected [that is, without sending them out for external peer review]. Extensive and/or careless LLM usage often results in false claims, misrepresentations, or hallucinated content, including hallucinated references. As stated in our previous blog post: hallucinations of this kind would be considered a Code of Ethics violation on the part of the paper’s authors. We have been desk -rejecting, and will continue to desk -reject, any paper that includes such issues.
Similarly:
reviewers [of submitted papers] are responsible for the content they post. Therefore, if they use LLMs, they are responsible for any issues in their posted review. Very poor quality reviews that feature false claims, misrepresentations or hallucinated references are also a code of ethics violation as expressed in the previous blog post. As such, reviewers who posted such poor quality reviews will also face consequences, including the desk rejection of their [own] submitted papers.
It is clearly not possible to stop scientists from using AI tools to check and improve their papers, nor should this be necessary, provided authors flag up such usage, and no errors are introduced as a result. A policy of the kind adopted by the ICLR requiring transparency about the extent to which AI has been used seems a sensible approach in the face of increasingly sophisticated tools like OpenAI’s Prism.
Follow me @glynmoody on on Bluesky and Mastodon.
Filed Under: academia, ai slop, citations, collaboration, ethics, fakes, gpt, latex, nature, peer review, refactoring, reviewers, transparency, vibe coding, whiteboard, workspace
Companies: ft, openai
Comments on “OpenAI’s New Scientific Writing And Collaboration Workspace ‘Prism’ Raises Fears Of Vibe-Coded Academic AI Slop”
Decide your position
Techdirt staff really need to decide if techdirt as a company wants to embrace AI, reject AI, or take a more nuanced position such as ‘cautiously optimistic’ or ‘distrustful but willing to experiment’
It’s really annoying to read an article glazing AI by one author, one blazing it from another, an article criticizing a game publisher for rejecting AI assisted games from a third, and a fourth author worrying about AI generated scientific papers ruining science, and seeing an ad for AI in techdirt’s direct content ads
The commentators are generally in the ‘distrustful of ai’ space, the negative-towards-ai end of the spectrum, so if you do consolidate your opinions you might want to move towards that end of the pool. But I’d also be fine with Techdirt taking a consistent ‘just the facts’ view instead of editorializing, or even consolidating in the positive end of the pool.
I just want to have a known slant to correct for for the whole site, instead of a random slant based on the guest author.
Re:
It seems to say a lot more about you and what you expect from a website that you think there needs to be a stated position or “a slant” from an entire publication on any particular technology.
Is it that hard to believe that there are a variety of nuanced opinions held by the writers of this site, and even that the nuances and opinions change over time as the data changes?
But, no, that’s too difficult for you, the guy who keeps insisting that free speech is for suckers.
Maybe instead of demanding something pointless like an announced “slant” you can use your fucking brain to read an article and address the points in that article.
Too hard? No wonder you demand censorship.
Re:
Nah, you don’t speak for the rest of us. I prefer individuals retain their own thoughts rather than adopt the boss’ chosen slant. That’s exactly what’s wrong with Bezos owning the Washington Post. It’s not journalism, just Bezos’ slant on stuff happening.
You should fucking applaud Mike for not trying to force Techdirt writers to assimilate a single perspective on a nuanced topic.
Imagine wanting less free thought and intellectual honesty and sincerity in the world…
OpenAI doesn't care
“Not only is AI-driven work prone to circling the same crowded problems, but it also leads to a less interconnected scientific literature, with fewer studies engaging with and building on one another.”
Non-scientists might not grasp how serious this problem is; but people in the field do. But OpenAI doesn’t care: burn down science, burn down literature, burn down journalism, burn down the power grid, burn down the environment, burn down the Internet, burn down everything, it doesn’t matter to them — as long as they can keep feeding their egos and their greed.
But the “reproducibility crisis” has been going on for 10-15 years now (OpenAI’s existed for 10, but the recent craze is barely 5 years old), and people had been raising concerns with less fanfare since the 1960s. So this could actually be good. People might be more able to get funding for replication and verification studies. And if they’re actually gonna look up the cited things, rather that just hitting a paywall and giving up, maybe it’ll speed up the demise of paywalls for scientific papers and technical standards.
It hardly matters how much computer assistance was used if the results can be independently replicated.
Re:
It’s pretty much true with any AI-generated content. Theses companies doesn’t publish anything but tools, and only authors is responsible for their final works.
If someone manages to cure lung cancer, using only AI or after 30 years of hard work on a cold basement eating only raw rice, only the result matters. No Nobel Price has ever been awarded to people only because they have suffered being locked in an office all their life.
Same goes for books or music (where authors use computers), cooking (even great cooks use microwaves), or anything else, as long as we don’t judge the final work using LLM, only we (as human) can decide to reward or blame authors.
As for studies, if the budget allocated to a team of researchers is only based of the number of citations, we need to blame this decision, and not the tools that have been used by researchers. And there have been already a bunch of very bad studies and terrible authors long before AI was created (in the 1970s) .
LLMs can help produce clearer texts with stronger logic
According to whom?
The “positive benefits” found were more citation (because we know how to game citation as with search) and AI-users “reaching leadership roles faster”, which, define those terms, and explain why it is better, and why they are being shoveled into leadership roles.
It is clearly not possible to stop scientists from using AI tools to check and improve their papers,
It’s clear they have failed to be scientists at all outside their narrow discipline, and probably within it.
That’s a start, but if the infraction is that egregious quite frankly they should be blackballed entirely. There needs to be long term consequences.
RE: Number of Citations
“AI adopters have published three times more papers, received five times more citations, and reach leadership roles faster than their AI-free peers.”
It seems a little strange and too business-minded to use these as metrics. Volume of papers published doesn’t indicate the worth of what is written (and there are several, if anecdotal, examples of it being different flavors of red flag)
Citations is also strange as a critical paper will also be citing the original work. Just having more doesn’t make the paper strictly better.
And it’s and old trope sure but I have a pretty mixed view, to be charitable, of people promoted to leadership roles. I won’t call them all disconnected sycophants but they are typically the sort to be dazzled by stats that don’t actually back up what they claim, and/or get all hot and bothered over the newest tech buzzwords.
Nature and Science both into the fuck-it bucket, then.
Also Techdirt, unless its credibility finds a way out of the LLM worship gutter I’m watching it crashing into.
“AI adopters have published three times more papers, received five times more citations, and reach leadership roles faster than their AI-free peers.”
…and…
” More seriously, entire fake publications can be generated using AI, and sold by so-called “paper mills” to unscrupulous scientists who wish to bolster their list of publications to help their career. In the field of biomedical research alone, a recent study estimated that over 100,000 fake papers were published in 2023.”
…seem to be very related to me.