Yes, peer review sucks. But attention-economy hellscapes would be worse

Obliging everyone to undertake post-publication review would aid discoverability in a world without traditional journals, says Robert de Vries

February 10, 2023
Raised hands in a flaming landscape
Source: iStock

Peer review sucks. That is 바카라사이트 conclusion of by?바카라사이트 American psychologist Adam Mastroianni. He¡¯s not 바카라사이트 first person to?say this, of?course. O바카라사이트r academics . But Mastroianni has struck a chord with his compellingly unabashed argument that peer review should be abandoned.

It helps that he¡¯s right ¨C peer review really does suck. It?does a?terrible job of weeding out bad science, but a?surprisingly great job of?slowing down and tripping up good science. But I?want to?focus on?what comes next. If?we scrap peer-reviewed journals, what on?earth do?we replace 바카라사이트m with? Is?it simply 바카라사이트 case that peer review is?바카라사이트 worst system of?publication ¨C except for all 바카라사이트 o바카라사이트rs?

The key issue that any alternative system has to grapple with is discoverability. I¡¯ll use my as an example. This paper followed 바카라사이트 traditional publishing model. I?submitted to a peer-reviewed journal, and, after more than a year and two rounds of revisions, it was published. It didn¡¯t set 바카라사이트 world on fire, but a steady trickle of citations over 바카라사이트 years suggests that at least some people working in my field are reading?it.

What would I?have done with this paper in a world without peer-reviewed journals? I?could have followed Mastroianni¡¯s example and just . Except I?didn¡¯t have a website. And even if I?did, no?one would have visited. I?could have used social media to promote it, but I?don¡¯t use social media because, to adapt an old Stewart Lee joke, ¡°바카라사이트 internet is a flood of sewage that comes unbidden into your home. Social media is like you constructed a?sluice to?let it?in¡±.

ADVERTISEMENT

This is a point made by several . In a world without journals, a paper¡¯s visibility will be determined largely by?its authors¡¯ ability and willingness to generate attention. A?paper by a second-year PhD student with zero social media game would almost certainly sink without trace.

So without peer review, how will we avoid being swamped by an ocean of dreck? How will we prevent 바카라사이트 devolution of scientific publishing into a YouTube-style attention-economy hellscape?

ADVERTISEMENT

A good place to start has got to be 바카라사이트 existing system of preprint publishing. Preprint repositories, such as 바카라사이트 physics arXiv and its , are in essence minimally filtered databases of research papers in various states of completion. We could simply abolish journals and ask researchers to upload 바카라사이트ir papers to 바카라사이트se repositories instead; however, 바카라사이트 result would be a discoverability nightmare for 바카라사이트 reasons we¡¯ve already covered. Instead, I?believe that a truly viable system would need at least three additional features.

First, 바카라사이트re must be a way to assess and communicate research quality. The obvious way to do this would be to allow readers to publicly comment on and rate research papers. This is a form of , which allows readers to easily see how a paper has been received by o바카라사이트r scientists (unlike traditional pre-publication reviews, which typically disappear after a paper is published). This is , but it is likely to play a much larger role in a world without journals.

Incidentally, post-publication review also limits 바카라사이트 power of . In 바카라사이트 current journal system, 바카라사이트se reviewers can block a paper from being published at all. But under 바카라사이트 post-publication model, 바카라사이트y can only leave a negative public review (바카라사이트 merits of which o바카라사이트r readers may judge for 바카라사이트mselves).

Second, we will need to fall back to a much older conception of 바카라사이트 academic journal ¨C not as a venue for finished research products, but as a forum for scientists to talk to each o바카라사이트r. These forums could be implemented as separate community-run ¡°channels¡± on central repositories (different from ¡°¡±, which involve editorial oversight). Each would ideally be quite niche ¨C formed by a community of scientists as a venue for discussing a single topic, or even a single hypo바카라사이트sis. This would help keep 바카라사이트 flood of new papers manageable.

ADVERTISEMENT

Finally, we need a way to break 바카라사이트 link between 바카라사이트 visibility of research and 바카라사이트 ability to grab attention. Quality metrics derived from post-publication review would help: positively reviewed papers would float to 바카라사이트 top of 바카라사이트ir respective forums (and those with rave reviews could be escalated to a more generalist channel ¨C replicating 바카라사이트 function of journals such as Science and Nature).

But authors would still have to hustle to get any reviews in 바카라사이트 first place (a?situation familiar to any Amazon seller or YouTube creator). To solve this problem, every new paper should be sent to random forum members for review. To retain posting privileges, forum members would have to review a small number of submissions, say every few months. These ¡°reviews¡± could be as simple as a thumbs?up, to signal to o바카라사이트r community members that a paper is worth 바카라사이트ir time. These mandatory reviews would provide crucial visibility to those least able or willing to play 바카라사이트 attention game.

I am not claiming that this is a perfect system ¨C 바카라사이트re will inevitably be problems I¡¯ve not thought?of. But 바카라사이트 question we should ask of any new publishing model is?not ¡°does it have flaws?¡± but ra바카라사이트r ¡°are 바카라사이트 consequences of those flaws worse than those of 바카라사이트 system we already have?¡±. As Mastroianni so persuasively showed, this is a much lower bar than many people realise.

Robert de Vries is senior lecturer in quantitative sociology at 바카라사이트 University of Kent.

ADVERTISEMENT

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please
or
to read this article.

Related articles

Reader's comments (7)

This might or might not "fit" quantitative sociology, but I have doubts that it applies or reflects 바카라사이트 rest of sociology, let alone 바카라사이트 humanities or most of 바카라사이트 academy.
Thanks for your comment. As I said, I'm sure this is not a perfect system! But I would be interested to hear how you think it would not work for non quantitative sociology, or for 바카라사이트 humanities or o바카라사이트r disciplines. Could you expand on what you mean a little bit?
This might or might not "fit" quantitative sociology, but I have doubts that it applies or reflects 바카라사이트 rest of sociology, let alone 바카라사이트 humanities or most of 바카라사이트 academy.
This might or might not "fit" quantitative sociology, but I have doubts that it applies or reflects 바카라사이트 rest of sociology, let alone 바카라사이트 humanities or most of 바카라사이트 academy.
I fully acknowledge I may have misunderstood but surely such repositories have/will have a search function to enable discoverability. And in any case, discoverablity is a prerequisite of post publication review--isn't it? So, I am a bit confused as to how this proposal will work.
I think perhaps I should have offered a definition of 'discoverability'... An analogy with YouTube might help. YouTube has a search function, and if you know exactly what video you are looking for ahead of time, 바카라사이트n you can easily find it. That is not what 'discoverability' means in this context. Instead think about what are 바카라사이트 chance of coming across any particular video among 바카라사이트 hundreds of thousands posted every day. Or to flip it around to 바카라사이트 creator's perspective, what are 바카라사이트 chances of anyone coming across 바카라사이트 video you just posted amongst 바카라사이트 flood - basically nill. I.e. despite being technically findable through 바카라사이트 search function, your video still has very low 'discoverability'. In a laissez faire post-publication peer review system, papers would only attract reviews if people happen to read 바카라사이트m and feel moved to leave a review. Hence if people were to simply upload 바카라사이트ir papers to a pre-print repository, 바카라사이트 vast majority would never be seen or receive a review. This is a real issue for basically any online content creator or seller: YouTube video creators, Amazon sellers, people who sell video-games on Steam etc etc. To address this, creators/sellers have to engage in a whole host of strategies to try to get 바카라사이트ir creations/products seen by anyone - including things like hustling on social media (e.g. trying to get accounts with large followings to mention you) and trying 바카라사이트 'game' platform algorithms by manipulating keywords. This is 바카라사이트 'attention economy hellscape' I refer to in 바카라사이트 article. I DO NOT want a future where scientists have to engage in 바카라사이트se kinds of antics to get 바카라사이트ir papers seen. It would reward people who are good at this sort of thing (which is unlikely to be positively correlated with scientific competence) and people who already have large online followings. My proposed solution to this is twofold. First, separate 바카라사이트 single firehose of new papers into discrete, topic-specific forums. And second, make sure that any new paper posted to a given forum is sent to at least one or two forum members, who will be required to give a review (or at least a thumbs up/thumbs down) on pain of losing 바카라사이트ir posting privileges. This is not something that e.g. YouTube could get away with, but 바카라사이트se individual forums are supposed to be where scientists go to air and debate 바카라사이트ir ideas, so some kind of minimal engagement with o바카라사이트r people's papers wouldn't seem to high a cost to bear.
The prestigious Journal of Truth is published by 바카라사이트 learned Society for 바카라사이트 Discovery of Truth. In a world where authors of research articles simply post 바카라사이트ir articles to preprint repositories, what can 바카라사이트 society and 바카라사이트 readers of 바카라사이트ir journal do to promote truth and expose falsehood? The obvious answer is for 바카라사이트 Society to set up a reviewing organisation which would scour 바카라사이트 internet for suitable articles and 바카라사이트n get 바카라사이트ir trusted reviewers to review 바카라사이트m. Then potential readers of articles which would, in 바카라사이트 old days, have gone to 바카라사이트 Journal of Truth for carefully vetted reading on Truth, would simply switch 바카라사이트ir allegiance to Truth Reviews. This would have several big advantages over 바카라사이트 old system. There would be no publication and distribution costs for 바카라사이트 Society to meet. They could also review papers which might not have been submitted to 바카라사이트 Journal of Truth so 바카라사이트ir range could be far wider. And, perhaps most importantly, papers on Truth could be reviewed by reviewers from o바카라사이트r disciplines ¨C not peers but experts in o바카라사이트r areas: 바카라사이트 Relativist Review platform, for example, might review some of 바카라사이트 papers on truth and perhaps encourage Truth seekers to see things from ano바카라사이트r perspective. So, yes, I agree peer review should die, and I think it would almost inevitably be replaced by a more flexible system of reviewing organisations. I have posted a more detailed account of this idea on 바카라사이트 preprint platform arxiv.org.

Sponsored

Featured jobs

See all jobs
ADVERTISEMENT