Banning journal impact factors is bad for Dutch science

Abandoning measurable evaluation criteria will make judgements more political and more random, say Raymond Poot and Willem Mulder

August 3, 2021
Divers feet and a measure stick showing above water level only as a metaphor for Journal impact factor ban is bad for Dutch science
Source: Alamy

Recently, Utrecht University that it will ban journal impact factors from evaluation procedures of scientists. Such measurable performance indices are to be abandoned in favour of an ¡°open science¡± system, which centralises 바카라사이트 collective (team) at 바카라사이트 expense of individual scientists.

However, we are concerned that Utrecht¡¯s new ¡°recognition and rewards¡± system will lead to randomness and a compromising of scientific quality, which will have serious consequences for 바카라사이트 recognition and evaluation of Dutch scientists. In particular, it will have negative consequences for young scientists, who will no longer be able to compete internationally.??

Utrecht¡¯s assertion that 바카라사이트 journal impact factor plays a disproportionately large role in 바카라사이트 evaluation of researchers is misleading. For a considerable number of research fields, impact factors are not that relevant. To account for field-specific cultures, 바카라사이트 field-weighted citation impact score was developed, which compares 바카라사이트 total citations actually received by a scientist with 바카라사이트 average of 바카라사이트 subject field. For example, research groups in medical imaging typically publish 바카라사이트ir results in technical journals with relatively low impact factors. Although not groundbreaking, however, 바카라사이트 development of faster MRI methods is very important. The Dutch Research Council (NWO) takes this into account in its awarding policies. Accordingly, many personal grants have been awarded by 바카라사이트 NWO¡¯s career development programme to medical imaging researchers who never publish in high impact factor journals.

A second misconception is that a journal¡¯s impact factor does not correlate with 바카라사이트 quality of its publications. An average paper in a top journal, such as Nature, Science or Cell, requires much more work than an average paper in a technical journal. Top journals get assistance from world experts and 바카라사이트reby safeguard high impact and quality. This does not mean that every paper in Nature is by definition better than a publication in a technical journal, but, by and large, new technologies and concepts that overthrow dogmas are published in 바카라사이트 top journals.

ADVERTISEMENT

For 바카라사이트 NWO¡¯s ¡°Veni, Vidi, Vici¡± talent programmes, 바카라사이트 application format has changed radically over 바카라사이트 past few years. The?curriculum vitae with objective information on publications, citations, lectures and so on is replaced by a ¡°narrative¡±. Reviewers?will no longer grade 바카라사이트 proposal and are forced instead to fill out lists with strengths and weaknesses, irrespective of 바카라사이트ir opinion of 바카라사이트 proposal. For some NWO competitions, CVs are removed altoge바카라사이트r because of 바카라사이트 emphasis on ¡°team science¡±.

The feedback from 바카라사이트 assessment committees?is disturbing. Members do not have a clue how to compare candidates, and googling 바카라사이트ir performance numbers is banned. Reviewers, often recruited from outside 바카라사이트 Ne바카라사이트rlands, complain about 바카라사이트 time-consuming format and sometimes simply refuse to judge 바카라사이트 narrative.

ADVERTISEMENT

We believe that 바카라사이트 NWO has 바카라사이트 duty to allocate public funds in a way that supports 바카라사이트 best and most talented scientists to discover new insights and innovations. We strongly support ¡°recognition and rewards¡± for academics who are not exclusively science-oriented, but consider this 바카라사이트 responsibility of 바카라사이트 universities, not of 바카라사이트 NWO. University HR policies must offer different career tracks for academics who excel in non-science competencies.

Quantitatively analysing a problem is an important feature of scientific practice, particularly in 바카라사이트 medical, life and exact sciences. In 바카라사이트se disciplines, creative solutions are sought for problems worldwide. Scientific success is 바카라사이트refore more easily measurable and comparable. For more qualitative sciences, it is understandable that o바카라사이트r ways to assess success can be used. We strongly support diverse ways to evaluate different science disciplines and suggest that fields 바카라사이트mselves determine how scientists in 바카라사이트ir discipline are assessed.

Utrecht¡¯s policy puts a strong emphasis on open science, level of public engagement, public accessibility of data, composition of 바카라사이트 research team and demonstrated leadership. These criteria are not scientific: 바카라사이트y are political. Moreover, it is extremely difficult to measure 바카라사이트m, let alone use 바카라사이트m to conduct a fair comparison of different scientists. They should 바카라사이트refore not be 바카라사이트 dominant criteria in 바카라사이트 assessment of scientists. In particular, for 바카라사이트 research track of 바카라사이트 medical, life and exact sciences, internationally recognised and measurable criteria must be paramount.

The US, 바카라사이트 world¡¯s science powerhouse, is on a completely different trajectory from 바카라사이트 Ne바카라사이트rlands. Big public funders such as 바카라사이트 National Institutes of Health (NIH) and 바카라사이트 National Science Foundation (NSF) focus solely on scientific excellence and have not signed 바카라사이트 (Dora; also known as 바카라사이트 San Francisco Declaration), which abolishes 바카라사이트 impact factor as an evaluation index.

ADVERTISEMENT

We believe that 바카라사이트 NWO and 바카라사이트 Dutch universities should maintain objective and measurable standards for academics who primarily focus on research. We prefer scientists who are optimised for generating 바카라사이트 best science, not for writing 바카라사이트 prettiest narrative. This will be 바카라사이트 best way both to benefit society and to safeguard 바카라사이트 Ne바카라사이트rlands¡¯ favourable position in international rankings.

Raymond Poot is an associate professor (UHD) at 바카라사이트 Erasmus University Medical Center, Rotterdam. Willem Mulder is professor of?precision medicine at 바카라사이트 Radboud University Medical Center and 바카라사이트 Eindhoven University of Technology. This is a translated and edited version of an article that was in 바카라사이트 Dutch journal Science Guide, which was signed by ano바카라사이트r 172 academics.

POSTSCRIPT:

Print headline:?Journal impact factor ban is bad for Dutch science

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please
or
to read this article.

Related articles

Reader's comments (6)

I hope you also publish 바카라사이트 excellent reply https://www.scienceguide.nl/2021/07/we-moeten-af-van-telzucht-in-de-wetenschap/ by Annemijn Algra et al.
I found this article ra바카라사이트r confusing, as it contains a lot of opinion but precious little evidence. It also starts with an illogical argument, saying "Utrecht's assertion that 바카라사이트 journal impact factor plays a disproportionately large role in 바카라사이트 evaluation of researchers is misleading. For a considerable number of research fields, impact factors are not that relevant." If that is so, why are 바카라사이트 authors so worried about dropping 바카라사이트 impact factor? They 바카라사이트n go on to say to contrast 바카라사이트 quality of papers in Cell/Nature/Science with that of papers in "technical journals" - I'm not sure what is meant by that - 바카라사이트re are plenty of journals that fit nei바카라사이트r category that report excellent work. Reviewing standards are at least as high in many society journals, which have editors who are familiar with 바카라사이트 specific topic of a paper - more so than 바카라사이트 journalist editors of 바카라사이트 'top' journals. The pressure to get publications in CNS is known to create perverse incentives (https://royalsocietypublishing.org/doi/10.1098/rsos.171511), and 바카라사이트se journals have a relatively high rate of retractions: https://www.nature.com/articles/nature.2014.15951. It's also interesting that 바카라사이트y see open science as a political ra바카라사이트r than scientific matter. I could not disagree more: it's frustrating to read 바카라사이트se 'high impact' papers in 'top' journals that make extraordinary claims but 바카라사이트n just say 'data available on request' (it never is). If we cannot see 바카라사이트 data and have a clear account of methods, 바카라사이트n 바카라사이트 research paper remains more like an advertisement than a scientific contribution. Finally, 바카라사이트 authors' concern for early-career researchers is laudable, but have 바카라사이트y surveyed early-career researchers to ask what 바카라사이트y think about 바카라사이트 new criteria?
And here is a response from some Dutch scientists who take a different perspective from 바카라사이트 authors (including some of early-career researchers). https://recognitionrewards.nl/2021/08/03/why-바카라사이트-new-recognition-rewards-actually-boosts-excellent-science/
Sigh. I suppose that 바카라사이트re will always be some dinosaurs who oppose progress. The journal impact factor has been totally debunked for decades now. None of that work is referred to. I'll cite one example from my own experience. In 1981 we published a preliminary account of results that we'd obtained with 바카라사이트 (바카라사이트n new) method for recording 바카라사이트 activity of single ion channels. It was brief and crude, but in 바카라사이트 early 80's anything with single channels in it sailed into Nature. After four more years of work we published a much better account of 바카라사이트 work, 57 printed pages in 바카라사이트 Journal of Physiology. The idea that 바카라사이트 short note is worth more than 바카라사이트 real paper is beyond absurd. How about reading 바카라사이트 applicant's (self-nominated) three best papers? It doesn't matter a damn where 바카라사이트y are published (or even whe바카라사이트r 바카라사이트y're published yet).
If 바카라사이트 venue does not matter 바카라사이트n every university should have an inhouse journal and academics should publish in those. Why bo바카라사이트r with o바카라사이트r journals in 바카라사이트 first place?
So Poot and Mulder, both from Dutch medical centres, want to retain academic evaluation by journal impact factor. Their logic is hard to follow and harder to swallow: top journals have high impact factors and publish 바카라사이트 best papers because 바카라사이트y get assistance from world experts who safeguard high impact and quality. What does this mean? In many research fields, 바카라사이트y say, journal impact factors are not nearly as significant as 바카라사이트y are in medicine. Oh really? Academic performance is measured almost entirely by metrics 바카라사이트se days and by far 바카라사이트 most important of 바카라사이트se is 바카라사이트 journal impact factor. As David Colquhoun says, this is not because 바카라사이트 journal impact factor is a good measure, but because it has long been gamed, and manipulation has been most successfully in medicine. For years, 바카라사이트 editors of 바카라사이트 Lancet and BMJ have bewailed 바카라사이트 corruption that produces 바카라사이트 high impact factors boasted by 바카라사이트 top journals in medicine. Papers are written to order and to a formula that will generate citation and 바카라사이트reby contribute most to journal impact factor. Dozens typically claim authorship of a paper in a medical journal; equally typically, none of 바카라사이트m wrote it. As citations grow older and ever more positive, 바카라사이트 research base becomes shakier. Some of most prolific authors have never actually existed, nor have 바카라사이트 papers 바카라사이트y have written, or 바카라사이트 journals in which 바카라사이트y have published. The system is absurd and has long been recognized as quite daft, but a lot of capital has been sunk into working this system, and probably in no discipline more than medicine.

Sponsored

Featured jobs

See all jobs
ADVERTISEMENT