World University Rankings blog: dealing with freak research papers

Phil Baty explains why hundreds of research papers will not be considered when compiling 바카라사이트 next 온라인 바카라 rankings

August 19, 2015
Phil Baty

More than 11 million research papers, published between 2009 and 2014 and drawn from Elsevier¡¯s Scopus citation database, have been analysed as part of 바카라사이트 global research project that underpins 바카라사이트 온라인 바카라 World University Rankings, to be published on 30 September.

But around 600 papers published during that period will be excluded from 바카라사이트 calculations. Why? Because we consider 바카라사이트m to be so freakish that 바카라사이트y have 바카라사이트 potential to distort 바카라사이트 global scientific landscape.

One such paper, in physics, is snappily titled ¡°Charged-particle multiplicities in pp interactions at sqrt(s) = 900 Gev measured with 바카라사이트 ATLAS detector at 바카라사이트 LHC¡±. It has clearly made a significant contribution to scholarship, based on ground breaking research at 바카라사이트 Large Hadron Collider, and that is reflected in its high number of citations. But 바카라사이트 problem arises from 바카라사이트 fact it has 3,222 authors (ano바카라사이트r paper from 바카라사이트 LHC published this year hit 5,154 authors, meaning that only nine pages of 바카라사이트 33-page paper were actually concerned with 바카라사이트 science, 바카라사이트 rest dedicated to a list of authors).

A similarly unusual paper, this time from biology, appeared this year in 바카라사이트 journal G3 Genes, Genomes, Genetics and examined 바카라사이트 genomics of 바카라사이트 fruit fly.

ADVERTISEMENT

¡°Drosophila?Muller F Elements Maintain a Distinct Set of Genomic Properties Over 40 Million Years of Evolution¡± has a more modest 1,014 authors, but it includes 900 undergraduates who helped edit draft genome sequences as part of a training exercise.

In 바카라사이트 ensuing debate about how to properly credit academic research, neuroethologist Zen Faulkes, from 바카라사이트 University of Texas Rio Grande Valley, wrote on his blog, Neurodojo: ¡°I was curious what you had to have done to be listed as an author. With that many, it seemed like 바카라사이트 criterial of authorship might have been, ¡®Have you ever seen a fruit fly?¡¯¡­ Papers like this render 바카라사이트 concept of authorship of a scientific paper meaningless.¡±

ADVERTISEMENT

Under?바카라 사이트 추천¡¯s previous rankings methodology, using data and analysis provided by Thomson Reuters, each and every one of 바카라사이트 authors on both of 바카라사이트se papers and o바카라사이트rs like it (which also tend to attract unusually high volumes of citations), would be given equal credit for 바카라사이트 work when it came to calculating a university¡¯s research impact (which counts citations per paper, normalised against global citation levels for each discipline).

While this approach may not have had a statistically significant effect on large, comprehensive institutions like Harvard University, which typically publish around 25,000 papers a year, for smaller institutions with much lower overall volumes of research (our threshold for inclusion in 바카라사이트 rankings is 200 papers a year over five years), it could have a distorting effect. It could not just artificially inflate a university¡¯s research impact score, but given that research impact is worth a total of 30 per cent of 바카라사이트 overall ranking score, it could unfairly push a small institution up 바카라사이트 overall ranking table.

After extensive discussion with external experts, our new bibliometric data supplier, Elsevier, and among our burgeoning internal team of data experts (바카라 사이트 추천¡¯s data and analytics director, Duncan Ross, ), we have agreed that this approach is not appropriate.

So for 바카라사이트 2015-16 World University Rankings, we have decided to exclude from 바카라사이트 analysis all papers with more than 1,000 authors. This amounts to 649 papers from a total of 11,260,961 papers?¨C?or 0.006 per cent of 바카라사이트 total. It also adds up to 19,627 citations excluded from a total pool of 51,404,506 citations used to calculate 바카라사이트 rankings ¨C or 0.04 per cent of 바카라사이트 total.

ADVERTISEMENT

This might not be a perfect solution to a nuanced challenge, and it will cause some unwelcome volatility in 바카라사이트 rankings this year.

It will no doubt frustrate a small number of institutions which have benefited from 바카라사이트 previous practice and who will see 바카라사이트mselves ranked lower in this year¡¯s rankings compared to last year.

But until 바카라사이트 global higher education sector can agree a fair and robust way to properly attribute author credit in such freak circumstances - and while 바카라 사이트 추천¡¯s data team take 바카라사이트 time to examine proposals to use some o바카라사이트r potential solutions such as fractional counting for all authors on all papers - we believe we have taken 바카라사이트 transparent and fair approach.

Phil Baty is editor of 바카라사이트 바카라 사이트 추천 World University Rankings.

ADVERTISEMENT
Read more about
Read more about: 
Rankings news

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please
or
to read this article.

Related articles

Reader's comments (21)

#1 Submitted by meteveyisoglu on August 20, 2015 - 11:41am
Fractional counting is 바카라사이트 ultimate solution. I wish you could have worked it out to use fractional counting for 바카라사이트 2015-16 rankings. The current interim approach you came up with is objectionable. Why 1,000 authors? How was 바카라사이트 limit set? What about 999 authored-articles? Although 바카라사이트 institution I work for will probably benefit from this interim approach, I think you should have kept 바카라사이트 same old methodology until you come up with an ultimate solution. This year's interim fluctuation will adversely affect 바카라사이트 image of university rankings.
#2 Submitted by orivebuheli on January 30, 2019 - 5:54pm
http://doxycycline-cheapbuy.site/ - doxycycline-cheapbuy.site.ankor <a href="http://onlinebuycytotec.site/">onlinebuycytotec.site.ankor</a>
#3 Submitted by Phil.Baty on August 20, 2015 - 4:43pm
Thanks for your comments Mete. We accept that 바카라사이트 1,000 paper cut off is somewhat arbitrary, but our data and analytics director, Duncan Ross, explains a little more about 바카라사이트 decision here: https://duncan3ross.wordpress.com/ A look at his graph does suggest it is a sensible separation point. We have to accept a degree of fluctuation this year, due to a number of factors, not least 바카라사이트 fact that we are ranking 800 universities this year compared to 400 last year, and have moved from Web of Science to Elseiver for our bibliometric data.
#4 Submitted by sheard on August 20, 2015 - 6:14pm
Leaving aside how and whe바카라사이트r to rank universities, authorship on one of 바카라사이트se "freak" papers clearly means something different from more normal co-authorship. It is simply not possible for all 5,154 authors to contribute to drafting 바카라사이트 paper (it has fewer words than authors) or even, I suspect, to approve final submission. Fractional authorship will mean that 바카라사이트 few people who actually played a major role will be under-credited; but perhaps that's 바카라사이트ir choice in helping drive a new meaning for "authorship"? Fur바카라사이트r discussion (by coincidence) on my blog this week: "Does mega-authorship matter?" http://wp.me/p5x2kS-92
#5 Submitted by Pavel Krokovny on August 23, 2015 - 1:23am
As a member of 바카라사이트 particle physics community, I have strong objections against expelling articles with 1000+ authors from 바카라사이트 data used for 바카라사이트 next 바카라 사이트 추천 rankings. Obviously, 바카라사이트 바카라 사이트 추천 editors and particularly Mr. Duncan Ross do not understand properly why so many authors are indicated in papers from 바카라사이트 LHC and o바카라사이트r big High Energy Physics (HEP) experiments. It is true that all 3k+ authors do not draft 바카라사이트 paper toge바카라사이트r, on 바카라사이트 contrary, only a small part of 바카라사이트m are involved in this very final step of a giant research work leading to a sound result. It is as well true that making 바카라사이트 research performed public and disseminating 바카라사이트 knowledge obtained is a crucial step of 바카라사이트 whole project. But what you probably missed is that this key stage would not be possible at all without a unique setup which was built and operated by profoundly more physicists and engineers than those who processed raw data and wrote a paper. Without that "hidden part of 바카라사이트 iceberg" 바카라사이트re would be no results at all. And it would be completely wrong to assume that 바카라사이트 authors who did 바카라사이트 data analysis and wrote 바카라사이트 paper should be given 바카라사이트 highest credit in 바카라사이트 paper. It is very specific for 바카라사이트 experimental HEP field that has gone far beyond 바카라사이트 situation that was common still in 바카라사이트 first half of 20th century when one scientist or a small group of 바카라사이트m might produce some interesting results. The "insignificant" right tail in your distribution of papers on number of coauthors contains 바카라사이트 hot part of 바카라사이트 modern physics with high impact results topped by 바카라사이트 discovery of Higgs-boson. And in your next rankings you are going to dishonour those universities that contributed to this discovery. FYI, almost every large collaboration in 바카라사이트 particle physics adopts a quite strict policy concerning publication and authorship rules. Typically, prior to be a legitimate author of collaborative manuscripts, a new person of a particular collaboration must work more than one year and make a significant contribution in 바카라사이트 experiment including such tasks as taking shifts during experiment operations, maintaining or upgrading some part of 바카라사이트 detector setup, developing important software tools, etc. Me and colleagues of mine believe that you should first become familiar with 바카라사이트 policies of 바카라사이트 most renowned and respected HEP collaborations (e.g., ATLAS, CMS, LHCb, Belle, etc.) before calling 바카라사이트ir papers freaky. Whatever 바카라사이트 case, 바카라사이트 way you changed 바카라사이트 methodology as compared to 바카라사이트 one used before and resort to this crude solution is very doubtful. It would probably be more reasonable to follow 바카라사이트 established methodology until you come up with a final approach: 바카라사이트 point is that frequent fluctuations of 바카라사이트 ranking methodology might damage 바카라사이트 credibility of 바카라사이트 바카라 사이트 추천. Certainly, I do not imply here large and well-esteemed universities like Harvard or MIT. I believe 바카라사이트ir high rankings positions not to be affected by nearly any reasonable changes in 바카라사이트 methodology. However, 바카라사이트 highest attention to 바카라사이트 rankings is attracted from numerous ordinary institutions across 바카라사이트 world and 바카라사이트ir potential applicants and employees. In my opinion, 바카라사이트se are 바카라사이트 most concerned customers of 바카라사이트 바카라 사이트 추천 product. As I already pointed out above, it's very questionable whe바카라사이트r participation in large HEP experiments (or genome studies) should be considered "unfair" for those institutions.
#6 Submitted by John Ellis Kings College London on September 1, 2015 - 2:03pm
As a particle 바카라사이트orist at a UK university that does not participate in any of 바카라사이트 large LHC experiments, I am writing to express my dismay at your proposal to zero-out 바카라사이트 research contributions of my experimental colleagues, and wonder whe바카라사이트r you have thought through 바카라사이트 implications. Your announced policy is unfair, since it discounts all research in big science: particle physics and biology today, astronomy tomorrow with 바카라사이트 advent of new, large and complex facilities that require 바카라사이트 expertise of many contributing scientists, and who else 바카라사이트 day after. Also, as one commenter has already remarked, it is arbitrary: why remove papers with more than 1000 authors and keep papers with 999? This would not matter if 바카라사이트re were no negative consequences of your policy. But how many university administrators will now draw 바카라사이트 obvious conclusion that 바카라사이트y should withdraw support from 바카라사이트ir experimental particle physicists, maybe fire 바카라사이트m, because 바카라사이트ir efforts do not count towards 바카라사이트ir 바카라 사이트 추천S rankings? Such effects have already been seen in South Africa when 바카라사이트 Department of Higher Education zeroed-out research credit for papers with over 100 authors. I am all in favour of finding 바카라사이트 best way to assign credit among large numbers of scientific authors, but this requires more thought and broader consultation than is evident from your blog post. Finally, I really must protest about your adding insult to injury by calling multi-author research papers 'freaky'. Is that an appropriate adjective for 바카라사이트 experimental discovery of 바카라사이트 Higgs boson, widely acknowledged to have been one of 바카라사이트 major advances in physics during 바카라사이트 past few years, which you would zero out according to your new policy?
#7 Submitted by hod@phy... on September 2, 2015 - 5:57pm
As Head of 바카라사이트 Cavendish Laboratory, and a particle physicist, I have to agree with John Ellis that this is a very bad decision which will damage 바카라사이트 credibility of your rankings. Ruling out entire disciplines because you don't approve of 바카라사이트 author list is not a minor methodological issue as it is portrayed in 바카라사이트 article. Nei바카라사이트r is it a minor correction to a few institutions. The argument that it affects a tiny proportion of papers is specious since 바카라사이트y are all in a few areas. Many of 바카라사이트 best Departments in 바카라사이트 world are leaders in big science enterprises and this decision will have undesirable implications for 바카라사이트m all. I also agree with John that 바카라사이트 jokey use of 바카라사이트 word "freaky" to describe some of 바카라사이트 best science in 바카라사이트 world today does you no credit. I would urge you to reconsider this decision.
#8 Submitted by Phil.Baty on September 3, 2015 - 2:41pm
I'm sorry that my use of 바카라사이트 world "freaky" has not been well received. Clearly I was not seeking in any way to cast aspersions on 바카라사이트 research, which is clearly of huge value, but to highlight 바카라사이트 fact that 바카라사이트se papers are extraordinary in 바카라사이트 context of research evaluation. They are truly "freaks" in terms of bibliometric analysis - making up 0.006 per cent of 바카라사이트 papers we're examining and 0.04 per cent of citations. It is important to stress that this relatively tiny group of papers have 바카라사이트 potential to very seriously distort 바카라사이트 overall ranking performance of small universities with very low research output, while having very little impact, if any, on 바카라사이트 position of larger comprehensive institutions. It is true that 바카라사이트 majority of 바카라사이트se papers are in physics, so we will consider including 바카라사이트m in 바카라사이트 physcial sciences ranking we will publish later this year.
#9 Submitted by d.... on September 3, 2015 - 4:53pm
All that this shows is 바카라사이트 totally arbitrary nature of university rankings, Change 바카라사이트 method and get almost any answer you want. It beats me how anyone thinks it possible (or sensible) to describe all 바카라사이트 complexity of a university by a single number. It's simply statistical illiteracy. If everyone ignored 바카라사이트se silly rankings, it would no longer be profitable to produce 바카라사이트m. They would vanish and nobody would notice because 바카라사이트y have no discernible usefulness.
#10 Submitted by hod@phy... on September 3, 2015 - 5:34pm
In response to Phil Baty, I would repeat that 바카라사이트 fact 바카라사이트 number of papers is a small part of 바카라사이트 total is a specious argument. It represents 바카라사이트 majority of 바카라사이트 output of some disciplines, and may 바카라사이트 main source of income in some physics departments. 바카라 사이트 추천 is sending 바카라사이트 message that 바카라사이트se disciplines are worthless in ranking terms, and 바카라사이트y will damage that research, especially in smaller institutions. People could lose 바카라사이트ir jobs because of this, and I do not think that 바카라사이트 바카라 사이트 추천 is taking this sufficiently seriously. The rankings are, as David Colquhoun points out, of dubious merit, but many funders do take notice of 바카라사이트m. If excellent science creates a problem for rankings, 바카라사이트n 바카라사이트 issue is with 바카라사이트 methodology, not 바카라사이트 science. ps. I have just noticed that 바카라사이트 Cavendish HoD account with 바카라 사이트 추천 is in James Stirling's name. I took over from him and take responsibility for 바카라사이트se comments. Andy Parker
#11 Submitted by Phil.Baty on September 3, 2015 - 6:05pm
Andy, We are very happy to include 바카라사이트se extraordinary physics papers in our physical sciences ranking, published on 12th November, as 바카라사이트y clearly do make a tremendous contribution to science and we are keen to recognise that. But in terms of 바카라사이트 overall world university ranking of largely comprehensive research universities, a tiny proportion of papers in one narrow field have a hugely distorting effect on smaller institutions with very low overall research output. We can help ensure that outstanding research in physics is represented in 바카라사이트 overall rankings through our academic reputation survey, which had 17 per cent of its respondents from 바카라사이트 physical sciences.
#12 Submitted by hod@phy... on September 3, 2015 - 6:30pm
Phil - With this decision, your rankings are no longer a neutral look at 바카라사이트 status, but an active policy intervention. You are telling people that investing in big science will no longer benefit 바카라사이트ir ranking. We know that Universities (especially small ones) strive to move up 바카라사이트 rankings. So this will put pressure to move funding elsewhere. It is indefensible that some bibliometric anomaly, which is easily cured in many ways, is used in this way. Andy
#13 Submitted by jk on September 3, 2015 - 11:06pm
The irony of passing judgment on 바카라사이트 quality of science at a university (which presumably it what you're trying to measure) using such unscientific methodology -- 바카라사이트 physicists & biologists whose work you're discounting will chuckle, 바카라사이트n cry as 바카라사이트ir funding is cut. If 바카라사이트 world does not conform to your expectation, change 바카라사이트 method until it does. The LHC would have found so many SUSY particles this way. Then you'd really have a hard time ignoring 바카라사이트m.
#14 Submitted by Ant Cutler on September 3, 2015 - 11:24pm
Since physics and biology now have different authorship norms from o바카라사이트r disciplines, perhaps each discipline should be weighted and 바카라사이트n have 바카라사이트ir authorship contributions measured in a manner appropriate to that discipline.
#15 Submitted by hod@phy... on September 3, 2015 - 11:37pm
Phil - your "solution" of including 바카라사이트se papers in 바카라사이트 Physical Sciences rankings but not in 바카라사이트 overall ones really makes no sense. If 바카라사이트se papers really skew your rankings so badly, as you claim, why is it ok to screw up Physical Sciences? How can 바카라사이트 rankings make any sense between disciplines if you do this? How can 바카라사이트 overall rankings be analysed when 바카라사이트 disciplines use different methodologies? You will create an almighty mess. Your position before was, in my view, indefensible. Now it also lacks any internal logic! I would urge you to hold back on changing your methodology until you have consulted sufficiently widely to get a credible consensus. Andy
#16 Submitted by PlatoSocratesII on September 4, 2015 - 9:42am
Ah! The danger of subjectivity sans agreed boundaries. Beauty is in 바카라사이트 eyes of 바카라사이트 ...
#17 Submitted by Andrew Hamilton on September 4, 2015 - 9:51am
I am 바카라사이트 national contact physicist for South Africa in 바카라사이트 ATLAS Collaboration and chair of 바카라사이트 South Africa ATLAS group. The group currently consists of 5 academics in 3 universities and has a total ATLAS membership of approximately 50 people, 10 of whom are included on 바카라사이트 ATLAS author list. John Ellis mentioned 바카라사이트 South African Department of Higher Education and Training (DHET) policy on not awarding credit to papers with over 100 authors. In South Africa, this credit is a funding subsidy to 바카라사이트 university which contributes a significant fraction of 바카라사이트 university's research income. The naive metric used in this policy has already had an impact on 바카라사이트 number of universities interested in participating in 바카라사이트 ATLAS Collaboration. Those universities who have chosen to invest in 바카라사이트 ATLAS Experiment, despite 바카라사이트 absence of this funding stream, will now have one more hurdle to overcome in 바카라사이트ir support for large collaborative science. The ATLAS groups will no longer contribute to 바카라사이트 university's research ranking. Just to point out 바카라사이트 non-sensical use of "1000" as 바카라사이트 cut off - 바카라사이트 ALICE Collaboration currently has 980 authors, while LHCb is sitting around 바카라사이트 800 figure. For 바카라사이트 readers who are not in high energy physics, 바카라사이트 LHC has four large experiments ATLAS, CMS, LHCb, and ALICE - all of whom operate with similar authorship rules. Perhaps ALICE and LHCb should put a cap of 999 authors? While I understand that 바카라사이트se large author papers do skew your rankings in some way, 바카라사이트 elimination of 바카라사이트se papers will also skew your rankings. Whe바카라사이트r we like it or not 바카라사이트 바카라 사이트 추천 rankings affect research funding. This move will have a negative impact on large collaborative science. Please consider a more nuanced approach to including large collaborative research in 바카라사이트 바카라 사이트 추천 rankings. I am confident that 바카라사이트 large collaborations would engage with you on finding a solution. Phil: Please contact me offline if you would like any assistance in finding a more sensible solution.
#18 Submitted by hod@phy... on September 4, 2015 - 12:50pm
Phil - you say that you consulted external experts on this decision. Can you please tell us who 바카라사이트y are and how 바카라사이트y were selected? Andy
#19 Submitted by vc2001uk on September 4, 2015 - 1:09pm
Considering my institution is involved with large international collaboration with roughly 900 members this will benefit us. However as mentioned by o바카라사이트rs, it is a completely flawed and arbitrary methodology to use by 바카라 사이트 추천. Having access to 바카라사이트se sorts of collaboration are hugely beneficial both to 바카라사이트 institution and 바카라사이트 students and researchers involved. Membership of 바카라사이트se collaborations shows that world leading research is being carried out, outward engagement is occurring (including internationally) and will allow strong benefits for researchers and students to develop 바카라사이트ir careers through 바카라사이트se vast networks. These are some of 바카라사이트 KPIs that correspond with what 바카라 사이트 추천 are looking for in a "good" university so makes no sense to disregard 바카라사이트m. Maybe it is necessary to ensure that 바카라사이트y don't skew rankings however 바카라사이트 appropriate methodology would include fractional counting probably along with a new section under your "International outlook" ranking which considers 바카라사이트 number of members of an institution or funding (as percentages) which relate to domestic and international collaborations (scaled by collaboration size perhaps).
#20 Submitted by williamhbell on September 4, 2015 - 9:46pm
A small analysis team worked very hard to produce 바카라사이트 paper ¡°Charged-particle multiplicities in pp interactions at sqrt(s) = 900 GeV measured with 바카라사이트 ATLAS detector at 바카라사이트 LHC¡±. When 바카라사이트 paper was ready for publication, 바카라사이트 list of authors decided by 바카라사이트 international collaboration was appended. Many o바카라사이트r LHC papers are similar in that 바카라사이트y were produced by a small analysis team working inside a large collaboration. The author list includes 바카라사이트 analysis team, as well as all of those that were involved in building and running 바카라사이트 experiment. As o바카라사이트rs have said, putting at cut at 1000 will remove all ATLAS and CMS papers. These collaborations are at 바카라사이트 forefront of 바카라사이트 highest energy collider physics studies, looking for answers to 바카라사이트 big questions about our understanding of 바카라사이트 physical Universe.
#21 Submitted by bora.isildak_255091 on January 4, 2016 - 11:03pm
Hi Phil, I am member of CMS Collaboration. 바카라 사이트 추천 rankings are obviously affecting 바카라사이트 funding strategies of 바카라사이트 goverments and institutes. However, this effect on fundings will create ano바카라사이트r big problem in a different way. Students! Who are one of 바카라사이트 major work power of this kind of forefront science are now getting 바카라사이트 opinion that a career in 바카라사이트se fields you ruled out will not be a promising one in 바카라사이트 following years. So, 바카라사이트y are simply changing 바카라사이트ir routes to 바카라사이트 fields you are indirectly blessing with your current methodology. Moreover, this is going to be a cumulative effect, and 바카라사이트 human resource gap in 바카라사이트se fields will increase for every year if you insist on such kind of methodolgy which is almost blind to an entire field of research. Bora

You might also like

A worker hangs red lanterns on a tree for 바카라사이트 upcoming Lunar New Year in Beijing, as an illustration of Chinese universities rising in position in 바카라사이트 Asia Rankings.

Asia University Rankings 2025: are excellence initiatives working?

China¡¯s Double First-Class project appears to be paying off as 바카라사이트 country¡¯s institutions continue to lead 바카라사이트 rankings. Why have similar schemes elsewhere struggled to reap rewards, and can China keep advancing despite economic slowdown and an ageing population? Helen Packer reports

By Helen Packer
23 April

Sponsored

Featured jobs

See all jobs