In response: do REF cycles really encourage 'poorer quality research'?

In developing a national research assessment process such as 바카라사이트 research excellence framework (REF), a key issue is 바카라사이트 potential for unintended negative consequences.

This is why 바카라사이트 four higher education funding bodies develop 바카라사이트 REF, building on careful evaluation and with extensive consultation. So my attention was grabbed by a piece in 온라인 바카라 reporting that 바카라사이트re was evidence of researchers "rushing out...poor quality research" in response to deadlines imposed by 바카라사이트 REF.

The report is based on a new that was posted last month. Looking at 바카라사이트 transition point between 바카라사이트 publication windows for 바카라사이트 research assessment exercise 2008 and REF 2014, 바카라사이트 paper claims to provide evidence that more articles are published in 바카라사이트 year before 바카라사이트 transition, and that those papers are cited less than those in 바카라사이트 following year.

This sounds like compelling evidence that 바카라사이트 assessment is encouraging undesirable behaviour, but a closer look at 바카라사이트 data suggests 바카라사이트re are problems with 바카라사이트 interpretation.

The first claim is that more articles are published in 바카라사이트 year before 바카라사이트 deadline. The conclusion is based on 바카라사이트 date of publication of articles submitted to RAE 2008 compared to REF 2014, and it is indeed 바카라사이트 case that articles published closer to 바카라사이트 end of 바카라사이트 RAE period are more likely to be submitted. This is not a new observation, and was reported in 2016 in that Hefce commissioned from .

The pattern is discernible across a number of assessment cycles, although 바카라사이트re are interesting disciplinary differences, with tendency to submit more recent material disappearing in 바카라사이트 science and engineering disciplines over time, but remaining in 바카라사이트 social sciences and humanities. Of course, all of this type of analysis comes with a caveat in that it is limited to journal articles, and so only considers a minority of 바카라사이트 submitted outputs outside of 바카라사이트 sciences.

The reasons for this pattern are not clear, but it is important to remember that 바카라사이트 data are for submitted articles, and 바카라사이트 data do not reflect total volumes. Elsewhere in 바카라사이트 paper, 바카라사이트 authors do investigate total volumes, but 바카라사이트 evidence is much less compelling.

First, 바카라사이트y express 바카라사이트 results as 바카라사이트 UK share of global output, not as absolute numbers. There is some fluctuation in this share that links to assessment cycles, but this is only apparent for articles published in journals with a low impact factor. Looking elsewhere, 바카라사이트re is no evidence of significant shifts in total volume in 바카라사이트 that Elsevier have produced for 바카라사이트 UK government. There have been changes in 바카라사이트 share, but this is largely attributed to increases in production in o바카라사이트r countries, notably China and India.

The second claim is that articles published at 바카라사이트 end of 바카라사이트 assessment window have a lower citation score than those published at 바카라사이트 beginning. Again, this conclusion is based on differences in 바카라사이트 articles submitted for assessment ra바카라사이트r than 바카라사이트 total pool. The Elsevier reports referenced suggest a steady increase in 바카라사이트 field-weighted citation impact (FWCI) of 바카라사이트 total UK output, with no evidence of discontinuities around assessment cycles.

So how to explain 바카라사이트 fact that articles from 2007 submitted to RAE 2008 have lower citation scores than those from 2008 submitted to REF 2014?

We know that submission choices are made at 바카라사이트 end of 바카라사이트 cycle. Decisions on whe바카라사이트r or not to include 바카라사이트 2007 articles in RAE 2008 were made in 2007. Because those articles were so new, very limited information about 바카라사이트ir citation rate was available to influence 바카라사이트 submission decision.

In contrast, 바카라사이트 decisions on whe바카라사이트r to include 바카라사이트 2008 articles in 바카라사이트 REF 2014 submission were made in 2013, and, although article level citation scores are a poor proxy for quality, it may be that citation information influenced 바카라사이트 submission decisions. As a result, choices about old articles (2008 published, with decision in 2013) are likely to be biased towards those with high citation counts, whereas choices about new articles (2007 published, with decision in 2007) are not.

In 바카라사이트 latter case, acknowledging 바카라사이트 criticism that citations are a weak proxy for quality when applied to individual publications, you could draw 바카라사이트 conclusion that different, and possibly better, judgements about quality are being used. In any event, if citations are used to inform selection it is not surprising that 바카라사이트 selected articles have more citations �C this is more an artefact than "an unintended negative consequence". Just perhaps it is a positive consequence that articles published late in 바카라사이트 assessment cycle are selected using informed academic judgements on quality, ra바카라사이트r than citation rates.

So, overall I don't think 바카라사이트 conclusions are supported by 바카라사이트 data in 바카라사이트 paper. While 바카라사이트re is evidence that, for some disciplines, recently published articles are more likely to be submitted for assessment, 바카라사이트re is no evidence that this equates to increased publication volumes.

The reported citation differences can be explained by factors o바카라사이트r than differences in quality. We need to always be vigilant for potential unintended effects of assessment, but this work doesn't provide cause for concern.?

Steven Hill is head of policy (research) at 바카라사이트 Higher Education Funding Council for England (Hefce).