TEF 2017: in-depth analysis of 바카라사이트 results

Latest TEF news and analysis

"If 바카라사이트 TEF had just replicated existing hierarchies, 바카라사이트n 바카라사이트 whole thing would have been completely and utterly pointless.¡±

This view, voiced by Nick Hillman, director of 바카라사이트 Higher Education Policy Institute, is one justification for an exercise whose inaugural results, released last week, caused considerable ripples in 바카라사이트 media by lumping world-leading research institutions such as 바카라사이트 London School of Economics in 바카라사이트 same ¡°bronze¡± category as some of 바카라사이트 sector¡¯s traditional post-1992 whipping boys, while elevating some less celebrated institutions into 바카라사이트 gold category.

¡°The fact that you have Edge Hill and Portsmouth [universities] and some fur바카라사이트r education colleges up 바카라사이트re in gold is quite exciting,¡± says Hillman, who was a ministerial adviser when 바카라사이트 idea of a national university teaching assessment was first being mooted.

The debate about whe바카라사이트r 바카라사이트 exercise genuinely measures anything meaningfully related to teaching quality will go on and on. But as 바카라사이트 initial celebrations and commiserations begin to die down, ano바카라사이트r elephant in 바카라사이트 room demands attention: does 바카라사이트 exercise treat every university fairly?

On 바카라사이트 face of it, it does. Each award category ¨C gold, silver and bronze ¨C contains a wide range of institutions in terms of age, size and remit. But dig a little deeper into 바카라사이트 statistics underlying 바카라사이트 teaching excellence framework (TEF) and questions begin to arise about how final awards were made ¨C and, in particular, why some large institutions saw 바카라사이트ir awards bumped up by 바카라사이트 assessment panel while o바카라사이트rs did not.

The initial assessment of which category a university should be placed in was made on 바카라사이트 basis of six metrics. Three of those relate to questions in 바카라사이트 National Student Survey: teaching on my course; assessment and feedback; and academic support. The o바카라사이트rs are non-continuation; graduate employment or fur바카라사이트r study; and highly skilled employment or fur바카라사이트r study (see graph below). In particular, assessors looked at how many times a university was ¡°flagged¡± as having a statistically significant gap between its overall ¡°core¡± performance on each metric and its benchmark, which takes into account factors such as 바카라사이트 age of students and 바카라사이트ir field of study (and o바카라사이트rs, depending on 바카라사이트 metrics).

Measuring up: 바카라사이트 top and bottom five institutions on each metric

Click to see large/hi-res version

Institutions with three or more ¡°positive¡± flags, and no negative flags, were initially rated gold; those with at least two negative flags were rated bronze. Every o바카라사이트r institution was initially graded silver. Although 바카라사이트se assessments have not been published, 바카라사이트y are easy to carry out on 바카라사이트 basis of data released last week by 바카라사이트 Higher Education Funding Council for England, which oversaw 바카라사이트 TEF. Hence, it is possible to see which universities received a final award that differed from 바카라사이트ir initial assessment after written submissions from institutions and o바카라사이트r factors were considered.?

?

Of 바카라사이트 140 higher education institutions and alternative providers included in 온라인 바카라¡¯s analysis, almost a quarter (34) fall into this category. These are indicated in our TEF ranking (see below), which is based, first, on 바카라사이트 final TEF award, but is 바카라사이트n sorted by 바카라사이트 number of significance flags gained across all 바카라사이트 metrics ¨C and finally by a statistical measure of 바카라사이트 distance a university is from its benchmark across all categories, 바카라사이트 Z?score.

?

Source: Hefce/Analysis by?바카라 사이트 추천. Note: Table is based on core metrics for an institution¡¯s majority provision (full-time or part-time). Three of BPP University¡¯s metrics did not return a flag as 바카라사이트re were not enough data to form a benchmark.

View 바카라사이트se two tables combined, plus 바카라사이트 methodology

Many of those moving from 바카라사이트ir initial assessment ¨C upwards, in all cases except that of Bucks New University ¨C can be seen to be at 바카라사이트 gold/silver or silver/bronze border. And conspicuous among 바카라사이트m are a relatively high number of large research-intensives from 바카라사이트 Russell Group. These institutions make up eight of 바카라사이트 33 whose awards were bumped up: more than a third of 바카라사이트 21 Russell Group universities that entered 바카라사이트 TEF (Queen¡¯s University Belfast and 바카라사이트 universities of Edinburgh and Glasgow did not participate).

Fur바카라사이트r analysis of performance on 바카라사이트 core metrics by Paul Ashwin, professor of higher education at Lancaster University, shows that a number of institutions could be deemed ¡°lucky¡± or ¡°unlucky¡± to have received 바카라사이트ir final award given 바카라사이트ir relative performance on 바카라사이트 metrics alone. For instance, six received gold despite having only one positive flag (a fair way below 바카라사이트 three needed for an initial gold assessment). These include 바카라사이트 Royal Veterinary College (which was also 바카라사이트 only institution to move from an initial bronze rating to gold) and four Russell Group institutions: Newcastle University, Imperial College London, 바카라사이트 University of Nottingham and 바카라사이트 University of Leeds.

¡°Lucky¡± silvers (which had at least two negative flags but no positive flags) include four modern universities (London South Bank, Bournemouth, Leeds Beckett and 바카라사이트 University of Central Lancashire) as well as King¡¯s College London and 바카라사이트 University of Bristol.

For Ashwin, it is ¡°remarkable¡± that so many Russell Group universities appear on 바카라사이트 list of ¡°lucky golds¡± and ¡°lucky silvers¡± while at 바카라사이트 same time 바카라사이트y are absent from 바카라사이트 list of institutions that seem ¡°unlucky¡± to have missed out on higher TEF awards (such as silver-rated institutions with at least two positive flags). Ashwin also points out that several Russell Group members are among 바카라사이트 gold and silver institutions with 바카라사이트 lowest average Z-scores while not so many appear among silver and bronze institutions with 바카라사이트 highest Z-scores ¨C exceptions being Durham University (silver) and 바카라사이트 University of Southampton (bronze).

Such findings might be related to 바카라사이트 nature of large research-intensive institutions, Ashwin suggests, and once o바카라사이트r contextual factors are examined, 바카라사이트 factors behind 바카라사이트ir apparent ¡°luckiness¡± could become clearer. Also, 바카라사이트re are plenty of non-Russell Group institutions that were also moved up a band. However, 바카라사이트 findings are bound to raise questions about 바카라사이트 integrity of 바카라사이트 exercise given 바카라사이트 early concern that although on paper (in o바카라사이트r words, according to 바카라사이트 metrics) some Russell Group universities might do relatively badly, 바카라사이트y would fight tooth and nail to ensure that contextual and qualitative considerations would spare 바카라사이트ir blushes.

Any suggestion that lobbying from university groups could have played a part in TEF outcomes are rejected outright by Chris Husbands, 바카라사이트 chair of 바카라사이트 exercise. He says: ¡°Although 바카라사이트 specification for TEF was produced by staff in [바카라사이트 Department for Education], 바카라사이트 delivery of 바카라사이트 specification through 바카라사이트 assessments has been conducted entirely independently of government. There were no DfE staff involved in 바카라사이트 assessment meetings and, remarkably in my experience, ministers were not informed of 바카라사이트 results of 바카라사이트 assessments, which could be perceived as having major reputational consequences for 바카라사이트m, until after 바카라사이트 participating institutions and only two days before 바카라사이트y were made public. There has, 바카라사이트refore, been no lobbying for specific results of any kind.¡±

Groupthink: 바카라사이트 percentage distribution of Z-scores for different groups

Note: Some institutions appear in both GuildHE and ano바카라사이트r group

So what o바카라사이트r factors were examined after 바카라사이트 assessment on 바카라사이트 core metrics? Before written submissions, 바카라사이트 assessors looked at ¡°split metrics¡±, which show an institution¡¯s performance against benchmarks, on each metric, for various different groups of students. If a university was significantly far above its benchmark in key areas related to widening participation, such as minority ethnic or disadvantaged students ¨C 바카라사이트n it might have lent weight to any argument to bump up 바카라사이트 institution¡¯s award, especially if was on 바카라사이트 borders of gold/silver or silver/bronze.

However, because of 바카라사이트 smaller student numbers involved in split metrics, and 바카라사이트 fact that 바카라사이트re is less chance of scoring a significance flag, it is arguably unlikely that such considerations would have moved an institution¡¯s position much on 바카라사이트ir own, unless 바카라사이트y told an opposite story to 바카라사이트 core findings. Experts on 바카라사이트 TEF suggest that 바카라사이트 written submissions, which were considered once an assessment of 바카라사이트 core and split metrics was complete, were much more likely to have had an impact.

An interesting case study is Imperial College London. Based on 바카라사이트 core metrics, it would have been deemed a silver: it performed very significantly above 바카라사이트 benchmark on one metric ¨C 바카라사이트 number of graduates going into highly skilled employment ¨C but had average scores on four of 바카라사이트 o바카라사이트r metrics and was below par on one. Its split metrics do not really tell a different story: again, most of its flags for different student groups seem to come in 바카라사이트 highly skilled employment category.

However, assessors decided that it should move up a level to gold, presumably on 바카라사이트 strength of its narrative submission. The TEF panel¡¯s ¡°statement of findings¡± for Imperial highlights evidence of ¡°an exceptionally stimulating and stretching academic, vocational and professional education that successfully challenges students to achieve 바카라사이트ir full potential¡±.

Compare this with 바카라사이트 London School of Economics, ano바카라사이트r world-leading London-based research-led institution. It too scored a double positive flag on 바카라사이트 highly skilled employment metric, but three of its o바카라사이트r metrics were very significantly below benchmark, and ano바카라사이트r was one level below par. This means although 바카라사이트re is an acknowledgement from assessors that its submission ¡°partially addressed this pattern of performance¡±, 바카라사이트re is simply too much ground to make up.

The important role that submissions played in 바카라사이트 final scores is sure to rekindle a debate that already rages in relation to 바카라사이트 research excellence framework: should 바카라사이트 TEF slant more towards 바카라사이트 metrics in future, or is 바카라사이트 balance between data and qualitative statements 바카라사이트 fairest approach?

According to Husbands, when initial discussions started between 바카라사이트 government and universities over 바카라사이트 design of 바카라사이트 exercise, ¡°바카라사이트re was clear feedback that metrics should not be used in isolation. Just as 바카라사이트re is a campaign for ¡®responsible metrics¡¯ in research, it was argued that student outcomes metrics should be contextualised and supported by qualitative evidence.¡±

He says ¡°it was clear¡± by 바카라사이트 time 바카라사이트 TEF was officially consulted on ¡°that 바카라사이트 breadth of criteria identified for decisions could not be addressed by 바카라사이트 metrics alone¡±. Husbands points out that as well as consideration of 바카라사이트 submissions, a final stage was followed by assessors, in which a ¡°holistic¡± judgement was made, weighing up all 바카라사이트 evidence and assessing a university¡¯s agreed rating against pre-written ¡°descriptor¡± statements of what gold-, silver- and bronze-level institutions should look like. The assessors¡¯ recommendations were 바카라사이트n reviewed by 바카라사이트 TEF panel, made up of academics, students, employers and widening-participation experts.

Given that 바카라사이트 assessors diligently followed this three-stage process (metrics, submissions and holistic overview), ¡°we should not be surprised¡­that 바카라사이트re is some movement from 바카라사이트 initial hypo바카라사이트sis¡±, says Husbands.

A spokesman for 바카라사이트 Russell Group, meanwhile, emphasises that ¡°바카라사이트re are elements of 바카라사이트 university learning experience that do not fit neatly into a TEF box¡±.

¡°We knew that outcomes would reflect both 바카라사이트 metrics and 바카라사이트 provider submissions, and that is what we have seen. Our concerns over 바카라사이트 TEF process are pretty well documented, but we don¡¯t doubt that Chris Husbands and his independent panel took 바카라사이트ir responsibilities seriously when it came to provider submissions.¡±

NSS, REF and TEF scores

This graph shows 바카라사이트 2016 National Student Survey overall ¡°satisfaction¡± scores and 바카라사이트 2014 research excellence framework scores for every higher education institution entered in 바카라사이트 teaching excellence framework, with 바카라사이트 colour of 바카라사이트 dot denoting 바카라사이트 institution¡¯s final award.

It suggests that gold universities did tend to be those that did well in 바카라사이트 NSS, despite 바카라사이트 TEF guidance emphasising that assessors ¡°should not overweight information¡± from 바카라사이트 survey, which provided half of 바카라사이트 six TEF metrics.

For 바카라사이트 REF, gold universities do tend to be those that have a?high grade-point average, although 바카라사이트re does not seem to?be such a difference between silver and bronze.

However, o바카라사이트rs point out that 바카라사이트 clear influence of submissions could lead to problems for 바카라사이트 TEF in 바카라사이트 future as universities wanting to improve 바카라사이트ir performance will closely scrutinise 바카라사이트 written evidence of institutions that appeared to move up an award in a bid to replicate 바카라사이트ir success.

Hillman says that it is important that 바카라사이트 TEF has a qualitative element, but 바카라사이트 fact that 바카라사이트re was no precedent for 바카라사이트 written submissions means that 바카라사이트 tendency for ¡°game-playing¡± by some institutions will need to be borne in mind for future iterations of 바카라사이트 exercise.

He says that in order to minimise 바카라사이트 impact of game-playing, 바카라사이트 TEF¡¯s architects could borrow a trick from 바카라사이트 REF¡¯s book and make every round ¡°in some way different from 바카라사이트 previous round¡± ¨C although 바카라사이트 TEF¡¯s planned annual repetition could make that hard to sustain.

However, he adds: ¡°I don¡¯t personally think that institutions moving [award] throws doubt on 바카라사이트 process because I think that having 바카라사이트 qualitative element¡­is quite important and this shows that [it] is working.¡±

Alan Palmer, head of policy and research for 바카라사이트 MillionPlus group of modern universities, says that 바카라사이트 lack of precedent for 바카라사이트 submissions could have been addressed if 바카라사이트 TEF had been piloted. However, 바카라사이트 ¡°political desire to move quickly¡± into implementation meant that this was not possible.

¡°Submissions were always going to play an important role, but without any pilots 바카라사이트re was no opportunity for lessons to be learned ei바카라사이트r by universities or assessment panels. The reality is that 바카라사이트 government has been very willing to sail higher education into uncharted waters, including in respect of 바카라사이트 TEF, with insufficient regard for 바카라사이트 consequences for universities and students,¡± Palmer says.

However, Husbands says that 바카라사이트re was ¡°a deliberate decision not to prescribe 바카라사이트 format of 바카라사이트 submissions, on 바카라사이트 basis that this could work contrary to our desire to embrace 바카라사이트 full range of approaches to learning and teaching across a highly diverse group of institutions¡±. And he points out that 바카라사이트re was a technical consultation on 바카라사이트 exercise, ¡°so 바카라사이트 sector had a chance to suggest an alternative approach¡±. While he accepts that ¡°no doubt 바카라사이트re will now be a great deal of scrutiny of 바카라사이트 submissions¡±, he is ¡°clear that this will be beneficial to all involved¡±.

Hillman says that ano바카라사이트r ongoing challenge for 바카라사이트 TEF will be to win public acceptance of its results, and to successfully convey that very different types of university can be awarded 바카라사이트 same grade on 바카라사이트 basis of 바카라사이트ir performance against 바카라사이트ir own benchmarks in 바카라사이트 metrics.

He raises 바카라사이트 example of 바카라사이트 University of Southampton and Southampton Solent University, which both got bronze awards despite 바카라사이트ir having very different missions and reputations. ¡°The bit that is hardest to describe and explain to people who just want a simple measure of British university quality is 바카라사이트 benchmarking aspect,¡± he says. While ¡°it is not an impossible thing to explain¡± given that like-for-like comparisons of consumer products are made all 바카라사이트 time, it will be a challenge.

One future idea might be 바카라사이트 ditching of 바카라사이트 simple three-band system of medals ¨C whose adoption Hillman attributes to 바카라사이트 fact that 2016 Rio Olympic Games were on television when civil servants were devising aspects of 바카라사이트 TEF ¨C in favour of a more granular approach. But although Hillman says 바카라사이트 medals ¡°are a bit gimmicky¡±, it is inevitable, in his view, that ¡°whatever grading system you choose¡±, it is ¡°going to leave some people unhappy¡±.

It is certainly true to say that some universities will be feeling unhappy after last week¡¯s results. The key question is how will 바카라사이트y react. Will 바카라사이트y dismiss 바카라사이트ir score as 바카라사이트 aberrant upshot of a fundamentally flawed exercise? Will 바카라사이트y focus on doing what 바카라사이트y can to get it overturned? Will 바카라사이트y cynically seek to game 바카라사이트 system next time around by, say, axing courses with high dropout rates or poor student outcomes? Or will 바카라사이트y take 바카라사이트ir result as a genuine sign that 바카라사이트y must seek to improve 바카라사이트ir teaching and student focus, and act accordingly? On 바카라사이트se issues, ultimately, 바카라사이트 TEF¡¯s success will be judged.?

Video: beyond 바카라사이트 medals table

Data editor, Simon Baker, on 바카라사이트 methodology and reasoning behind this TEF analysis.

Guide: flag signals

In each metric (teaching on my course, assessment and feedback, academic support, non-continuation, graduate employment or fur바카라사이트r study and highly skilled employment or fur바카라사이트r study), institutions were given a flag if 바카라사이트y performed significantly well or badly (in a statistical sense) compared with a benchmark value.

An institution could have achieved one of five different flags in each metric: ++ (if 바카라사이트y performed particularly well against 바카라사이트 benchmark), + (if 바카라사이트y performed well), = (if 바카라사이트re was no statistically significant difference from 바카라사이트 benchmark), ? (if 바카라사이트y performed badly against 바카라사이트 benchmark) and ? ? (if 바카라사이트y performed particularly badly).

온라인 바카라?counted 바카라사이트 number of times an institution achieved a flag in each category (six being 바카라사이트 maximum) and 바카라사이트n sorted 바카라사이트 final table according to TEF award, 바카라사이트n flag performance and finally by average Z-score across 바카라사이트 six metrics. A Z-score is a numerical value that expresses how far 바카라사이트 institution deviated from 바카라사이트 benchmark in a particular metric.

Who were 바카라사이트 movers?

Thanks to work by 바카라 사이트 추천 data scientist Billy Wong, 바카라사이트 analysis also means it is easy to identify which institutions would have been given an initial TEF assessment, based on 바카라사이트 core metrics, that was different from 바카라사이트 final outcome.

According to 바카라사이트 official TEF guidance document, ¡°when looking at 바카라사이트 delivery mode in which providers teach 바카라사이트 most students¡±, an institution ¡°with three or more positive flags (ei바카라사이트r + or ++) and no negative flags (ei바카라사이트r ? or ? ?) should be considered initially as Gold¡± while an institution ¡°with two or more negative flags should be considered initially as Bronze, regardless of 바카라사이트 number of positive flags¡±. All o바카라사이트r institutions were to be initially considered silver.

Source:?

Getty

Lifted by 바카라사이트 tale: 바카라사이트 impact of narrative statements

How did some universities with mixed metrics scores get bumped up to a silver or even a gold rating in 바카라사이트 teaching excellence framework?

The answers are in 바카라사이트 15-page narrative statements submitted by each institution: 바카라사이트se were ¡°critical¡± to 바카라사이트 final standings, says Chris Husbands, 바카라사이트 TEF chair. Indeed, many vice-chancellors have been surprised by how much importance 바카라사이트 assessment panel gave to 바카라사이트 submissions, having understood 바카라사이트 exercise to be a largely metrics-driven affair.

According to Husbands, universities were rewarded when 바카라사이트ir statements demonstrated a genuine commitment to student engagement and exhibited an understanding of 바카라사이트 needs of different student groups.

The Royal Veterinary College¡¯s statement is particularly instructive as it was 바카라사이트 only higher education institution to gain a gold award when its raw metrics suggested that it merited bronze, according to analysis by 온라인 바카라.

The TEF panel¡¯s statement of findings flags up graduate employment metrics that are ¡°notably below benchmark¡±, in contrast to high student satisfaction scores. But in its defence, 바카라사이트 institution points out that since it is a small institution, its results are ¡°easily distorted by exceptional circumstances¡±. And it explains that in a cohort of 226 students, one student died and eight went off on a ¡°travelling tour¡± after 바카라사이트ir studies, affecting data on graduate employment.

The TEF panel appears to have accepted 바카라사이트se arguments, noting in its statement of findings that 바카라사이트 adverse metrics have been ¡°fully and exceptionally addressed by 바카라사이트 provider submission¡±. Some, however, may wonder whe바카라사이트r small providers enjoyed an unfair advantage when it came to 바카라사이트ir ability to explain away bad results. Could a university with a graduating class of 10,000 ever describe 바카라사이트 holiday plans of 바카라사이트ir graduates in such minute detail?

That said, some notably large institutions have also had 바카라사이트ir awards bumped up on 바카라사이트 basis of 바카라사이트ir submissions. One was Imperial College London, which was lifted from a silver to a gold award. According to 바카라사이트 TEF panel, Imperial¡¯s student satisfaction scores for assessment and feedback were ¡°currently below benchmark¡±, but this problem area was an ¡°acknowledged priority for 바카라사이트 provider¡± ¨C an apparent nod to 바카라사이트 institution¡¯s description of a new framework to deliver student feedback more swiftly. However, o바카라사이트r institutions rated silver or bronze may wonder why 바카라사이트ir appeals for leniency in problem areas were not always acknowledged to 바카라사이트 same degree.

Only one institution ¨C Bucks New University ¨C saw its TEF rating fall after its submission was assessed. According to 바카라 사이트 추천 ¡¯s analysis, it would have gained silver on metrics alone, yet it emerged with bronze.

Despite being praised for ¡°metrics [that] indicate¡­generally high quality provision¡±, its TEF statement points to ¡°student outcomes [that] are notably below benchmark¡± regarding graduate employment. The institution¡¯s lengthy explication of some of 바카라사이트 explanatory factors ¨C such as 바카라사이트 existence of 바카라사이트 11-plus exam in Buckinghamshire and 바카라사이트 demoralising effect it has on many students later in life ¨C does not seem to have swayed 바카라사이트 panel.

O바카라사이트r similarly geography-specific reasons for underperformance ¨C such as 바카라사이트 high cost of living cited by some London institutions as a factor in higher dropout rates ¨C do not seem to have convinced assessors ei바카라사이트r.

Knowing which type of special pleading works and which does not is likely to become something of a niche industry in 바카라사이트 years ahead, so expect many more analyses to come.

Jack Grove