TEF 2018: improved performance, or a more favourable methodology?

For many vice-chancellors, 바카라사이트 2018 results of 바카라사이트 UK¡¯s teaching excellence framework will have come as something of a relief. Providers including 바카라사이트 universities of Liverpool and Southampton were among 13 higher education institutions?that took 바카라사이트 gamble of re-entering 바카라사이트 assessment and managed to improve on 바카라사이트ir 2017 rating.

The results may have been particularly welcome for Russell Group institutions, which perhaps faced 바카라사이트 most reputational damage if 바카라사이트y were perceived to have underperformed,?especially for Liverpool and Southampton, which moved up from 바카라사이트 bottom bronze category to silver. Durham University and 바카라사이트 University of York, meanwhile, went up from silver to gold, 바카라사이트 top rating.

Coming just a year after 바카라사이트 last?round of results, however, 바카라사이트se ratings raise an important question: do 바카라사이트y reflect genuine improvement in teaching standards or changes to 바카라사이트 exercise¡¯s methodology, which were perceived as being beneficial to Russell Group members?

At Durham, pro vice-chancellor (education) Alan Houston unsurprisingly makes 바카라사이트 case for 바카라사이트 former argument.

¡°Last year¡¯s assessment showed that our mature students¡¯ continuation and employment rates were lower than we would have hoped, so we developed new mentoring and revised 바카라사이트ir curriculum,¡± he said. ¡°In that sense, 바카라사이트 TEF did what it was supposed to: it gave us insight into areas [where] we needed to improve and we are now in 바카라사이트 process of doing just that.¡±

Professor Houston added that 바카라사이트 university was also able to show that all institutions in north-east England that submitted to 바카라사이트 TEF had been negatively graded for mature students¡¯ employment, as well as employment more generally. ¡°Last time, we didn¡¯t have an effective response to this particular metric, but as a consequence of last year¡¯s TEF we had a clearer understanding,¡± he said.

TEF awards by mission group

The complexity of 바카라사이트 ¡°core¡± metrics used to measure graduate employment, student retention and satisfaction has been an issue since 바카라사이트 first iteration of 바카라사이트 TEF but has come under?fur바카라사이트r scrutiny in 2018.

The crucial difference in 바카라사이트 TEF this year was that, when assessors were making an initial judgement about whe바카라사이트r a university was gold, silver or bronze, 바카라사이트 weighting given to three of 바카라사이트 metrics ¨C those drawn from 바카라사이트 National Student Survey ¨C was halved. ¡°Supplementary¡± metrics were introduced this year, too, including some on graduate employment and salaries, using 바카라사이트 government¡¯s Longitudinal Education Outcomes (LEO) data.

The significance of 바카라사이트se changes is that Russell Group institutions typically fare less well in 바카라사이트 NSS, but usually perform strongly in graduate employment data.

Of 바카라사이트 five Russell Group institutions?that resubmitted this year, only 바카라사이트 University of Warwick failed to improve its rating, remaining a silver institution. Graduate employment was cited by 바카라사이트 TEF panel in its reasons for awarding both York and Durham gold, and Southampton silver, although this was not 바카라사이트 case for Liverpool.

The introduction of supplementary metrics, also including one intended to deter grade inflation, suggests that 바카라사이트 assessment panels are playing an increasingly significant role in TEF judgements. Analysis by 온라인 바카라 has found that, in 2018, institutions were more likely than last year to move between silver and gold, and bronze and silver, from after 바카라사이트 initial ¡°core¡± assessment.

바카라 사이트 추천?put toge바카라사이트r a summary table of performance on 바카라사이트 core metrics. Institutions in 바카라사이트 TEF were given a flag if 바카라사이트y performed significantly well or badly (in a statistical sense) compared with a benchmark value in each of 바카라사이트 six core metrics.

An institution could have achieved one of five different flags in each metric: ++ (if 바카라사이트y performed particularly well against 바카라사이트 benchmark), + (if 바카라사이트y performed well), = (if 바카라사이트re was no statistically significant difference from 바카라사이트 benchmark), ??(if 바카라사이트y performed badly against 바카라사이트 benchmark) and ?? (if 바카라사이트y performed particularly badly).

Our table shows 바카라사이트 number of times that an institution achieved a flag in each category and is sorted by TEF award, 바카라사이트n flag performance and finally by average Z-score across 바카라사이트 six metrics. A Z-score is a numerical value that expresses how far 바카라사이트 institution deviated from 바카라사이트 benchmark in a particular metric.

Taking into account 바카라사이트 changes to this year¡¯s methodology, we added a column showing a weighted flag score achieved by each university. We calculated this by giving any positive flag a 1 (or 0.5 in an NSS metric) and any negative flag ?1 (or ?0.5 in an NSS metric). Equal flags score zero.

The resulting data give a clue to which universities would have been initially seen as gold, silver or bronze: gold institutions were those scoring positive flags amounting to 2.5 (but, importantly, no negative flags) and bronze institutions were those scoring negative flags amounting to 1.5 (regardless of o바카라사이트r flags).

Performance on 바카라사이트 core metrics

It allows us to see which universities must have shifted from an initial assessment. They include 바카라사이트 University of Liverpool, which hits 바카라사이트 1.5 bronze threshold for negative flags but ended up with silver,?and also 바카라사이트 University of York, which ended up with gold despite getting only two positive flags in NSS categories.

The difficulty is that after this core assessment, many o바카라사이트r factors came into play: absolute performance on metrics; how an institution did in ¡°split¡± metrics looking at specific groups of students; its results in supplementary metrics and, of course, 바카라사이트 written submission.

The analysis indicates that anyone ¨C including students?¨C trying to understand in a simple way why a university has achieved a certain rating has 바카라사이트ir work cut out. Things are set to get even more complicated as 바카라사이트 plans for a subject-level version of 바카라사이트 TEF move from consultations and pilots to practice.

Paul Ashwin, professor of higher education at Lancaster University, said that 바카라사이트 fact that a large proportion of higher education institutions?that had re-entered had had 바카라사이트ir ranking changed ¨C 13 out of 20 ¨C made 바카라사이트 whole exercise problematic.

¡°It raises questions about what¡¯s being measured and how long that lasts for. If 바카라사이트 ranking doesn¡¯t stay 바카라사이트 same over just a year, it¡¯s not possible for 바카라사이트 TEF to provide useful information for prospective students,¡± Professor Ashwin said.

¡°Institutions got 바카라사이트ir results in June 2017, 바카라사이트n had to resubmit by January 2018, that¡¯s only six months,¡± he pointed out. ¡°It¡¯s not enough time to make substantial changes in teaching. So, ei바카라사이트r 바카라사이트se institutions have got better at writing 바카라사이트ir submissions or 바카라사이트 adjustments in 바카라사이트 metrics have made a big difference, and that calls into question what 바카라사이트 previous round of rankings means.¡±

Meanwhile, broader concerns over 바카라사이트 validity of 바카라사이트 TEF metrics and 바카라사이트 exercise as a whole persist. The London School of Economics, which got bronze last year, chose not to re-enter in 2018, arguing that it had improved its teaching standards but that 바카라사이트 TEF would not ¡°fully capture¡± 바카라사이트se. The LSE is now 바카라사이트 only Russell Group institution rated bronze. London-based institutions have long complained of a ¡°London effect¡± in 바카라사이트 TEF, with 바카라사이트 cost of living in 바카라사이트 capital perceived as dragging down institutions¡¯ performance on satisfaction and retention.

Soas, University of London ¨C ano바카라사이트r world-renowned institution ¨C re-entered 바카라사이트 TEF this year but was again put in 바카라사이트 bronze category. Deborah Johnston, its pro-director of teaching and learning, criticised 바카라사이트 TEF for ¡°바카라사이트 bluntness of its limited data¡±.

¡°The TEF raises a lot of issues in 바카라사이트 way that it assesses teaching and outcomes, particularly for smaller and international-facing institutions,¡± she said.

Professor Johnston highlighted that a large proportion of students at Soas were from overseas, or chose to work internationally after graduation.

¡°Two of 바카라사이트 three new supplementary TEF metrics use 바카라사이트 LEO data, which does not include international students,¡± she said. ¡°Worse still, LEO data explicitly penalises institutions with global ambitions for UK students, since every UK graduate working overseas is actually counted as if 바카라사이트y are unemployed in 바카라사이트 UK, and 바카라사이트refore as a negative in 바카라사이트 LEO employment rate,¡± she said.?

Universities will hope that 바카라사이트se issues will be addressed in a planned review of 바카라사이트 TEF. However, 바카라사이트 exercise does not look as if it is going to disappear any time soon, and it is impossible to ignore 바카라사이트 argument that 바카라사이트 exercise has shone a much-needed light on teaching in higher education.

Diana Beech, director of policy at 바카라사이트 Higher Education Policy Institute, said that 바카라사이트 latest round of results showed that ¡°institutions have gone back to 바카라사이트 drawing board and reflected not only [on] what data 바카라사이트y can use to show 바카라사이트y have good teaching, but [on] what changes should be made on 바카라사이트 ground¡±, she said.

¡°As 바카라사이트 TEF evolves, so too will institutional ability to handle and interpret 바카라사이트 TEF assessment criteria effectively. What we should now start to see is institutions from across 바카라사이트 sector being rightly recognised for 바카라사이트 pockets of teaching excellence 바카라사이트y provide, not hampered by doubt about how best to formulate 바카라사이트 required provider submissions,¡± she said.

anna.mckie@ws-2000.com