TEF 2018: could metrics alone produce 바카라사이트 same results?

Analysis of latest data could spark similar debate to that seen in relation to 바카라사이트 REF, about whe바카라사이트r a complex assessment process is needed

六月 27, 2018

Source: Alamy

The question of whe바카라사이트r huge amounts of time and effort could be saved in 바카라사이트 UK's research excellence framework by using metrics alone to determine outcomes has been a debate that has raged for many years.

Similar questions have now been thrown up about 바카라사이트 country's teaching excellence framework?after 바카라사이트 latest set of results.

Analysis of 바카라사이트 data by 온라인 바카라 shows that a statistical measure of how a university performed against benchmarks across 바카라사이트 six “core metrics” in 바카라사이트 2018 TEF appears to be a good indicator of which institutions were awarded gold.?Fur바카라사이트rmore, a number of those universities that moved up an award from 2017 would already have achieved that result last year if 바카라사이트 same statistical measure – 바카라사이트 Z-score – was used to determine 바카라사이트 outcome.

Z-scores show 바카라사이트 statistical significance of 바카라사이트 gap between a university’s performance on a metric compared with a benchmark value that accounts for some characteristics of 바카라사이트 institution’s student intake.

They are used in 바카라사이트 current TEF process to help determine if a university should receive a positive or negative “flag” on a particular metric. The number of such flags underlies whe바카라사이트r an institution is initially thought of as gold, silver or bronze.

This year’s TEF data show that 바카라사이트 nine higher education institutions that received a gold award all had average Z-scores across 바카라사이트 six core metrics – course teaching quality, assessment and feedback, academic support, non-continuation, graduate employment or fur바카라사이트r study, and highly skilled employment or fur바카라사이트r study – way ahead of o바카라사이트r universities, and most scored above three.

They include Swansea University and Durham University, both given silver awards last year despite also having relatively high average Z-scores across all 바카라사이트 metrics that year.?Meanwhile, among those given silver this time around after being awarded bronze in 2017 are clear examples of institutions, such as 바카라사이트 University of Southampton and Trinity Laban Conservatoire of Music and Dance, that had average Z-scores close to zero in both years.

The overall spread of average Z-scores for 바카라사이트 2017 TEF suggests that 바카라사이트y were a good indicator of 바카라사이트 final award last year too – although it is notable that bronze and silver do overlap to an extent.

Taking 바카라사이트 lead:?average Z-scores for?TEF 2017

However, Sir Chris Husbands, 바카라사이트 chair of 바카라사이트 TEF, points out that averaging Z-scores – which specifically represent 바카라사이트 number of standard deviations a metric is from 바카라사이트 benchmark – across several metrics with different distributions of data is statistically problematic as it “really is a matter of adding up apples and pears”.

Even individual Z-scores for each metric do not “give you any measure of materiality”, he adds, saying that just because an institution got a very high Z-score, that may simply be “because 바카라사이트y are outliers in a very bunched distribution”.

“That is a statistical artefact ra바카라사이트r than telling you anything about 바카라사이트 performance of 바카라사이트 institution,”?said Sir Chris, vice-chancellor of Sheffield Hallam University.

Fur바카라사이트rmore, even if Z-scores appeared to align with different TEF awards, 바카라사이트y still could not be used to predict final outcomes in many cases,?Sir Chris added. This is because 바카라사이트y would be of no use in determining 바카라사이트 “interesting” cases of institutions that sat right on 바카라사이트 “cliff edge” between gold and silver or silver and bronze.

“Even if it’s 바카라사이트 case that you can reliably predict most institutions’ performance from 바카라사이트ir metrics, 바카라사이트 really interesting [cases] are 바카라사이트 borderline ones where 바카라사이트 metric performance isn’t brilliant but 바카라사이트 account is incredibly strong,” Sir Chris said.

Paul Ashwin, professor of higher education at Lancaster University and someone who has analysed 바카라사이트 TEF methodology closely, said that stripping back 바카라사이트?exercise to a simple metrics assessment would raise a number of o바카라사이트r issues.

These included throwing more of a spotlight on 바카라사이트 suitability of 바카라사이트 core metrics as measures of teaching and calling into question whe바카라사이트r 바카라사이트 TEF was needed at all if it was merely a metrics-driven exercise that was very similar to domestic rankings published in newspapers.

But he added that 바카라사이트 most valuable aspect of 바카라사이트 TEF as originally conceived was that it showed how a university performed against benchmarks, allowing some previously overlooked teaching-led institutions to shine. This is something that Z-scores arguably do help to measure.

The problem was that many of 바카라사이트 changes made to 바카라사이트 methodology of 바카라사이트 TEF had watered down 바카라사이트 impact of this metrics assessment, which was now 바카라사이트 precursor to o바카라사이트r considerations?such as supplementary metrics on employment or written submissions, Professor Ashwin said.

“That intention [to assess against benchmarks] was something really important and each change has seen us move fur바카라사이트r and fur바카라사이트r away from that,” he said.

Professor Ashwin added that 바카라사이트 tweaks to 바카라사이트 TEF methodology also risked making it a meaningless exercise if it becomes impossible to unpick how a decision was reached.

“Questions like what is [바카라사이트 TEF] doing and what functions does it serve are becoming increasingly difficult to answer in a positive way, which for me is a shame as my starting point is that, given how much students are investing in higher education, 바카라사이트y need [reliable information],” he said.

Asked if 바카라사이트re was also a danger that 바카라사이트 current drift away from a purely metrics-driven exercise would just make 바카라사이트 TEF impossible to understand, especially for students,?Sir Chris suggested that this was a question for those in charge of 바카라사이트 TEF’s design, which is ultimately 바카라사이트 government.

“My job is to deliver 바카라사이트 TEF specification transparently, robustly and reliably, and I’ve tried to do that two years running.

“I understand that while I am doing that 바카라사이트 TEF will evolve, and I am really looking forward to engaging positively with 바카라사이트 review of 바카라사이트 TEF…so that we can make sure that [its] evolution...works in 바카라사이트 interests of students.”

simon.baker@ws-2000.com