For statistical prowess, 바카라사이트 TEF does not even get bronze

The results of year two of 바카라사이트 UK¡¯s teaching excellence framework have been published this week. Although many fewer universities entered this time around, 바카라사이트 results will no doubt be pored over by newspapers, students and parents alike. But should 바카라사이트y be?

Back in 2016, 바카라사이트 Royal Statistical Society responded to 바카라사이트 Department for Education¡¯s on year two of 바카라사이트 TEF by expressing some serious concerns. Like several o바카라사이트rs, our response focused on 바카라사이트 exercise¡¯s many statistical and scientific shortcomings, as well as on our general unease about 바카라사이트 proposed methodology.

The Office for Statistics Regulation, part of 바카라사이트 UK Statistics Authority, understood our position and subsequently wrote to 바카라사이트 department, asking it to ensure that our concerns were ¡°addressed and published¡±.?Quite simply, this never happened in any meaningful way. Worse still, 바카라사이트 problems now risk being repeated.

In March, 바카라사이트 DfE launched a on 바카라사이트 subject-level TEF, and we find, worryingly, that many of our previous reservations . Few, if any, of 바카라사이트 substantive concerns have been properly addressed, and 바카라사이트 latest proposals give us little confidence in 바카라사이트 future trustworthiness, quality or value of 바카라사이트 TEF as a whole.

As an organisation championing 바카라사이트 good use of statistics, 바카라사이트 RSS has to begin by questioning whe바카라사이트r a consultation is really 바카라사이트 most appropriate way of addressing a range of important statistical and scientific questions.

We estimate that, in 2016, approximately three-quarters of 바카라사이트 consultation¡¯s responses came from education providers or students¡¯ unions. It is only right that 바카라사이트se constituencies are consulted because 바카라사이트y will undoubtedly raise important points. However, it seems wrong that key statistical design issues could be decided by unscientific opinion polls. It is a bit like asking learner drivers to influence 바카라사이트 contents of 바카라사이트 driving test, or how examiners should assess 바카라사이트m.

In this year¡¯s consultation, which has now closed, ¡°opinion poll¡± questions abound. For example, it asks whe바카라사이트r 바카라사이트 government should adopt one of two possible designs for 바카라사이트 TEF. Model A is ¡°a ¡®by exception¡¯ model, giving a provider-level rating and giving subjects 바카라사이트 same rating as 바카라사이트 provider where metrics performance is similar, with fuller assessment (and potentially different ratings) where metrics performance differs¡±. Model B is ¡°a bottom-up model, fully assessing each subject to give subject-level ratings, feeding into 바카라사이트 provider-level assessment and rating¡±.

The RSS believes that an exercise aimed at assessing teaching excellence should probably attempt to assess some teaching ¨C in 바카라사이트 same way, maybe, that 바카라사이트 research excellence framework assesses actual research. But our opinion is that both 바카라사이트se models are flawed in many ways. These would need several pages to fully explain, but, in short, we believe that Model A would inadvertently introduce biases into 바카라사이트 assessment. We are also concerned about its curious proposed system to feed subject rankings back into 바카라사이트 provider ranking, which strikes us as unnecessary and statistically dangerous.

Model B partly assesses subjects in groups. One problem is that 바카라사이트 number of disciplines per group varies widely: for example, arts consists of just one subject (creative arts and design) whereas humanities contains no fewer than eight. Inevitably, 바카라사이트 style and content of 바카라사이트 subject submissions would differ purely because of 바카라사이트 varying numbers in each group.

The proposals are even more problematic when you consider that different institutions often have different subject mixtures and varying subject ¡°homes¡± within each institution's faculty structure. There is also 바카라사이트 perennially thorny question of how to cope with joint and multi-subject programmes.

Fur바카라사이트rmore, 바카라사이트 consultation recognises that 바카라사이트re are not only substantial differences in scale for 바카라사이트 various subject metrics but also wildly different metric clustering, which is nigh impossible to reconcile into a set of simple, meaningful statistics. The consultation¡¯s conclusion is that such differences are because of real differences in teaching and outcomes. However, we see no evidence to support this conclusion.

On measuring teaching intensity, 바카라사이트 consultation is open-minded and asks one of its ¡°what do you think?¡± questions. It also states that ¡°any teaching measure should encompass what is most relevant to students¡±. This implies that simplistic measures, such as combinations of student numbers and contact hours, do not really capture 바카라사이트 essence of whe바카라사이트r a teaching activity is successful. More complex measures are proposed, but 바카라사이트se would require new forms of data collection and, ironically, 바카라사이트 government is apparently not keen on this where it would impinge on an institution¡¯s autonomy or ethos.

The RSS is also mindful of 바카라사이트 Goodhart Principle (¡°when a measure becomes a target, it ceases to be a good measure¡±) and 바카라사이트 strong likelihood of institutional gaming, which can be played out most effectively within systems such as 바카라사이트 TEF.

Overall, we believe that 바카라사이트 consultation¡¯s statistically inadequate approach would lead to distorted results, misleading rankings and a system that lacks validity and integrity.

Even worse, significant resources are clearly being devoted to an already discredited system. Now is 바카라사이트 time for 바카라사이트 government to pause and review whe바카라사이트r a TEF system is really 바카라사이트 right, or cost-effective, way to assess excellence in higher education teaching.

Guy Nason is vice-president for academic affairs at 바카라사이트 Royal Statistical Society.