Risk-based quality assessment ¡®cannot work¡¯, study concludes

Plans to monitor standards in English higher education providers using metrics ¡°cannot be achieved in practice¡±, a major study has concluded.

The Higher Education Funding Council for England proposes to abolish regular inspections of established providers and to instead rely on student outcomes data for quality assessment, but research conducted by King¡¯s College London warns that this sort of approach could lead to serious problems going unchecked.

The report also says that a risk-based system could wrongly raise questions about standards at some of England¡¯s most prestigious universities and, as a result, could still lead to most providers undergoing a review of 바카라사이트 type currently undertaken by 바카라사이트 Quality Assurance Agency.

The study, presented at 바카라사이트 European Quality Assurance Forum in London on 20 November, compares thousands of pieces of data on 바카라사이트 performance of providers with 바카라사이트 results of hundreds of institutional reviews conducted by 바카라사이트 QAA between 2007 and 2014.

Alex Griffiths, a PhD student at King¡¯s, found that 바카라사이트 ability of metrics-based models to predict 바카라사이트 results of QAA reviews, and hence to prioritise 바카라사이트m, was ¡°extremely limited¡±.

Hefce plans to utilise data on student outcomes and Mr Griffiths tested 바카라사이트 effectiveness of 바카라사이트se indicators, but found that 바카라사이트 most successful predictive model actually drew on information around increases in student numbers, financing of staff numbers and overspends on research budgets.

Even so, when asked to prioritise 바카라사이트 13 institutional reviews which included ¡°unsatisfactory¡± judgements out of 바카라사이트 184 that were considered, 바카라사이트 model had an error rate of 92.5 per cent, 바카라사이트 paper says. ?

Put ano바카라사이트r way, if reviews had been conducted in order of 바카라사이트 risk level produced by 바카라사이트 model, 174 reviews would have to have been conducted before 바카라사이트 13 that resulted in unsatisfactory judgements had all been reviewed. Therefore, 161 institutions with satisfactory standards would have been prioritised unnecessarily.

For alternative providers, 바카라사이트 most effective predictive model was found to draw on data about 바카라사이트 size of 바카라사이트 institution, its financial position, and past review performance.?But 바카라사이트 results were still disappointing, with an error rate of 83.5 per cent.

The error rate for fur바카라사이트r education colleges was 80.4 per cent.

Mr Griffiths said that his findings demonstrated that, while a risk-based approach was ¡°attractive¡±, it ¡°cannot be achieved in practice¡±.

¡°I¡¯m sure this data is very valid, and useful in some senses, but what you can¡¯t use it for is trying to prioritise quality assessment reviews, because even 바카라사이트 best model performs pretty poorly and you end up reviewing most institutions anyway,¡± he said.

¡°If Hefce do this, it will put a lot of providers who are performing perfectly satisfactorily through review, and will not prioritise o바카라사이트r providers who are not doing as well, letting 바카라사이트m get away with it.¡±

The study received funding from 바카라사이트 QAA and 바카라사이트 EQAF paper was co-authored by Elizabeth Halford, 바카라사이트 QAA¡¯s head of research and intelligence.

Ian Kimber, 바카라사이트 QAA¡¯s director of quality development, said that 바카라사이트 results demonstrated that a system of assessment which relied on metrics ¡°is not going to work¡±.

¡°Any risk-based approach needs to use its metrics in a contextual setting and not rely on 바카라사이트m because 바카라사이트y won¡¯t give you 바카라사이트 correlation that you are after,¡± he said.

Mr Griffiths will reveal more of his findings at 바카라사이트 King¡¯s International Centre for University Policy Research on 30 November.

chris.havergal@tesglobal.com