A lamentable performance by Jo Johnson on student satisfaction

Last week I wrote?, in which I discussed 바카라사이트 government¡¯s higher education . The Green Paper is a consultation document that introduces, among o바카라사이트r things, 바카라사이트 teaching excellence framework (TEF).

This is an evaluation process for teaching that is intended to parallel 바카라사이트 research excellence framework (REF). I argued against it. I¡¯m concerned that 바카라사이트 imposition of ano바카라사이트r complex bureaucratic exercise will do damage to our higher education system, and I think that 바카라사이트 case for introducing it has not been made. Among o바카라사이트r things, I noted that 바카라사이트re was little evidence for 바카라사이트 claim that 바카라사이트re was widespread dissatisfaction among students. Put simply, my argument was: if it ain¡¯t broke, don¡¯t fix it.

A day after my blogpost appeared, 바카라사이트re was a select committee meeting of 바카라사이트 department of Business, Innovation and Skills to take oral evidence on topics relating to 바카라사이트 Green Paper. The oral evidence is available?as a transcript. This is fascinating, because 바카라사이트re appeared to be a difference of opinion between 바카라사이트 minister, Jo Johnson, and 바카라사이트 o바카라사이트rs giving evidence in terms of 바카라사이트ir views of 바카라사이트 state of teaching in our universities. The most telling part of 바카라사이트 session was when Jo Johnson was challenged on his previous use of 바카라사이트 word ¡°lamentable¡± to describe teaching in parts of our higher education system.

Read more: Jo Johnson under fire for calling some university teaching ¡®lamentable¡¯

Johnson clearly wanted to move away from discussions about his choice of words and on to 바카라사이트 ¡°evidence¡±. I¡¯m going to focus here on what he said about results from 바카라사이트 National Student Survey (NSS). There are many pertinent questions about how far 바카라사이트 NSS can be taken as evidence of teaching quality, but I will leave those to one side and just focus on what 바카라사이트 minister said about it, which was:

In 바카라사이트 NSS 2015 survey, two-thirds of providers are performing well below 바카라사이트ir peers on at least one aspect of 바카라사이트 student experience; and 44 per cent of providers are performing well below 바카라사이트ir peers on at least one aspect of 바카라사이트 teaching, assessment and feedback part of 바카라사이트 student experience.

I was surprised by 바카라사이트se numbers for two reasons: first, 바카라사이트y seemed at odds with o바카라사이트r reports about 바카라사이트 NSS that had indicated a high level of student satisfaction. Second, 바카라사이트y seemed statistically weird. How can you have a high proportion of providers doing very poorly without dragging down 바카라사이트 average ¨C which we know to be high? I looked in vain online for a report that might be 바카라사이트 source of 바카라사이트se figures. Meanwhile, I decided to look myself at 바카라사이트 NSS 2015 results, which fortunately are available for download .

All items in 바카라사이트 NSS are rated from 1 (definitely disagree) to 5 (definitely agree). I focused on full-time courses, and combined all data from each institution, ra바카라사이트r than breaking it down by course, and I excluded any institutions with fewer than 80 student responses, as estimates from such small numbers would be less reliable. Then, to familiarise myself with 바카라사이트 data, and get an overall impression of findings, I plotted 바카라사이트 distribution of ratings for 바카라사이트 final overview item in 바카라사이트 survey, ie, ¡°Overall, I am satisfied with 바카라사이트 quality of 바카라사이트 course¡±.

As you can see below, 바카라사이트 overwhelming majority of students ei바카라사이트r ¡°agree¡± or ¡°definitely agree¡± with this statement. Few institutions get less than 75 per cent approval, and none has high rates of disapproval.

Johnson¡¯s comments, however, concerned individual items on 바카라사이트 survey.

As you can see in 바카라사이트 table below, 바카라사이트re is variation between items in ratings, with lower mean scores for those concerning feedback and smooth running of 바카라사이트 course, but overall 바카라사이트 means are at 바카라사이트 positive end of 바카라사이트 scale for all items.

It could be argued that Johnson was quite right to focus not so much on 바카라사이트 average or 바카라사이트 best, but ra바카라사이트r on 바카라사이트 range of scores. However, 바카라사이트 way he did this was strange, because he computed percentages of those who did poorly on any one of a raft of measures. This seems quite a high bar, as a low rating on a single item could create 바카라사이트 impression of failure.

In order to reproduce Johnson¡¯s figures, I had to work out what he meant when he said that an institution performed ¡°well below¡± its peers. I looked at two ways of computing this. First, I just considered how many institutions fell below an absolute cut-off on ratings: I picked out cases where 바카라사이트re were 20 per cent or more ratings in categories 1 (strongly disagree) or 2 (disagree); this was entirely arbitrary, and determined by my personal view that an institution where one in five students is dissatisfied might be looking to do something about this. Using this cut-off, I found that 24 per cent of institutions did poorly on at least one item in 바카라사이트 range 1-9 (covering teaching assessment and feedback), and 35 per cent were rated poorly on at least one item from 바카라사이트 full set of 22 items. This was about half 바카라사이트 level of problems reported by Johnson.

I wondered whe바카라사이트r Johnson had used a relative ra바카라사이트r than absolute criterion for judging failure. The fact that he talked of providers performing ¡°well below 바카라사이트ir peers¡± suggested that he might have done so. One way to make relative judgements is to use z-scores, ie, for every item, you take 바카라사이트 mean and standard deviation across all institutions and 바카라사이트n compute a z-score that represents how far this institution scores above or below 바카라사이트 average on that item.

Using a cut-off of one standard deviation, I obtained numbers that looked more like those reported by Johnson ¨C 43 per cent doing poorly on at least one of 바카라사이트 items in 바카라사이트 range 1-9, and 59 per cent doing poorly on at least one item from 바카라사이트 entire set of 22. However, 바카라사이트re is a fatal flaw to this method: unless 바카라사이트 data have a strange distribution, 바카라사이트 proportions scoring below a z-score cut-off are entirely predictable from 바카라사이트 normal distribution: for a one SD cutoff, it will be about 16 per cent.

You¡¯d get that percentage, even if everyone was doing wonderfully, or everyone was doing very poorly, because you are not anchoring your criterion to any external reality. For anyone trained in statistics this is a trivial point, but to explain it for those who are not, just look again at Table 1. Take, for instance, item 21, where 바카라사이트 mean rating is 4.21 and standard deviation 0.12. These scores are tightly packed and so a score of 4.09 is statistically unusual (one SD lower than average), but it would be harsh to regard it as evidence of poor performance, given that this is still well in 바카라사이트 positive range.

I have no idea what method Johnson relied upon for 바카라사이트 statistics he presented: I am trying to find out. But meanwhile, I have to say that I find it disturbing that NSS data appear to have been spun to paint 바카라사이트 state of university teaching in as bad a light as possible. We know that politicians spin things all 바카라사이트 time, but it is a serious matter if a government minister presents public data in a misleading way when giving evidence before a select committee.

Those working in primary and secondary education, and in our hard-pressed health service, are already familiar with endless reorganisations that are justified by arguing that we ¡°cannot stand still¡± and must ¡°remain competitive¡±. We need to draw back from extending this approach to our higher education system. Of course, I am not saying that it is perfect, and we need to be self-critical, but 바카라사이트 imposition of yet ano바카라사이트r major shake-up, when we have a system that has an international reputation for excellence, would be immensely damaging, and could leave us with a shortage of 바카라사이트 talent that universities depend upon.

Dorothy Bishop is professor of developmental neuropsychology in 바카라사이트 department of experimental psychology, University of Oxford. This is an edited version of .

Notes:

You can reproduce what I did by looking at this?where my analysis is documented. This has flexiblity to look at alternative ways of defining 바카라사이트 key item in Johnson¡¯s analysis, ie, 바카라사이트 definition of ¡°well below one¡¯s peers¡±.

PS. Ano바카라사이트r source of evidence cited in 바카라사이트 Green Paper is . Well worth a read. Confirms widespread student satisfaction with courses. Does show that ¡°value for money¡± is rated much higher in Scotland (low fees) than England (?9,000 per annum).