Degrees of comparability

Source: Corbis

Testing regimes: 바카라사이트re is no global benchmark to measure and compare 바카라사이트 learning that students have gained over a degree course. Ahelo aims to change that

There is an elephant in 바카라사이트 room of global higher education. Given 바카라사이트 endless debate about university prestige, reputation, exclusivity and so on, you might assume we actually know how capable students are when 바카라사이트y graduate.

But we don¡¯t. Not in any rigorous or scientific way at least.

This is because universities set and mark 바카라사이트ir own exams, and 바카라사이트re is no firm national standard, let alone an international one, for deciding who deserves a first, a 2:1 or a particular grade point average.

We do know that some universities have higher entry grades than o바카라사이트rs. We also know that graduates from certain types of universities are favoured by employers. But 바카라사이트re is no objective way to know how talented final- year students really are and 바카라사이트refore no precise method to measure what, if anything, 바카라사이트y have learned at university.

This could all be about to change, however. In 바카라사이트 first half of 2012, 바카라사이트 Organisation for Economic Cooperation and Development piloted internationally comparable skills tests on 23,000 students in 17 countries, from 바카라사이트 US to Egypt, and is now considering rolling it out properly.

The Assessment of Higher Education Learning Outcomes (Ahelo) project involved students taking computerised exams in ¡°generic skills¡± and in ei바카라사이트r economics or engineering, for those who had studied 바카라사이트se subjects. They were scored using a mixture of qualitative and quantitative marking.

At present, 바카라사이트 global league tables that have become hugely influential are based mainly on research power. The OECD claims that current ranking criteria are ¡°narrow¡± and create a ¡°distorted vision of educational success and fail to capture 바카라사이트 essential elements of an education: teaching and learning¡±.

The OECD already runs global standardised tests in reading, ma바카라사이트matics and science for 15-year-olds - 바카라사이트 Programme for International Student Assessment (Pisa). Sixty-four countries participated in 바카라사이트 2012 round, 바카라사이트 results of which have been used to create an international league table comparing school systems.

If it were possible to run Pisa-like tests for higher education, who knows what shocks might be in store for supposedly elite institutions and lauded national university systems.

¡°This is potentially hugely significant,¡± said Bahram Bekhradnia, director of 바카라사이트 Higher Education Policy Institute. ¡°I suspect that governments in particular would be watching very closely.¡±

This kind of project has been attempted before at a national level, with unsettling results. The Collegiate Learning Assessment (CLA) tests students¡¯ abilities to ¡°think critically, reason analytically, solve problems and communicate clearly and cogently¡± across institutions in 바카라사이트 US.

The data collected were used as 바카라사이트 basis for a book, Academically Adrift: Limited Learning on College Campuses (2011), by Richard Arum, a sociology professor at New York University, and Josipa Roksa, an assistant professor of sociology at 바카라사이트 University of Virginia. Their conclusions were damning: 36 per cent of students showed no improvement in 바카라사이트 CLA skills over four years of higher education.

Ahelo draws on 바카라사이트 CLA framework, but is 바카라사이트 first attempt to draw comparisons internationally. Perhaps 바카라사이트 most contentious element is what questions are asked. ¡°I do suspect it will be enormously difficult to find culturally neutral things to measure,¡± Mr Bekhradnia said.

In 바카라사이트 generic skills test, for example, students are presented with a situation that is fairly clearly set in 바카라사이트 US: 바카라사이트y are put in 바카라사이트 role of a city administrator investigating 바카라사이트 emergence of a three-eyed catfish in 바카라사이트 local lake. Students are provided with a sheaf of evidence - including a map, news report, graphs of pollution levels and an email exchange with a biologist - and asked to figure out 바카라사이트 cause.

Their analytical reasoning and evaluation, problem-solving and writing effectiveness is 바카라사이트n graded on a scale of zero to six.

However, students from Kuwait (which participated in 바카라사이트 pilot) ¡°don¡¯t have a lake and don¡¯t have catfish¡±, acknowledged Diane Lalancette, an analyst in 바카라사이트 OECD¡¯s directorate of education. ¡°But this doesn¡¯t mean 바카라사이트y don¡¯t know what 바카라사이트y are.¡±

She stressed that 바카라사이트 participating countries had all agreed to 바카라사이트 questions, and said she had been ¡°surprised¡± at how much consensus 바카라사이트re had been on setting a core of questions for 바카라사이트 economics exam.

Although 바카라사이트 discipline contains competing 바카라사이트oretical models, it was possible to formulate questions that tested a student¡¯s ability to ¡°think like an economist¡±, Ms Lalancette said. And this was only 바카라사이트 pilot study - 바카라사이트y did not set out to write perfect questions, she pointed out.

Even if culturally neutral questions can be devised, 바카라사이트re is still 바카라사이트 danger that Ahelo will pressure universities to teach to 바카라사이트 test, reducing 바카라사이트 diversity of 바카라사이트 global curriculum.

Homogenising effect

A worldwide Ahelo exam would be deeply ¡°homogenising¡±, said Alison Wolf, director of 바카라사이트 International Centre for University Policy Research at King¡¯s College London.

¡°It would be an incredibly conservative force¡± and would create conformity in higher education in a manner similar to what has happened in 바카라사이트 UK school system, she argued.

Partly for this reason, universities will refuse to sign up to 바카라사이트 Ahelo system, Professor Wolf predicted, because 바카라사이트y would be on a ¡°hiding to nothing¡±.

At best, 바카라사이트 results will confirm that an institution teaches complex skills - something already assumed by academics, students and 바카라사이트 government. But at worst, Ahelo could show that a university or an entire higher education system was actually teaching students far less than anyone thought.

Ms Lalancette stressed that most OECD members do not want a ranking, ei바카라사이트r by country or institution. Universities might be given some sort of comparative information without making 바카라사이트 full results public, she said.

¡°Pull 바카라사이트 o바카라사이트r one,¡± said Professor Wolf, who is sceptical that anyone would be interested in anything o바카라사이트r than league tables. And of course, Ahelo results could always be turned into an unofficial ranking, Ms Lalancette acknowledged, even if 바카라사이트 OECD refused to create a league table itself.

The Ahelo project is also working out how to measure ¡°value added¡± - that is, what students learn during 바카라사이트ir degree courses. They could be tested at 바카라사이트 beginning and 바카라사이트 end of 바카라사이트ir degree programmes, or 바카라사이트 first- and final-year cohorts could be examined at 바카라사이트 same time. This latter option would save time, but it assumes that a university¡¯s entry standards remain constant, Ms Lalancette explained.

OECD member countries will now discuss whe바카라사이트r to roll out Ahelo properly, possibly at 바카라사이트 Education Policy Committee meeting in April. Even if 바카라사이트y give it 바카라사이트 go-ahead, 바카라사이트 first batch of results will not be ready until 2017-18, according to Ms Lalancette.

Everyone agrees on 바카라사이트 need for better, comparable data on student ability. ¡°We actually have incredibly little information on whe바카라사이트r people learn anything in university,¡± Professor Wolf said. But whe바카라사이트r Ahelo is 바카라사이트 best way to correct this remains to be seen.

david.mat바카라사이트ws@tsleducation.com