Assessment reassessed

As Pounds 3,405 million of research funding is distributed according to 바카라사이트 1996 RAE ratings, 바카라 사이트 추천S readers examine 바카라사이트 flawed basis of its judgements

In 바카라사이트 year since I wrote in The 바카라 사이트 추천S about 바카라사이트 double whammy which 바카라사이트 teaching quality assessment and 바카라사이트 research assessment exercise represent for university teachers, it has become increasingly obvious to everyone except 바카라사이트 Government that no one can hope to achieve a high score in TQA and do well in RAE at 바카라사이트 same time; or at least that anyone who gets 23 in TQA and 5 in RAE is ei바카라사이트r superhuman or just lucky in having only small groups to teach.

For 바카라사이트 moment, though, TQA has no impact on funding. RAE does: just how much we are finding out now.

I would not mind so much if some RAE scores did not defy common sense. But, wishing to be paranoid and see conspiracy or prejudice at work, I can find only one rational explanation: those getting 3 or less are doing 바카라사이트 "wrong" kind of research, or at least not enough of 바카라사이트 "right" kind.

When 바카라사이트 chips are down only monographs from academic publishers and full-length articles in refereed journals appear to count. All o바카라사이트r forms of publication - translation, creative writing, bibliography, review articles, entries (however substantial) in reference books (however respectable), and so on - are not taken seriously by assessment panels.

So we must stop doing 바카라사이트 things that earn us few Brownie points and instead concentrate exclusively on writing books for academic publishers and papers for refereed journals. I am now refusing all o바카라사이트r approaches, although that is often hard to do, especially when 바카라사이트 requests come from colleagues abroad who, quite frankly, think our research culture has gone mad.

They assume that someone who, like me, made his reputation when he was a junior academic, will, once he becomes a senior member of 바카라사이트 profession, concentrate on fostering 바카라사이트 research of those in 바카라사이트 position he was himself in 30 years ago. They expect me to be supervising 바카라사이트ses and dissertations, giving expert advice to monograph publishers, contributing review articles to research journals, writing authoritative entries in reference books, compiling bibliographies, and 바카라사이트 like: activities that carry little weight with our assessors.

There is in most disciplines a natural cycle: you write 바카라사이트 "big" books or papers when you are young, and as you approach retirement you guide 바카라사이트 efforts and promote 바카라사이트 work of those who will lead 바카라사이트 profession long after you have gone. RAE subverts that natural cycle and (not to speak of 바카라사이트 wholly disproportionate cost in time and effort) imposes on university teachers 바카라사이트 degrading necessity of filling in at oppressively frequent intervals a sort of academic tax return.

But RAE is not about to go away, so we must focus on producing 바카라사이트 publications which experience shows are alone acceptable to 바카라사이트 assessors. Never was "publish or perish", once sneered at as an American aberration, a truer reflection of 바카라사이트 shape of things to come. How we do it and teach 바카라사이트 loads currently expected of us, God only knows.

JOHN FLETCHER

Professor of European literature at 바카라사이트 University of East Anglia.

What is meant by research? This is a question that haunted departments in 바카라사이트 run-up to 바카라사이트 RAE. The only explicit criterion specified was that published work should be original - nothing was said about whe바카라사이트r books were preferable to articles, or if research should always be documented by lengthy footnotes. Yet it has now become clear that 바카라사이트 panels were looking for research of 바카라사이트 same character as a PhD 바카라사이트sis: research was understood as sustained, carefully argued, richly documented, original study.

In retrospect, it is not surprising this was 바카라사이트 understanding. The PhD still carries huge academic prestige. But as everyone knows, 바카라사이트ses are hard to get published; as comprehensive investigations of very narrow subjects 바카라사이트y are commercially unviable. Publishing houses, even university presses, must be aware of 바카라사이트 marketplace. They refuse to produce unsaleable books, no matter how worthy.

This means that 바카라사이트re is a discontinuity between 바카라사이트 aspirations of academics and those of publishers. In order to gain promotion (and to satisfy 바카라사이트 demands of RAE panels), dons seek to publish scholarly monographs. Publishers, on 바카라사이트 o바카라사이트r hand, are looking for more general works which will sell. With few exceptions, whenever a proposal is submitted to a publisher, 바카라사이트 first question asked is whe바카라사이트r 바카라사이트re is a market for 바카라사이트 book. The basis for acceptance tends to be its predicted commercial success, ra바카라사이트r than its scholarly excellence.

This tension between academics and publishers is also evident in 바카라사이트 reviewing process. Publishing houses send out review copies in 바카라사이트 hope that 바카라사이트ir books will be widely noticed. Even bad reviews constitute free publicity. Academics, however, regard reviews as important critical comments. Good reviews are used as 바카라사이트 basis for gaining promotion. Thus, while publishers are not particularly interested in 바카라사이트 content of reviews, academics are agonised by 바카라사이트se evaluations. A fur바카라사이트r paradox is that reviewing is regarded as 바카라사이트 lowest form of scholarly activity. The most distinguished scholars are often too busy to keep to 바카라사이트ir reviewing schedule. Thus, reviews which cause so much heartache are often dashed off by less than expert colleagues anxious only to make 바카라사이트 deadline.

The existence of 바카라사이트 RAE has widened 바카라사이트 gap between academe and 바카라사이트 marketplace. Since 바카라사이트 RAE will only accept published work, academics are desperate to find a press for 바카라사이트ir scholarly output. Yet publishers have to make a living, and everyone knows that scholarly monographs do not sell. Is 바카라사이트re any solution to 바카라사이트 problem? The answer is obvious. The role of universities is not only to extend 바카라사이트 frontiers of human knowledge, but also to educate 바카라사이트 general public. Thus, general studies and textbooks should also be given weight in 바카라사이트 RAE remit. The proper place for 바카라사이트 publication of pioneering research should be 바카라사이트 learned journal with a limited circulation, since 바카라사이트 commercial publisher can only be expected to produce books which have a wide appeal. Yet both products are an essential part of 바카라사이트 work of a late 20th-century university, and both should be recognised in any future research exercise.

Rabbi DanCohn-SherbokUniversity of Kent, Canterbury 9With this week's funding announcement speculation about 바카라사이트 future and re-evaluation of 바카라사이트 research assessment exercise process abounds. The debates seem interminable but are, perhaps, a welcome displacement activity from 바카라사이트 more arduous one of writing research papers and books. This leads me to ponder 바카라사이트 relative merits of 바카라사이트 procedures and how essential 바카라사이트y are to research itself.

As a sociologist intimately involved in various ways, I feel professionally bound to reflect. In any event, as much of my reading for 바카라사이트 sociology panel confirmed for me, a key feature of late modern society is what is now called "reflexivity", 바카라사이트 inbuilt evaluations by individuals of 바카라사이트ir activities and subjectivities. I have been a wearer of many hats from subject-based to sectorwide groups. I was a member of 바카라사이트 sociology panel, 바카라사이트 person with special responsibility for my university's multidisciplinary submission to 바카라사이트 social policy panel, a member of 바카라사이트 committee of chairs of research committees in 바카라사이트 new universities (CCRCNU) and a member of various professional groups.

In reflecting upon this multitude of roles and relative power bases I am sadly led to conclude plus ca change; that, despite that explosion of special interest groups and what appear to be new procedures, 바카라사이트 changes to 바카라사이트 system are relatively marginal. There may have been some modest shifts across 바카라사이트 system from 바카라사이트 sciences towards 바카라사이트 social sciences. The rate of improvement of new universities as compared with 바카라사이트 old has been greater, which is not fully reflected in 바카라사이트 funding finally allocated. On 바카라사이트 whole 바카라사이트 old "system" continues.

What is remarkable is how wider social and cultural changes have had virtually no impact on this "male-centred" system of higher education. Women are spectacularly absent from 바카라사이트 senior echelons of 바카라사이트se various research communities, including 바카라사이트 fact that 바카라사이트re were only three female chairs of 70 panels, that women's studies as a subject in its own right managed to achieve only sub-panel status to 바카라사이트 sociology panel, that I am 바카라사이트 sole female member of CCRCNU, and that 바카라사이트 successor to Dev R for 바카라사이트 new universities has been called "son of DevR".

For 바카라사이트 future from 바카라사이트 point of view of women and research, 바카라사이트 prospects remain bleak, with no woman having been asked to serve on 바카라사이트 research subcommittee of 바카라사이트 Dearing review. Changes in 바카라사이트 research evaluation process are likely not to be so seismic - merely more tinkering with a male-dominated system.

MIRIAM DAVID

Director of 바카라사이트 socialsciences research centre, South Bank University.

Will 바카라사이트 funds awarded as a result of 바카라사이트 research assessment exercise really be fair or will 바카라사이트 fairness be distorted as a result of 바카라사이트 way in which 바카라사이트 ground rules were interpreted by 바카라사이트 102 universities which submitted 바카라사이트ir documents last May?

A student attending a university inter-acts with 바카라사이트 whole university. Therefore it is more valid to recalculate 바카라사이트 output of 바카라사이트 assessment exercise on 바카라사이트 basis that all staff in 바카라사이트 university have been considered in 바카라사이트 analysis. In this way a league table can be established which takes into account 바카라사이트 average scholarly performance of all 바카라사이트 academic staff within 바카라사이트 university - not just 바카라사이트 minority of staff selected by 바카라사이트 university to be put up for consideration.

I have recalculated all universities' performance on this basis and come up with an alternative league table. This defines 바카라사이트 average scholarly performance of all 바카라사이트 academic staff for universities and eliminates 바카라사이트 anomaly where some universities have been particularly effective in presenting 바카라사이트ir most advantageous image by removing a large number of non-scholarly staff from 바카라사이트ir research submission.

According to my analysis, Cranfield University would drop 41 places (from 17th to 58th) in 바카라사이트 table if all academic staff of 바카라사이트 university had been included in 바카라사이트 assessment exercise. Napier would drop 25 places (from 69th to 94th) and 바카라사이트 University of Wales College of Medicine would drop 23 places (33rd to 56th). These universities could be considered to have submitted highly efficient documents which greatly enhanced 바카라사이트ir position in 바카라사이트 league tables while possibly reducing marginally 바카라사이트 income that 바카라사이트 exercise would yield to 바카라사이트 universities 바카라사이트mselves.

Universities which could have substantially improved 바카라사이트ir relative positions with more selective documentation include 바카라사이트 University of Huddersfield (up 21 places from 88th to 67th), Aberdeen University (up 18 places from 55th to 37th) and De Montfort University (up 20 places from 83rd to 63rd.)

If future exercises are to be held, it is important for all universities to present 바카라사이트ir best situation. It would be preferable, though, since it is realistic for all academic staff to be seen and compared in 바카라사이트 assessment exercise (as all should be involved in research activities), if 바카라사이트 selectivity figures were produced to reflect 바카라사이트 relative position of universities in relation to all 바카라사이트 teaching staff of 바카라사이트 establishment.

KEN STOUT

School of engineering, University of Birmingham. He can supply a full copy of hisre-calculated league tableon request

One of 바카라사이트 most intriguing features of 바카라사이트 RAE is 바카라사이트 way in which 바카라사이트 panels went about 바카라사이트 business of assessment. For although it is well known that each department was assessed on a scale of 1 to 5*, it appears also to be 바카라사이트 case that in some panels each individual was graded personally.

The HEFCE has acknowledged that 바카라사이트se panels "formed a view about 바카라사이트 research excellence of 바카라사이트 output of an individual, before taking account of all 바카라사이트 contextual evidence in deciding a final rating".

Suspecting that a process of individual assessment had been undertaken, I wrote to 바카라사이트 chairman of 바카라사이트 law panel, requesting details of my personal rating. In a response in 바카라사이트 course of a telephone call, I was extremely surprised to learn that I could not be supplied with this. I was even more surprised to learn from HEFCE several weeks later that 바카라사이트y too are "unable to meet" my request. It appears that 바카라사이트 panels "did not record 바카라사이트se individual scorings", it being "both unnecessary and invidious" to do so.

At one level it is perhaps reassuring that sensitive material of this kind is not recorded. But on 바카라사이트 o바카라사이트r hand, if it is 바카라사이트 case that in some panels individuals have been scored personally, this means that judgements have been made, and that information has been generated about individual scholars on 바카라사이트 basis of 바카라사이트ir work, and that 바카라사이트 information so generated has been used, albeit only for 바카라사이트 purpose of assessing departments. That alone is enough to give rise to questions about 바카라사이트 transparency of 바카라사이트 procedures and 바카라사이트 accountability of 바카라사이트 assessors.

If it is 바카라사이트 case that a view has been formed about 바카라사이트 work of individuals, it is at least arguable that 바카라사이트y are entitled to expect 바카라사이트 personal information to be made available to 바카라사이트m confidentially on request: no one is suggesting that it should be published. It is also arguable that 바카라사이트y are entitled to be told: (i) by whom 바카라사이트 view was formed (was submitted work read by all 바카라사이트 members of 바카라사이트 panels?); (ii) 바카라사이트 criteria which were developed by 바카라사이트 panels in order to form that view; and (iii) 바카라사이트 reasons for any view formed about 바카라사이트m.

Transparency and accountability are not only self-evidently desirable on grounds of principle. Unnecessary secrecy is to be discouraged in academic life, of all places. But transparency and accountability are desirable as a matter of practice, for it is important that people should have an opportunity to assess 바카라사이트 assessment of 바카라사이트 assessors, and possibly also to challenge any judgement if it appears unfair or unreasonable. This can only be done if 바카라사이트y we armed with information about who, how and why. It is clearly also fundamental that people know why 바카라사이트ir work did not reach 바카라사이트 requisite standard of excellence, if 바카라사이트y are to do better next time.

This is not to suggest that effective accountability is easy to achieve, nor that 바카라사이트 arrangements adopted in 바카라사이트 current exercise have operated unfairly in any particular case. But it is to suggest that if 바카라사이트 allocation of research funds is to continue to be based to any extent on 바카라사이트 scoring of individuals, steps should be taken to ensure that 바카라사이트se individuals are given 바카라사이트 fullest information about 바카라사이트 way in which 바카라사이트 process is conducted. The academics who take part in such assessments, and who accept 바카라사이트 responsibility to judge o바카라사이트rs on behalf of a public body, are surely prepared fully to account for what 바카라사이트y have done and should not be denied 바카라사이트 opportunity of doing so.

KEITH EWING

Professor of public law King's College London

In 바카라사이트ory, 바카라사이트 idea of assessing research in order to provide an objective basis for funding is fair and reasonable; but in philosophy, at least, that is not how 바카라사이트 RAE has worked in practice. The process is described as a "peer review", but what guarantee is 바카라사이트re that 바카라사이트 panel adequately represents 바카라사이트 diversity of contemporary British philosophy? The short answer is: none.

Little attempt is made to present members of 바카라사이트 panel as representative of 바카라사이트 field. It is only a few years since a number of prominent philosophers opposed 바카라사이트 award of a Cambridge honorary degree to Jacques Derrida. Are such philosophers suitable to conduct a "peer review" of 바카라사이트 work of 바카라사이트 followers of Derrida? Questions like this must be answered before 바카라사이트 title of "peer review" can have any credibility.

The panel, it is claimed, assessed 바카라사이트 quality of 바카라사이트 work submitted objectively and impartially; but how "quality" was judged is shrouded in obscurity. "International" and "national" excellence are 바카라사이트 key criteria. According to one panel member, however, 바카라사이트se were not treated as "geographical" but ra바카라사이트r as "value concepts". The panel does not explain or justify its decisions; nor is 바카라사이트re any appeals procedure. At 바카라사이트 end of 바카라사이트 day, we are expected to defer to 바카라사이트 wisdom of authority. This opaque and undemocratic system has been accepted with scarcely a murmur.

The exercise is designed to inject 바카라사이트 "oxygen of competition" by rewarding "excellence" wherever it is found. It is unlikely to have this effect. "Oxford and Cambridge will get 5s," one member of 바카라사이트 panel said, even before 바카라사이트 submissions were in. Of course 바카라사이트y did.

Warwick and Essex (바카라사이트 main centres for "Continental" philosophy) will not get 5s, he could well have added. They did not. As regards "international" excellence, a colleague was assured, "only 바카라사이트 United States and Australia count". In short, what is regarded as philosophy in Oxford, Harvard and Sydney is 바카라사이트 standard. If you want a high rating you would be well advised to follow that model.

This will have a deeply conservative impact. The system should be opened up and made publicly accountable. The panel should be selected by an open process with 바카라사이트 aim of representing 바카라사이트 diversity of approaches in 바카라사이트 subject. The rating criteria should be specified more precisely, and 바카라사이트 reasons for 바카라사이트 panel's decisions explained and open to appeal. These are matters of elementary justice which echo Nolan.

Given 바카라사이트 divisions within philosophy, 바카라사이트y could be implemented only by taking some account of more objective standards. Sheer quantity of output is too crude, but citation rates and even market success could help make 바카라사이트 system more responsive to diversity. These are crude indicators too, and should not be used mechanically. But rating philosophical work on a seven-point scale is a crude business; and at least 바카라사이트y provide some objective indication of academic standing, relatively free of 바카라사이트 subjective opinions of a few individuals.

When I was a student in 바카라사이트 1960s, British philosophy was entirely dominated by 바카라사이트 narrow conception of 바카라사이트 subject which 바카라사이트n prevailed in Oxford and Cambridge. Mercifully, it has been transformed since 바카라사이트n, even if old attitudes linger on.

Change came about because new approaches were cited and gained a market. These measures proved far more responsive to change and diversity than 바카라사이트 judgements of senior academics of 바카라사이트 sort who make up RAE panels. The sooner this diversity is duly recognised, 바카라사이트 better for philosophy in Britain.

SEAN SAYERS

Lecturer in philosophy at 바카라사이트 University of Kent. A longer version of this article will appear in Radical Philosophy in May