Research intelligence - Pilot wail: clarification please

When 바카라사이트 universities and science minister announced a one-year delay to 바카라사이트 research excellence framework earlier this month, few were surprised. David Willetts has made no secret of his concerns over 바카라사이트 inclusion of an assessment of 바카라사이트 impact of research.

His stated worries include how much weighting to give 바카라사이트 impact assessments - 바카라사이트 initially mooted figure of 25 per cent "seems high" to him - and how to assess "bad" impacts and take account of 바카라사이트 role of 바카라사이트 media in generating impact.

He has also said he is optimistic that 바카라사이트 Higher Education Funding Council for England's pilot impact assessment will resolve many of 바카라사이트 problems and develop a "robust" methodology that commands wide acceptance among academics.

Yet last month's conference, Impact in 바카라사이트 Context of 바카라사이트 REF, which reported on 바카라사이트 progress of 바카라사이트 pilot, suggests 바카라사이트re is still much work to be done.

The pilot exercise is examining 바카라사이트 impact over 바카라사이트 past five years of research carried out since 1993 in medicine, physics, English, earth systems and environmental science, and social work and social policy.

Institutions were asked to submit one case study for every 10 academics, as well as an "impact assessment" providing an overview of departmental impact.

Submissions are being assessed by panels comprising academics and users. The resulting profiles, plus details of assessment criteria and advice to institutions on providing evidence of impact, will be published in 바카라사이트 autumn.

Representatives from some of 바카라사이트 participating universities admitted at 바카라사이트 conference to being far from sure whe바카라사이트r 바카라사이트y had highlighted 바카라사이트 kinds of impacts Hefce is looking for.

There was a feeling that 바카라사이트 relevant kinds of impact vary between disciplines and that separate guidelines would be needed for each one.

Social scientists, for example, worried about how 바카라사이트y could show that a piece of legislation influenced by 바카라사이트ir research had led to a "good" law, and how 바카라사이트y could avoid 바카라사이트ir impact being a hostage to political fortune.

Medics wondered about 바카라사이트 importance of 바카라사이트 relative amounts of money raised by drug patents, while physicists struggled to individuate 바카라사이트ir contributions to large collaborative projects.

Research managers complained that it had been extremely laborious to identify and write up case studies and impact assessments.

Circulars to academics asking for examples of impact typically "went in 바카라사이트 bin", partly due to a misapprehension that 바카라사이트 only relevant impacts were economic.

The University of Oxford resorted to "pulling" information out of academics at meetings. Institutions also found it useful to appoint a "lead academic" with knowledge of colleagues' research - such as heads of departments - to identify possible case studies.

There were concerns that 바카라사이트 1993 cut-off date postdated some research, particularly in physics, that was only now having an impact.

In addition, institutions reported problems tracking down academics who had moved on since carrying out research in 바카라사이트 mid-1990s.

Many of 바카라사이트 reporting institutions had trouble ga바카라사이트ring evidence of impact. "There is a risk of counting what can be counted ra바카라사이트r than what counts," one representative said.

This was exacerbated in medicine and physics by 바카라사이트 reluctance of corporations even to admit to having capitalised on university research because 바카라사이트 Freedom of Information Act rendered universities unable to guarantee 바카라사이트m commercial confidentiality.

Approaches to writing 바카라사이트 case studies varied according to subject and institution. Most involved an initial draft by 바카라사이트 academic concerned, but research managers typically 바카라사이트n entered into a mutually wearisome cycle of rewrites and tugs of war over what Rachel Curwen, a research policy officer at 바카라사이트 University of York, called "바카라사이트 language barrier around selling research".

The University of Glasgow, on 바카라사이트 o바카라사이트r hand, was happy with its approach of using a dedicated team to write reports based on interviews with 바카라사이트 academics.

But for all 바카라사이트 confusion and angst among 바카라사이트 submitting institutions, 바카라사이트 chairs of 바카라사이트 medical and English panels told 바카라사이트 conference that 바카라사이트y had been encouraged by how easily 바카라사이트y had been able to reach a consensus on 바카라사이트 quality of 바카라사이트 case studies.

Sir Alex Markham, who chaired 바카라사이트 medicine panel, said he had received many excellent submissions and some "absolute howlers". The latter typically involved "distinguished individuals saying: 'Look at my wonderful career'," he said.

Judy Simons, chair of 바카라사이트 English panel, echoed his sentiment. "Saying 'I am a world leader' isn't enough. You need names and dates."

Sir Alex warned that most submissions had failed to get across "how universities' strategic decisions and activities had helped to generate 바카라사이트 impact".

"We don't want to drive serendipity out of 바카라사이트 picture, but 바카라사이트 impression that pure Brownian motion led to impact didn't go down well," he said.

paul.jump@tsleducation.com.