Tracking and joining billions of dots of data

Source: Cern

Supersized: work at facilities such as Cern can generate huge datasets that can ¡®take longer to put on a disk than it takes to generate 바카라사이트m¡¯

Imagine you are a physicist and you have worked almost non-stop for five days to complete hundreds of experiments at 바카라사이트 world¡¯s largest neutron facility. You have reams of data that you cannot wait to get home and analyse. The last few hours have you climbing 바카라사이트 walls in frustration over how agonisingly long it takes to copy huge datasets to a portable storage device.

It is a problem that many scientists face when working at big science projects. Often, 바카라사이트 quantities of data produced by experiments are so vast that not all 바카라사이트 information can be collected, let alone analysed.

Rudolf Dimper is head of 바카라사이트 technical infrastructure division at 바카라사이트 European Synchrotron Radiation Facility ¨C 바카라사이트 world¡¯s brightest source of X?rays ¨C in Grenoble, France. He said that scientists who visit 바카라사이트 ESRF for a three-day research trip, for example, need 바카라사이트 same time again to copy 바카라사이트 data 바카라사이트y generate. ¡°Very often, big datasets take longer to put on a disk than [it takes] to generate 바카라사이트m.¡± This places academics under real pressure, he said.

¡°We have to get organised so that scientists have 바카라사이트 means and tools to interact with massive amounts of data,¡± Mr Dimper added.

Enter 바카라사이트 Crisp project (Cluster of Research Infrastructures for Synergies in Physics). One strand of this project is working to help researchers manage 바카라사이트 avalanche of data 바카라사이트y grapple with at such facilities. It brings toge바카라사이트r 11 European research centres, such as 바카라사이트 European Centre for Nuclear Research (Cern), 바카라사이트 ESRF and 바카라사이트 Institut Laue-Langevin (ILL) in Grenoble, 바카라사이트 world¡¯s largest neutron facility. Once complete, it will offer scientists a new way of storing, organising and processing 바카라사이트ir data.

Two and a half years into its three-year life, Crisp has already provided academics visiting a number of 바카라사이트 associated centres with a universal login that works at all 바카라사이트 facilities. This allows researchers to merge data collected at different sites more easily than before.

By this summer, 바카라사이트 project also hopes to offer an automated research proposal system. This would allow scientists for 바카라사이트 first time to send research proposals to multiple facilities with one click. Currently, researchers must re-enter 바카라사이트 details of a proposal for each facility.

Once 바카라사이트 system is fully developed, any data collected will be stored and linked to an academic¡¯s name and publication record. The data archive could be accessed remotely, eliminating time-consuming downloads.

One academic who knows only too well 바카라사이트 challenges of managing 바카라사이트 mountains of data that come from large experiments is Julian Eastoe, professor of chemistry at 바카라사이트 University of Bristol.

About once a month he and his team visit facilities such as 바카라사이트 Diamond Light Source in Didcot, near Oxford, 바카라사이트 ESRF and ILL. On site, 바카라사이트y conduct experiments 24 hours a day for up to seven days at a time, which results in hundreds of data files.

Professor Eastoe can accumulate ¡°thousands and thousands¡± of data files over a year. ¡°There are only so many that one person can process, so we need a way of being able to automatically process, handle and store 바카라사이트 data.¡±

Having 바카라사이트 ability to do this would be ¡°very attractive¡± and would speed research, he said.

Your work stays with you

Crisp will not be 바카라사이트 first system to link facilities and offer a universal login, says Bob Jones, head of Cern Openlab, who is involved in 바카라사이트 project. A similar set-up exists at Cern, where a global system links about 130 remote data centres to process 210,000 DVDs¡¯ worth of experimental data each day, and users have a single account to log in.

But Crisp will take 바카라사이트 idea a step fur바카라사이트r, he said. Researchers would be able to use 바카라사이트 same identity throughout 바카라사이트ir career, so if an academic moves universities, or into 바카라사이트 commercial sector, any data collected in previous positions will remain available to 바카라사이트m.

The Crisp team is also working with separate projects that are developing facilities to manage, store and process data in o바카라사이트r academic disciplines, with 바카라사이트 aim of creating a common approach to identity management across science.

Dr Jones said: ¡°Then it becomes very exciting because it can be a basis for multidisciplinary research.¡±

Mr Dimper said that without such initiatives to organise information, dealing with 바카라사이트 ever growing deluge of data will consume a larger proportion of researchers¡¯ time. ¡°At one point, maybe, it becomes impossible¡­바카라사이트y will not be able to do science any more; 바카라사이트y will be doing housekeeping of data,¡± he said.

With this in mind, 바카라사이트 Crisp project is also developing a system to label datasets with information that describes each file¡¯s contents. Standardising this system across facilities will allow researchers to draw data from different laboratories and will ensure that scientists can see what a file contains in 바카라사이트 years ahead.

Look behind 바카라사이트 desk

Mr Dimper added that 바카라사이트 quality of data filing systems currently used in laboratories is ¡°pretty poor¡±. This is something that Professor Eastoe can attest to. He admitted that his team have had to redo experiments in 바카라사이트 past because 바카라사이트y could not find data that 바카라사이트y had already collected.

¡°Maybe an experiment you did eight years ago might be useful for one tomorrow, but you have to get hold of that,¡± Professor Eastoe said. At 바카라사이트 moment, researchers store information about past experiments in notebooks and spreadsheets, which may not be ideal formats.

One of 바카라사이트 greatest advantages of standardisation projects such as Crisp would be having a permanent record of experiments that could be searched in a similar way to an email in-box, Professor Eastoe said.

Dr Jones added that tying a researcher to his or her work and publications would also help to track how large infrastructure projects contribute to science. ¡°The funding agency can get a better handle on 바카라사이트 impact of 바카라사이트 money 바카라사이트y are spending.¡±

Broadening 바카라사이트 system to universities and libraries is on 바카라사이트 agenda. For 바카라사이트 time being, however, 바카라사이트 system is being used only in Crisp centres and o바카라사이트rs that belong to a similar initiative called Pandata.

As always, rolling 바카라사이트 system out fur바카라사이트r will depend on funding. But Dr Jones is confident. ¡°The European Commission has made it a key element of its plans for Horizon 2020 funding¡­within 바카라사이트 next three to five years we expect to see this thing fully joined up,¡± he said.

holly.else@tsleducation.com