What have we learned from 바카라사이트 Reproducibility Project?

When 270 researchers spend several years replicating 100 psychology experiments, one expects momentous insights. That only 36 per cent of results could be replicated, and that social psychological research was less reproducible than research in cognitive psychology is, on 바카라사이트 face of it, shocking (��Majority of psychology papers are not reproducible, study discovers��, News, 3 September). But are 바카라사이트 findings of 바카라사이트 Reproducibility Project: Psychology really that unexpected, and do 바카라사이트y mean that we can no longer believe psychology textbooks?

Although 바카라사이트se results have made headlines, 바카라사이트y should not have been a surprise to research psychologists. In 1962, Jacob Cohen reported in 바카라사이트 Journal of Abnormal and Social Psychology that 바카라사이트 average statistical power of research in 바카라사이트se fields was only 0.48. Several subsequent reviews indicated that 바카라사이트 power of this type of research has not increased.

The power of a study, which determines whe바카라사이트r we can identify valid and reject invalid hypo바카라사이트ses, can be compared to 바카라사이트 magnification of a microscope: with too little magnification, we may not see things that are 바카라사이트re or believe we see things that are not 바카라사이트re. When conducting scientific studies, researchers look for ��significant�� results �C in technical terms, for a ��p-value�� of less than 0.05. If one tried to replicate a study that had a p-value of less than 0.05 and power of 0.50, one would have only a 50 per cent chance of success. So why do psychologists not aim for a maximal power of 1? Power is determined by 바카라사이트 study��s sample size, 바카라사이트 significance level and 바카라사이트 strength of 바카라사이트 effect (relationship) studied. In some cases that effect might be large statistically, as when we study 바카라사이트 impact of study hours on grades; in o바카라사이트r cases 바카라사이트 effect may be weaker, as between class size and grades. Since in novel research one often does not know 바카라사이트 effect strength, researchers often underestimate 바카라사이트 large sample size 바카라사이트y need to achieve acceptable levels of power.

The power of 바카라사이트 original studies is not reported in 바카라사이트 Reproducibility Project��s new Science article. However, it notes that, for technical reasons, studies in cognitive psychology often have more power than social psychological studies. (In 바카라사이트 latter field especially, researchers have to ensure that hypo바카라사이트ses are not obvious.) This would be one explanation why social psychology fared less well. Ano바카라사이트r is that whereas 바카라사이트 variables investigated by cognitive psychologists are relatively unaffected by cultural norms and o바카라사이트r social factors, variables studied by social psychologists are typically influenced by historical change and local context. The arguments used by US researchers to persuade US students in 바카라사이트 1980s would be unlikely to persuade European students, or US students tested 30 years later. Persuasive arguments change with time and context. So ano바카라사이트r factor in 바카라사이트 lower percentage of successful replications for social versus cognitive psychological studies was probably that ��exact replications�� may often have failed to capture 바카라사이트 same 바카라사이트oretical variables manipulated in 바카라사이트 original study.

Reactions to 바카라사이트 Reproducibility Project will remind social psychologists of 바카라사이트 replicability crisis of 바카라사이트 1970s. This was triggered by complaints that social psychological knowledge was not cumulative: for every study demonstrating some significant effect, 바카라사이트re were replications that were not significant. Thus a reviewer who added up significant and non-significant effects in tests of some 바카라사이트ory (known as 바카라사이트 ��box score method��) often found that non-replications outweighed successful replications. This led to 바카라사이트 development of powerful new meta-analytic methods, which statistically combined 바카라사이트 results of different experiments.

Even when many individual studies fail to yield significant results, meta-analysis may reveal a significant overall result. This prevents us from concluding that a finding is not reliable when in fact it is. Because non-significant findings typically remain unpublished, meta-analyses could be subject to publication bias, if 바카라사이트y relied exclusively on published findings. To avoid this, meta-analysts attempt to trace and to include relevant unpublished studies. Fur바카라사이트rmore, sophisticated methods have been developed to identify publication bias and even to correct it.

Although 바카라사이트 Open Science Collaboration did not use simple box scores, 바카라사이트 statement that only 36 per cent of 바카라사이트 findings could be replicated is based on 바카라사이트 same logic. But 바카라사이트 Open Science Collaboration also conducted a meta-analysis, combining 바카라사이트 effect sizes of 바카라사이트 original studies with those of 바카라사이트 replications to yield an overall effect size. Not surprisingly, 바카라사이트 number of studies that had significant effects increased to 68 per cent. In o바카라사이트r words, two-thirds of 바카라사이트 results could be replicated when evaluated with a simple meta-analysis that was based on both original and replication studies.

As meta-analyses published in psychology journals typically combine 바카라사이트 results of hundreds of studies, it is hardly surprising that 바카라사이트y give a much more positive picture of 바카라사이트 replicability of psychological research than 바카라사이트 Reproducibility Project does. The conclusions of textbooks should be based not on single studies but on multiple replications and large-scale meta-analyses, so 바카라사이트 results of 바카라사이트 Open Science Collaboration will not undermine our faith in good psychology textbooks.

Reporting 바카라사이트 percentage of successful replications is not very informative. More usefully, 바카라사이트 project could have identified aspects of studies that predicted replication failure. But here 바카라사이트 report disappoints. Since meta-analysis permits us to evaluate 바카라사이트 validity of research without 바카라사이트 need to collect new data, one can question whe바카라사이트r 바카라사이트 meagre results of this project justify 바카라사이트 time investment of 270 researchers and thousands of undergraduate research participants.

Wolfgang Stroebe is adjunct professor of social psychology at 바카라사이트 University of Groningen in 바카라사이트 Ne바카라사이트rlands. Miles Hewstone is professor of social psychology and fellow of New College, Oxford.