| Rethinking Publication |
| Sunday, 01 October 2006 | |
Ana Vasiliu introduces the Journal of Spurious CorrelationsA number of academic journals have recently been established to make ‘negative’ results accessible in a peer-reviewed, citable form. However, the content at stake is much more diverse than simply negative results. There is a wide gap between failed randomized trials in medicine and psychology, quasi-experiments, theories disproved by empirical research, circumstantial evidence against theoretical expectations and anomalies in the social sciences. Such studies do, however, share the common factor of being embarrassing when submitted for peer review, and suffer a correspondingly higher burden of proof before their (relatively rare) appearance in print. The question is, can you tell a negative result when you see one? What is lost in the processes leading from research output to publication? What, if anything, has been done to count, name and review those findings that are felt only indirectly, as systematic shortcomings of published content? The Journal of Spurious Correlations (JSpurC), affiliated with the Standing Group on Political Methodology (SGPM) of the European Consortium of Political Research, has taken up the task of collecting, reviewing and interpreting negative results in the field of social science. JSpurC invites findings that qualify as negative, unexpected or traditionally ‘unpublishable’. As well as a journal, it is intended to be a novel exploration of the relationship between empirical research, theory and method in the social sciences. If you have ever wondered whether academic publication truly functions as a neutral accumulator of knowledge, there is actually a great deal of unexplored territory to be investigated. The effects of implicit publication standards on scientific knowledge have received little attention. Some indirect evidence comes from two vantage points on the scientific process: philosophy of science and policy. Understanding the dialogue between theory and empirical research as a methodological device is very much the territory of philosophers of science. Meanwhile, quantitative methods give us recipes for measuring publication bias, as well as helping us to ask when a result moves from being merely statistically significant to becoming ‘substantial’. The call for meta-analysis (the analysis of data from a number of independent studies of the same subject) of research results to inform policy has already produced incentives and instruments to use negative results, namely, randomized trials in medicine, education and development policy. For example, the MIT Poverty Action Laboratory investigates which methods of poverty reduction actually work, using randomized trials. Learning from failed predictions may also be valid when using the non-experimental methods of social science. To publish negative results on an equal footing with traditionally published content, we have to rethink the publication process. In a bid to bring together the practical experience with the negative results of a group of leading scholars in political science, philosophy and medicine, JSpurC and SGPM organized a short course entitled ‘Rethinking Publication’ at the annual conference of the American Political Science Association in August 2006. Direct evidence of what is lost, and where along the research process it happens, is not easy to come by; negative results, publication bias and the ‘file drawer’ problem—the tendency of academics not to bother to publish the results of ‘failed’ experiments—are often implicated. The literature gives no evidence on what is deliberately not published. A survey of nearly 200 social science academics found that ‘negative’ and ‘unpublishable’ are virtually regarded as one and the same. What keeps such findings hidden is likely to be wastepaper baskets rather than file drawers. It is more likely that priority is given to publishable results by the authors themselves during the research process, rather than by reviewers at the end. Based on the circumstantial evidence of the survey, publication bias operates most intensely well before peer-review comes into play. Unfortunately, creating an alternative forum in which to tackle publication bias does not open the floodgates of failed and filed research. Although averting the replication of ‘mistakes’, and the usefulness of all data for meta-analysis, are good reasons to publish negative results, the results may have greater potential in other areas. Publication of negative results could help inform new choices of research interests and could lead to an explicit understanding of how, through the research process, findings become positive, public and applied. Ana Vasiliu is a member of the Editorial Board of JSpurC |
| < Prev |
|---|
News Archives
| News Archives |
