Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth by Stuart Ritchie

Author:Stuart Ritchie
Language: eng
Format: epub
Publisher: Metropolitan Books
Published: 2020-07-20T23:00:00+00:00

* * *

The drawback of multiverse analysis is that it usually requires the services of a supercomputer, which aren’t available to most researchers. And while analyses of this sort are a great way to bring more clarity to hotly debated questions, they don’t rid scientists of the ever-present temptation and pressure to cherry-pick their most impressive-looking results and present them as their initial hypothesis. To tackle this, we might use another tool for fixing science: pre-registration.

Pre-registration has been mandatory for US government-funded clinical trials since 2000, and a precondition for publication in most medical journals since 2005.41 Registering a study involves posting a public, time-stamped document online that details what the researchers are planning to do, in advance of collecting any data. A public repository of experiments that are about to be run provides a baseline by which to check what proportion of these studies actually make it to publication. And it allows us to see what hypotheses the researchers intended to test, so we can check if any of them were switched mid-study.

In addition to pre-registering the fact that a study will take place, researchers can also pre-register a detailed plan for how they intend to analyse the data. We’ve seen how it’s the unplanned nature of statistical analysis – the undisclosed flexibility – that can lead scientists down forking paths to results that are statistically significant (and publishable) but don’t actually correspond to reality. The idea of pre-registering your analyses is a scientific Ulysses pact: by posting a plan for your analysis somewhere public, you lash yourself to the mast and stop yourself giving in to the Siren’s call of p-hacking.

Some would rightly object that if scientists permit themselves no wiggle room whatsoever, there’s no longer an opportunity for serendipitous findings (penicillin and Viagra are two of the more famous accidental discoveries that are often referenced as part of this argument).42 But that isn’t what pre-registration is about. In a pre-registered study, impromptu analyses to explore interesting patterns in the data are still allowed, they just can’t be spun to look as if they’d been planned in advance. These so-called exploratory analyses can lead to many important new insights and ideas – for example, you might unexpectedly find that a new drug works better in older than it does in younger participants and then build up a new research line to find out why. But as we’ve seen again and again in the previous chapters, numbers are noisy and you’re guaranteed to find something interesting-looking if you slice and dice your data in enough ways. Because you’ve given yourself more tries to find statistically significant results, positive findings from exploratory analyses are much more likely to be random flukes that won’t replicate in a new sample. Yet somewhat scandalously, the majority of science frames exploratory results as though they were confirmatory; as though they were the results of tests planned before the study started. Pre-registration lets you be clear with your readers whether you were using the data

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.