OS-neuroscience

Chapter 2: Preregistration and Open Protocols

Authors: Nadza Dzinalija, Eva van Heese, Lucas Baudouin; Reviewers: Juliette Castelot, Dorien Maas

In this Chapter

This chapter explores protocols and analysis plans in the scientific process, emphasising the importance of sharing these documents openly and early. It provides practical guidance on how to disseminate preregistrations and protocols, as well as tips for finding those shared by other researchers. The first part of this chapter focuses on preregistration, an approach that is applicable to all types of neuroscience projects. The second part highlights the use of open protocols, particularly in neuroscience-related laboratory work.

The Preregistration Revolution

Progress in scientific studies relies on two processes: prediction and postdiction. Prediction involves the generation of hypotheses from existing data (exploratory), and postdiction includes testing hypotheses with new data (confirmatory). The same data cannot be used to do both, but this can happen unintentionally as the distinction between the two is generally appreciated on a conceptual level, but not always respected in practice. The blurring between prediction and postdiction reduces the credibility of research as natural biases in human reasoning (i.e. the hindsight bias) are difficult to avoid (Nosek et al., 2018). For example, a researcher may collect data on the effectiveness of an intervention, and after half the data is collected, perform a preliminary analysis that shows the intervention is effective only in one subgroup of patients. When all the data is collected, the researcher may focus on this subgroup of patients and do post-hoc analyses in this group, despite not having had a hypothesis about this subgroup before seeing the data at the half-way point. In this way, the same data (or a part of it) is used for both generating the hypothesis and testing that same hypothesis.

Research ideas and hypotheses should be defined before observing outcomes to keep a clear separation between prediction and postdiction. This is the main aim of preregistration.

Preregistration is the practice of registering research questions and hypotheses, as well as the analysis plans before or during a study. Preregistration helps researchers to distinguish between conclusions based on exploratory and confirmatory investigations. Preregistrations can be made for a variety of practical set-ups, for example, when data already exists or for new studies.

Widespread adoption of the practice of preregistration will increase the distinction between hypothesis-generating and hypothesis-testing research and improve the credibility of science.

Common Fears Debunked

Many researchers have concerns or misconceptions about preregistration that hold them back in publishing a preregistration. Understanding the facts can help overcome these fears and highlight the true benefits of preregistering your research.

Pros and Cons

Writing a preregistration comes with pros and cons. The table below sets out the most essential points to consider.

Pro Con
Separates hypothesis-generating and hypothesis-testing research → improves the credibility of your research Takes time and additional preparation
Improves transparency of your research (avoids duplicate studies, reinventing the wheel, making the same mistakes as others) Creates additional work if plans change later (changes need to be reported)
Improves efficiency and planning of your research  
Improves the quality of your research  
Aids in clearly reporting your research  
Can help when publishing null results  

Practical Decisions

Who can publish a preregistration?

Anyone! Whether you are an undergraduate student, PhD candidate, postdoctoral or more senior researcher, you can write and publish a preregistration. Regardless of whether the final product of your research will be published in a scientific journal, your analysis plan can be described and published beforehand.

Types of preregistrations

There are a few main types of preregistrations:

Platforms

Several preregistration platforms exist, each with its own (dis)advantages (see Figure 1 for an overview). These platforms are not specific to neuroscience and can be used across a wide range of scientific disciplines. A personal account is required to view the available templates and select one for your preregistration. It is generally possible to invite future co-authors to collaborate on a pre-registration, allowing you to collect input from important stakeholders in the research.

Figure 1 - Comparisons of preregistration platforms (Haroz, 2022). ★ = the criteria deemed to be a bare minimum to meet the definition of a preregistration.

Essential Elements

It is essential to include all confirmatory analyses in your preregistration. This will look different for every study, so a good rule of thumb is to imagine you were writing the method section of your paper and include anything that you would typically include there. Among others, you can consider including:

It is not essential to include exploratory analyses when you do not (yet) have a clear idea of what you will test or what results you may expect. Consider including any additional analyses you know you want to perform. The rules are not so strict as they are for confirmatory analyses, so if it is not yet clear how you will define a particular variable, or if you need to see whether specific patterns emerge from the data before you do a particular test, that can be described here. It is encouraged but not always necessary to describe the statistical analyses and hypotheses for exploratory analyses in detail. There isn’t always a clear distinction between ‘exploratory’ and ‘confirmatory’ research, and for your own study, you are responsible for where you draw the line.

Publishing after Preregistration

After you’ve completed your preregistered study and gathered your results, it’s important to share what you found, even if the outcomes are null, unexpected, or not as exciting as hoped.

As you prepare a manuscript to report your preregistered research, here are a few key things to keep in mind:

Open Protocols

We have all faced the dilemma of who to seek advice from when we wish to test a particular antibody or protocol that we have encountered in a publication. Regrettably, we may not have anyone in our circle of colleagues who can provide the guidance we need, and we may end up conducting a series of fruitless trials before arriving at a definitive outcome. However, as fate would have it, we may later stumble upon a protocol that could have spared us valuable time and resources. This situation is not uncommon, and open protocols and sharing of negative results could remedy this (see more in Chapter 7). This section focuses on protocol sharing, as protocols serve as the foundation of many research projects but are not typically featured in scientific publications.

An additional option is to search for protocols in open-access repositories. These archives allow open sharing, with no peer-review procedure and no costs for publication or registration. Another advantage is that the articles made available are assigned a DOI for citation purposes and published under a CC-BY licence. This means that anyone is free to reuse these methods. Two major players are currently involved in these types of procedures: protocol.io and Protocol Exchange. Importantly, several publishers are working in collaboration with protocol.io to make more detailed protocols from their articles available, as is the case with the Lab Protocols initiative launched by Plos One and Nature Protocols.

References

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606.

Haroz, S. (2022, February 24). Comparison of Preregistration Platforms. https://doi.org/10.31222/osf.io/zry2u