How do I 'do' Open Research?
On this page:
We need measures to counter the natural tendency of enthusiastic scientists who are motivated by discovery to see patterns in noise.
It's all about transparency
The guiding principle for practising open research is transparency. With transparency should come reproducibility, rigour and trust. These are not inevitable consequences, but a default setting of transparency in research creates the necessary condition for better research.
Opportunities for transparency can be found all across the research lifecycle and in every research area. They are present in the initial conception of a research idea, in the planning of a project, data collection and sourcing of information, analysis and dissemination. Following a critical examination of research in your field, you might feel that transparency is not the default and you are probably right. However, the open research movement, supported by governments, funding bodies, research institutions and the public, are seeking to change that.
Below is a list of some of the ways that a researcher can increase the openness and transparency of their research and with these, produce better research that benefits themselves, their research community and the public.
These pages have been written with the aim of applying to the widest range of researchers. Inevitably, the content will reflect some fields better than others. However, open research is a broad and rapidly evolving movement and it is very likely that some form of open research community exists in your area.
What are pre-registration and Registered Reports?
A pre-registration or study protocol is a time-stamped record of decisions around study design, methods and analysis, that is created before data are collected or become accessible. The pre-registration document should be publicly available on a registry or repository.
A Registered Report is a type of journal article that involves peer review of the background, study design, methods, and analysis plan (i.e., the stage 1 manuscript) before data are collected. If the proposed study is accepted by the journal (receives in principle acceptance), the study’s results will be published regardless of the outcome, pending a second stage of peer review to verify that the researchers followed their study plan.
Pre-registrations and Registered Reports adhere to the principles of open science by encouraging transparent pre-specification of outcomes, methods and analyses, and (for Registered Reports) by stimulating peer discussion early in the lifecycle of scientific work.
How can pre-registrations and Registered Reports benefit research?
Pre-registrations and Registered Reports can help reduce questionable research practices such as HARKing (Hypothesising After the Results are Known) and p-hacking (collecting or selecting data or conducting statistical analyses until nonsignificant results become significant), and can decrease false discovery rates (Munafò et al., 2017). Pre-registrations and Registered Reports may also prevent biases such as selective reporting of outcomes or statistical analyses (e.g. based on statistical significance) because the expected outcomes and planned analyses are time-stamped and publicly available.
A primary purpose of pre-registration is to make a clearer distinction between planned, confirmatory hypothesis tests and unplanned, inductive discoveries or hypothesis-generating research. Pre-registration and Registered Reports are not meant to hinder or devalue exploratory research and inductive discovery; exploratory analyses can always be included, as long as they are marked clearly as such. Rather, pre-registration and Registered Reports disincentivize researchers presenting exploratory research as if it were confirmatory; and, thus, error rates are controlled.
Registered Reports can also help alleviate a phenomenon called publication bias or the file drawer problem (Rosenthal, 1979). That is, studies that do not produce a statistically significant result are less likely to be published than those that do produce a statistically significant result. Registered Reports facilitate the publication of statistically non-significant results, as the decision for publication is made before results are known (i.e. “in-principle acceptance” or Stage 1). This model eliminates bias towards results that are positive, novel, and eye-catching. At Stage 1 submission, a Registered Report typically requires a statistical power analysis, which encourages researchers to conduct studies that are well-powered to detect effects and produce more precise estimates.
How can pre-registrations and Registered Reports benefit researchers?
Pre-registration and Registered Reports encourage researchers to devote greater effort to planning rigorous study designs that are well reported and reproducible.
During analysis and write-up, having a pre-registration as a guide both serves as a useful memory aid, and can help keep authors from engaging in questionable research practices such as “data fishing,” p-hacking, and HARKing. It also streamlines the process of analysis. The process of pre-registration can help researchers build a reputation for openness and transparency. Pre-registration also gives researchers a time-stamped record of their ideas, allowing them to take credit for their hypotheses and predictions.
Registered Reports mitigate results-based critiques from peer reviewers, because the focus is on the research question and rigour of the methodology, rather than the novelty of the results. Registered Reports have the added benefit of expert input prior to data collection. This helps mitigate the worst effects on research quality of a competitive “publish first” research system and, instead, helps researchers and other stakeholders move towards a more collaborative system of open science.
How to: Pre-registration
Pre-registration requires submitting a study plan to a public, openly accessible registry (e.g. OSF: Open Science Framework) or, for some disciplines, publishing a full study protocol in an academic journal (e.g. BMJ Open) prior to data collection. To help create a complete study plan, there are a variety of online templates, tailored to specific disciplines and types of research designs (see Table 1 for full list).
Templates differ greatly in the level of detail required and the specific sub-headings used, but the common goal is to make the study plan transparent prior to data being collected/accessed. The OSF, PROSPERO, and AsPredicted templates include examples of information that should typically be included in a pre-registered study plan, e.g. conditions, all dependent measures, sample size, etc.
The purpose of this generic study design template is to create a concise study plan that is easy for a reader to use and apply. The OSF default template is detailed and provides more template instructions than AsPredicted. For those who prefer a less prescriptive template, the OSF has an “Open-Ended Registration” option.
More recently, pre-registration formats to boost the credibility of qualitative research have also been proposed (Haven and van Grootel, 2019; Jacobs, 2018; Kern and Gleditsch, 2017).
For some pre-registration templates (e.g. AsPredicted) it is best to complete the forms offline and then copy and paste answers into the online form. On OSF and PROSPERO, researchers are free to save their document in draft form while they continue to make edits. When the completed template is submitted, a time-stamped record of the document is made.
When researchers pre-register on the OSF, they have the option to either make the pre-registration public immediately or after an embargo period of up to four years. After the study is conducted, written up, and submitted to a journal, the link to the pre-registration should be provided to reviewers in the manuscript so that they can verify that the final study followed the preregistered study plan. There is also the option to provide anonymised view-only links to a preregistration for peer review.
Any deviations from the original pre-registration should be made explicit in the final manuscript, by being presented in a separate section, for example with the heading “Deviations to the planned study design”. Deviations from the planned methods may be necessary when problems with data become evident (e.g. missing data may result in a need for additional analyses to impute missing data) or when more advanced methods emerge. Such deviations are considered acceptable when they are made fully transparent in the final published manuscript.
Useful online resources on pre-registration best practices in psychology and other disciplines can be found here <add link> (see also van 't Veer, 2016; Claesen et al., 2019).
Additional preregistration resources and answers to FAQs are available here <add link>.
Available pre-registration templates, by main purpose and target discipline/area
| Template | Purpose | Discipline Area |
|---|---|---|
| Open Science Framework (OSF) Pre-registration | Multiple templates for preregistering wide range of studies | Any |
| AsPredicted | Standardised preregistration template | Any |
| PROSPERO | Study protocol registrations for systematic reviews with a health-related outcome | Health and Social Care, Welfare, Public health, Education, Crime, Justice, and International Development |
| International Standard Randomised Controlled Trials Number (ISRCTN) Registry | Primary clinical trial registry recognised by WHO and ICMJE | Any clinical research study |
| Bio-protocol | Peer-reviewed protocol journal that makes detailed protocols available online | Biological Sciences |
| American Economic Association Registry for Randomized Controlled Trials (AEA RCT) | Registration for randomised controlled trials | Economics, Political Science, and other Social Sciences |
| Registry for International Development Impact Evaluations (RIDIE) | Prospective registry of impact evaluations for development policies and programmes in low- and middle-income countries | Social Sciences |
| Evidence in Governance and Politics (EGAP) | Registration for experiments and observational studies | Governance and Politics |
How to: Registered Reports
The most important difference between Registered Reports and the conventional publication model is that peer review (and author revision) comes before data collection (or data access in a secondary data study), whereas in the conventional publication model, peer review comes after study completion.
The Center for Open Science maintains an up-to-date list of journals accepting Registered Reports. Journal requirements can vary: for example, some journals require ethical approval prior to the Registered Report’s submission.
Some journals also offer exploratory reports (McIntosh, 2017), which are similar to Registered Reports but focus more on exploratory, hypothesis-generating studies (e.g. International Review of Social Psychology).
For a Stage 1 Registered Report, authors submit a manuscript containing the introduction (e.g. literature review, hypotheses and research questions) and methods (including analysis plans). The submitted Registered Report is formatted according to the journal’s instructions to authors and is subject to editorial and peer review.
The editor may decide to accept or reject the submission, or preliminarily accept it with minor or major revisions. As with standard peer review, further rounds of revisions may be necessary. If the Stage 1 Registered Report is accepted, the final submission will be published by the journal regardless of the results, conditional on the authors following the accepted study plan.
After the Stage 1 Registered Report is accepted, the authors can collect data, conduct analyses, and write the results and discussion sections.
The Stage 2 Registered Report can be submitted with supplementary files containing all study materials, data sets, and code. The manuscript is subject to peer review to check that the researchers have followed the accepted study plan and that the conclusions are reasonable given the data. Again, any deviation from the pre-registered study plan should be explained and justified in the final manuscript.
Substantial deviations should be flagged up to the editor as soon as possible, even before Stage 2 submission. If deviations are substantial and have not been agreed, the editor may reject the manuscript but allow its submission through the conventional, non-Registered Report publication route. The OSF provides a template outlining what to include in a Registered Report and tips for avoiding a desk rejection.
