The aim of this project is to study whether researchers can accurately predict which of the experiments selected for the Brazilian Reproducibility Initiative will be successfully replicated. This will be done in two parts: a survey of beliefs and a prediction market. Separate surveys and markets will be held for each of the 3 methods in the first wave of experiments in the Brazilian Reproducibility Initiative (MTT assay, RT-PCR and elevated plus maze). You can choose to participate only in the survey or in both the survey and prediction markets. The study is being coordinated by Tiago Bortolini (Instituto D'Or de Pesquisa e Ensino - IDOR), Anna Dreber (Stockholm School of Economics) and Olavo Amaral (Universidade Federal do Rio de Janeiro) and has been approved by the Research Ethics Comittee (CEP) at IDOR.
Survey of Beliefs
In the survey of beliefs, you will be asked for (a) the probability that each of the 20 studies using each method will be successfully replicated (defined by the finding of a statistically significant result, defined by p < 0.05, in the same direction as in the original study in a meta-analysis of the three replications) and (b) the expected effect size for the replication. This will be done via an online questionnaire collected through SurveyMonkey and should take between 1 and 2h to complete for each set of 20 experiments.
Prediction Markets
In the prediction market, participants will bet on the possibility that each study will replicate successfully (using the same criterion described above). All participants will get US$50 in credits on their prediction account to trade with. By trading on ‘stocks’ for each study (similar to those in a stock market), which will be paid only when a study replicates, participants can earn more or less money depending on how they bet on different hypotheses. On average participants, will earn about $50 as Amazon gift cards, and payments will be resolved once we finish obtaining replication results (due in 2021). Prediction markets will be held for 2 weeks via an online platform from Cultivate Labs (
https://www.cultivatelabs.com). After finishing the survey, bets can be done in less than an hour and updated as desired throughout the 2 weeks.
If you are a researcher or have had previous experience with experimental research (including as a student) you are eligible to participate and can fill our preliminary registration below. Once the study begins - which is expected to happen in mid-November for the survey and early December for the markets - we will e-mail you a link to formally enroll in the study.
If you wish, you can also read more about surveys of beliefs and prediction markets in other reproducibility projects:
-Using prediction markets to estimate the reproducibility of scientific research - Anna Dreber et al. 2015
https://doi.org/10.1073/pnas.1516179112-Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 - Camerer et al. 2018
https://doi.org/10.1038/s41562-018-0399-z-Predicting replication outcomes in the Many Labs 2 study - Forsell et al. 2018
https://doi.org/10.1016/j.joep.2018.10.009If you need further information, feel free to contact
reprodutibilidade.br@gmail.com