Joe Bak-Coleman: We didn't do a full formal treatment on how our model relates to publication bias (only file-drawers). These are distinct but there seems to be some confusion about how ~90%+ of findings could be significant if there aren't massive file-drawers and heavy publication bias. Strap in!

We didn't do a full formal treatment on how our model relates to publication bias (only file-drawers). These are distinct but there seems to be some confusion about how ~90%+ of findings could be significant if there aren't massive file-drawers and heavy publication bias. Strap in!

The replication crisis is often described as a crisis of false positives, but is it? In our (re)new(ed) preprint, we show that false positives do not explain the replication crisis, and that varying rep. rates are often explained by replication sample size. #metascience osf.io/preprints/so...

The base rate of significance implied by our model is ~50% for a study just north of N=100. Let's say a researchers tests ten hypotheses an N of 120 or so. Our model suggests a 50% chance of significance such that for ten hypotheses they observe (let's say) 5 sig and 5 non-sig. ❌❌❌❌❌✅✅✅✅✅

Not ready to give up so easy, the tweak designs, increase sample size and try the initially null results again and *maybe* a third time then publish *everything* Second round: ❌❌✅✅✅ Third round: ❌✅ Resulting in a 90% significant published lit. ❌✅✅✅ ✅✅✅ ✅✅✅

In this scenario, a null initial finding had a 20% chance of getting published. The file-drawer is small, as we might imagine they publish all three null findings for that one hypothesis that didn't work out. Four out of seventeen tests are unpublished.

That's at N=120. What if they went for N=1500? They could wind up publishing the exact same claims with no publication bias and no file-drawer. ❌✅✅✅ ✅✅✅ ✅✅✅

This is a very distinct scenario that could occur under our model but not if false positives explain low replicability. In that case, only 10% of tests are of *real* effects, often at low power. The researcher (if they're lucky) finds: ❌❌❌❌❌❌❌❌❌✅ pubmed.ncbi.nlm.nih.gov/29861517/

On the Reproducibility of Psychological Science - PubMedpubmed.ncbi.nlm.nih.gov Investigators from a large consortium of scientists recently performed a multi-year study in which they replicated 100 psychology experiments. Although statistically significant results were reported ...

That ~10% chance with a new idea sure beats 5% with a previous null so they're better off testing new things. To get those 9 sig findings? ❌❌❌❌❌❌❌❌❌✅ ❌❌❌❌❌❌❌❌❌✅ ❌❌❌❌❌❌❌❌❌✅ ❌❌❌❌❌❌❌❌❌✅ ❌❌❌❌❌❌❌❌❌✅ ❌❌❌❌❌❌❌❌❌✅ ❌❌❌❌❌❌❌❌❌✅ ❌❌❌❌❌❌❌❌❌✅ ❌❌❌❌❌❌❌❌❌✅

Post