ReproducibiliTea Podcast

Episode 3 - Questionable Research Practices

This week we discuss the prevalence of questionable research practices in John, Lowenstein, & Prelec “Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling” http://journals.sagepub.com/doi/abs/10.1177/0956797611430953

Highlights:

[2:00] What are questionable research practices?

[3:00] QRP as a political term

[6:00] a pinch of transparency

[8:00] QRPs as a grey area?

[12:00] Does defining a QRP as ‘just’ questionable water down the message of ‘this is wrong’?

[14:00] Do John, Lowenstein, & Prelec practice what they preach in this paper? We discuss the methodology

[19:30] Figure 1 - What do we see? “Everybody’s doing it”

[22:00] Passing on QRPs throughout the (research) generations

[24:30] Defending QRPs

[28:00] What would the QRP rates look like now?

[29:30] You can't round down p values to be less than .05, stop cheating

[30:30] Figure 2 - “I’m fine and so are my collaborators, its other people that I doubt…”

[33:00] Our job is to be critical. We get sad about the pushback to being critical on QRPs

[35:00] Sam tries desperately to find a silver lining, Sophia advocates arson.

SpecialiTEA 1 – Tim van der Zee

In our first SpecialiTEA episode we talk to Tim van der Zee (@research_Tim) about error detection, navigating critique on social media, and the positive side of the replication crisis for early career researchers.

Highlights

[1:30] Hey Tim! What are you working on?

[2:30] Tim: I am meta-science BATMAN

[3:30] Error detection: Impossible numbers in papers

[5:00] Tim’s favourite errors: we’re not naming any names, but Tim will talk about some food studies…

[10:00] can you trust the results If the basic numbers are wrong?

[12:00] But I still believe these findings…

[15:00] critiques against error detection

[18:00] Tim’s twitter feed (@research_Tim) – spot the lively debates and interesting insults

[21:00] Effective communication

[22:30] Should we be allowed to share data? Is it ever ethical? Yes, of course it is.

[24:00] How do you navigate critical feedback, Tim?

[26:00] Dr. Hannah Hobson’s (@HannahMBuxton) question “Do you think some fields have the reproducibility issue addressed already? If so what could learn from them (speaking as a psychologist)?”

[28:00] We need the crisis, it’s a good thing. We should be questioning things.

[31:00] Tim’s amazingly positive outlook

Music credit: Kevin MacLeod - Funkeriffic

freepd.com/misc.php

Episode 2 - Examining analytic flexibility

This week discuss analytic flexibility in “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant” from Simmons, Nelson, & Simonsohn (2011) http://journals.sagepub.com/doi/abs/10.1177/0956797611417632

Highlights:

[1:00] What is a false positive anyway?

[3.30] Dead salmon fMRI study http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf

[4:30] Pregnant men as our example of false positives

[8:30] Garden of forking paths

[11:00] Gelman & Lokens (2013) http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf

[18:00] precognition - we needed results that broke physics to show us the problems in psychology

[21:00] Table 1 and simulating researcher degrees of freedom

[24:30] “Our goal as scientists is not to publish as many articles as

we can, but to discover and disseminate truth.”

[27:30] you can always get significances with optional stopping

[29:30] March of the P-values https://www.youtube.com/watch?v=5OL1RqHrZQ8, ‘Improving your statistical inferences’ on Coursera with Daniel Lakens (@lakens)

[32:00] Solutions, inc. transparency and preregistration

Music credit: Kevin MacLeod - Funkeriffic

freepd.com/misc.php

In our first journal club podcast we discuss "A manifesto for reproducible science" from Nosek et al. www.nature.com/articles/s41562-016-0021

Highlights

[0:00] Sam butchers everybody's names, right from the start

[7:00] "The first principle is that you must not fool yourself — and you are the easiest person to fool" - Richard Feynman

[9:00] Figure 1; the best figure; the every-figure

[13:30] HARKing (Hypothesising After the Results are Known), distinguishing confirmatory and exploratory analyses, preregistration

[17:00] Power and false positives

[19:00] Button et al. "Power failure: why small sample size undermines the reliability of neuroscience" https://www.nature.com/articles/nrn3475

[26:00] storytelling

[30:00] The garden of forking paths

[36:00] Education is key - for everyone

[40:00] suggestions to improve our practices

Music credit: Kevin MacLeod - Funkeriffic

https://freepd.com/misc.php

Amy, Sophia, and Sam introduce the ReproducibiliTea Journal Club Podcast - where we serve up mugs of ReproducibiliTea in blends of transparency, openness, and robustness with a spoonful of science.

In upcoming episodes we will cover (the pdfs can also be found at https://drive.google.com/drive/folders/1dpJ7LBLjXIBhiWBJMlG-B-UgziJxyS7D);

1. A manifesto for reproducible science https://www.nature.com/articles/s41562-016-0021

2. False-Positive psychology http://journals.sagepub.com/doi/abs/10.1177/0956797611417632

3. Measuring the prevalence of questionable research practices with incentives for truth telling http://journals.sagepub.com/doi/abs/10.1177/0956797611430953

4. Estimating the Reproducibility of Psychological Science http://science.sciencemag.org/content/349/6251/aac4716

5. Is the Replicability Crisis Overblown? Three Arguments Examined http://journals.sagepub.com/doi/pdf/10.1177/1745691612463401

6. Open Science: What, Why and How https://psyarxiv.com/ak6jr

7. The Natural Selection of Bad Science http://rsos.royalsocietypublishing.org/content/3/9/160384

8. The Preregistration Revolution http://www.pnas.org/content/pnas/early/2018/03/08/1708274114.full.pdf