ReproducibiliTea Podcast

Episode 9 - Did you know we love the open science community?

In this episode we tackle how the open science community is perceived, and our own experiences as members of it. We talk (in a non-specific way) about recent and recurring discussions concerning public critique and how harassment is dealt with on social media. We are happy to continue this discussion; if we have missed anything, or could have been more sensitive to specific issues let us know, we are interested in continuing the conversation. We love the open science community generally and want to reassure those unsure about open science that it is full of wonderful and helpful people.

Some links:

Start your own ReproduibiliTea journal club! Our materials are here https://osf.io/3qrj6/

Society for the Improvement of Psychological Science (SIPS) https://www.improvingpsych.org/ @improvingpsych

UK reproducibility network (UKRN) http://www.dcn.ed.ac.uk/camarades/ukrn/ @ukrepro

Links to add: Maths underlying statistics

Music credit: Alexander Nakarada – Nightmare

https://freepd.com/horror.php

SpecialiTea episode 5 – Peder Isager

We get talking to Peder Isager about his PhD work, equivalence testing, and the psychological science accelerator. You can find Peder on twitter (@peder_isager); he also has a website https://pedermisager.netlify.com/

Some Equivalence testing resources:

“Equivalence Testing for Psychological Research: A Tutorial” from Lakens, Scheel, and Isager http://journals.sagepub.com/doi/full/10.1177/2515245918770963

“TOST equivalence testing R package (TOSTER) and spreadsheet” http://daniellakens.blogspot.com/2016/12/tost-equivalence-testing-r-package.html

The psychological science accelerator:

The website https://psysciacc.org/

Twitter @PsySciAcc

Pangea – Jake Westfall: the paper (http://jakewestfall.org/publications/pangea.pdf), and the shiny app (https://jakewestfall.shinyapps.io/pangea/)

Music credit: Be Jammin – Alexander Nakarada

freepd.com/world.php

SpecialiTea Episode 4 – Leaving academia (part 1)

A few weeks ago we put out a survey to get a picture of peoples’ reasons for leaving academia and what advice they have for ECRs. We had lots of responses and dive into them here. There wasn’t quite enough time to really dig into the advice offered. But, that’s where you lovely people can come in. Our survey is still open (https://docs.google.com/forms/d/e/1FAIpQLSdad9GpUbqKebCxUSbUJXVvyOWH3gtC7zO77trFIo-SwUh-og/viewform), so if you have any advice to offer let us know there or DM us on twitter @ReproducibiliT

Some shownotes:

Oxford Reproducible research summer school https://osf.io/zk2em/

Last year’s oxford reproducible research summer school http://neuroanatody.com/category/robust-research/

Oxford-Berlin summer school on open, transparent, and reproducible research in the life sciences https://www.fosteropenscience.eu/event/berlin-oxford-summer-school-open-transparent-and-reproducible-research-life-sciences

Link to the Dorothy Bishop appreciation society (work in progress)

Psychological science accelerator https://psysciacc.org/

Seminar series on Reproducibility and Open Research http://users.ox.ac.uk/~phys1213/ReproAtASC.html

Music credit: Be Jammin – Alexander Nakarada

https://freepd.com/world.php

Episode 8 - The preregistration revolution

It’s preregistration time! Distinguishing confirmatory (or prediction) and exploratory (or postdiction) analyses. We discuss this awesome paper from Nosek, Ebersole, DeHaven, and Mellor (2018) “The preregistration revolution”. Find the paper here http://www.pnas.org/content/early/2018/03/08/1708274114

Highlights

[1:00] What is a preregistration anyhow?

[2:00] Prediction vs postdiction

[3:00] Why is preregistration important? Combatting bias; distinguishing confirmatory and exploratory analyses.

[4:45] In a perfect world, do we need preregistration?

[10:00] What else does preregistration guard against, or not?

[12:00] Getting critical; does prereg always work? Probably not, we need quality control. So, is prereg better than nothing? maybe not…

[14:30] Where are your preregistrations on the preregistration spectrum?

[16:00] Registered reports - yes, we mention them several times. Do we define or describe them, not at all…

[17:00] Is preregistration the ‘open science thing’ people are most likely to know about and use in their research, compared to other open science initiatives?

[24:00] Battle royale: preregistered and messy Vs clean and novel - which is more publishable?

[27:30] Preregistration and statistics - it’s called hypothesis testing for a reason.

[31:00] “All of a sudden we did 20 tests…”

[33:00] Challenges to preregistering your analyses

[39:00] We love preregistration, but that does not mean that it’s a simple and easy solution

[40:00] Preregistrations for existing data (SIPS hack-a-thon on secondary data analyses https://osf.io/djgvw/)

[40:00] Preregistrations for qualitative data (template https://osf.io/j7ghv/)

[43:00] Listener challenge - tell us the terrible reasons you’ve heard against preregistration

[48:00] What’s next for reproducibiliTea?

Music credit: Kevin MacLeod - Funkeriffic

freepd.com/misc.php

This week we talk about “The natural selection of bad science” from Paul Smaldino and Richard McElreath. We get bummed, but try to put a positive spin on things. You can find the paper here http://rsos.royalsocietypublishing.org/content/3/9/160384

Highlights

[1:00] “The natural selection of bad science” http://rsos.royalsocietypublishing.org/content/3/9/160384

[5:00] the need to distinguish yourself as a scientist - innovative and/or high output?

[6:45] Our next SpecialiTEA - we want to hear your experiences about leaving academia

[11:00] “when a measure becomes a target it ceases to be a good measure”

[13:30] could quality control help ‘fix’ metrics?

[15:00] Competition - how quickly can you move from open science topic A to B purely with ranting?

[18:00] I have the powerrrrrrrrr (I wish soundcloud would let me add gifs - millenial problems)

[26:00] We get a little confused about power in the simulations - we highly recommend reading the paper to get the details right :)

[29:00] Challenge to listeners - grab Smaldino & McElreath’s code and figure out how to create a scientific utopia!

[31:30] The simulation

[32:30] Watching out for forced retirements…

[35:00] Sam is super millenial and reads out this awesome tweet about terrible things in science https://mobile.twitter.com/aidybarnett/status/1036392482139865088

[37:00] Not enough detail in original papers to attempt to replicate in The Reproducibility Project: Cancer Biology (https://osf.io/e81xl/wiki/home/), also a thread from @chrisdc77 https://twitter.com/chrisdc77/status/1024589364238667776

[42:00] “The continuous degradation of scientific process”

[44:00] Does promoting replication attempts solve the problem?

[48:30] So, what the hell can we do to stop the decline of science? Community, hiring and funding shifts, and more

[56:45] Observed power

Music credit: Kevin MacLeod - Funkeriffic

freepd.com/misc.php

SpecialiTea 3: Julia Rohrer

We talk to Julia Rohrer (@dingding_peng)about her blog, skills training as a PhD and what doesn’t get talked about enough in the Open Science Community. You can find her amazing blog here: http://www.the100.ci, where she is joined by Anne Scheel (@annemscheel), Ruben Arslan (@rubenarslan) and Malte Elson (@maltoesermalte).

We want to note that this podcast had the world’s most WEIRDEST Skype setup as we forgot our headphone splitters. This meant that both Amy and Sophia were using separate headphones and were on muted Skype calls to each other while Sam was on the main Skpye call to Julia. Both Sophia and Amy heard themselves talking with a 1 second lag time, so we will not repeat that again soon (hopefully)!

The episode references to the following blog posts:

Julia’s blog post on Imposter’s Syndrome, which Amy thinks is actually the Bystander Effect (?? is she actually a social psychologist ??): http://www.the100.ci/2018/08/02/we-should-all-feel-a-bit-more-like-impostors/

James Heather’s blog post on skills building during your PhD, and other things you should know before starting one: https://medium.com/age-of-awareness/12-thing-you-should-know-before-you-start-a-phd-9c064a979e8

The loss of confidence project: https://lossofconfidence.com

Rebecca Willén @rmwillen has been publishing disclosure statements for her previous research. These statements make the past more useful, given that QRPs are the norm, and this transparency should be applauded. Rebecca’s publication list, with disclosure statements here https://rmwillen.info/publications/

Network of Open Science Initiatives in Germany: https://osf.io/tbkzh/

Music credit: Kevin MacLeod - Funkeriffic

freepd.com/misc.php

This week we talked about “Open Science: What, Why, and How” from Spellman, Gilbert, and Corker. You can find the paper on OSF https://osf.io/gv6r4/

0:14 Sophia is leaving Oxford :( (But the Podcast will continue)

0:48 This week’s focus: Open Science What Why and How; few in the JC read it but the discussion was awesome https://osf.io/gv6r4/

2:15 Shoutout to Matt Jaquiery @MJaquiery

2:40 Broad Meaning of Open Science -- what do we mean; potentially misleading to include things beyond Open?

5:10 The Centre for Science that’s Actually Science

6:05 Focussing on Open Data as “open signalling”

7:30 What even is Open Data? Criteria for Open Data; How it can go wrong

10:25 Open vs Usable Data

11:10 FAIR Guidelines -- Findable Accessible Open Source Interoperable Reusable

12:20 (Advantages of) Open Code

15:30 Why is Open Science just Science Done Right?

16:00 Answer Sam: Open Code -- Show Your Working

17:00 Answer Amy: Work cumulatively in order to avoid waste of taxpayers’ money

18:20 Answer Sophia: being open about subjectivities; slightly tautological argument of Open Science just is Science

21:10 Better system for citing code!

23:20 Not sharing data? Is it selfish?

27:30 BREAK

29:00 Shoutout to Remi Gau; Amy will be singing for you

29:40 Amy and Sophia compete for supremacy in the Table 2 challenge - what are the problems and solutions at each stage of the research process?

31:10 Challenging Two Psychologists Four Beers to the Crossover Event 5 Psychologists 50 Shots

32:00 Competition Begins: 1. Research Planning

34:35 AMY IS THE QUEEN

35:00 Competition part 2. Data Collection and Methods Reporting

36:30 Sam has a picture of James Heathers next to his bed (aka. Amy and Sophia try to make Sam look like a creep)

37:45 Competition part 3. Data Analysis and Reporting

38:54 https://jasp-stats.org

39:25 Competition part 4. Publication Process

41:30 Open Peer Review -- Do you have any good or bad experiences of this?

42:40 Competition part 5. Storing and Archiving

44:10 Why do we give publishers the chance to adapt to openness, when we should just get rid of them?

49:55 Amy wins 14-10, but is also the kindest <3

51:30 Give us your suggestions for cool ECRs/baby researchers

Music credit: Kevin MacLeod - Funkeriffic

freepd.com/misc.php

Episode 5 - Is the replicability crisis overblown?

This week talked about Pashier & Harris’s (2012) “Is the replicability crisis overblown? Three arguments examined.” http://journals.sagepub.com/doi/10.1177/1745691612463401

Highlights:

[1:00] Reproducible, replicable, robust, or generalisable? Table taken from Kirstie Whitaker, shared by @RemiGau http://sx.doi.org/10.6084/m9.figshare.4244996

[2:00] Argument 1: False Positive Rate

[3:00] False Positive Psychology - http://journals.sagepub.com/doi/abs/10.1177/0956797611417632

(which we mention more about in Episode 2)

[4:45] Dorothy Bishop on data simulation - http://deevybee.blogspot.com/2013/06/interpreting-unexpected-significant.html

[7:15] AMPPS paper Amy mentions - The Prior Odds of Testing a True Effect in Cognitive and Social Psychology

http://journals.sagepub.com/doi/pdf/10.1177/2515245918767122

[9:00] Shiny app to play around with power - https://lawsofthought.github.io/replication-crisis-demos/

[12:00] Precognition - https://slate.com/health-and-science/2017/06/daryl-bem-proved-esp-is-real-showed-science-is-broken.html

[14:00] Most published research findings are false - http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

[17:00] Argument 2: Replications

[19:00] Cyberball paradigm - https://www.youtube.com/watch?v=OwQ_VyOUGmY

[23:00] Flexible Measurement in aggression task - http://www.flexiblemeasures.com/crtt/

[26:45] Argument 3: Self-Correction

[29:30] Chris Chamber’s Book: https://www.amazon.co.uk/Seven-Deadly-Sins-Psychology-Scientific/dp/0691158908

[32:00] We have strong opinions on whether rigour and the credibility revolution “stifles creativity and innovation”.

[35:00] are these three arguments still being made?

[37:30] Dorothy’s blogpost on sowing seeds of doubt http://deevybee.blogspot.com/2018/05/sowing-seeds-of-doubt-how-gilbert-et.html

[40:00] We all (including senior academics) need to continue learning and let the revolutionaries pave the way forward.

Music credit: Kevin MacLeod - Funkeriffic

freepd.com/misc.php

Sophia and Sam headed to this year's Society for the Improvement of Psychological Science (SIPS) meeting. We got talking to some super interesting and awesome researchers (follow them on twitter, handles below) to share the experience with all of you!

Check out https://osf.io/ck28s/ for all things taught and created at SIPS 2018

Highlights:

[0:00] What is SIPS anyway?

[8:00] Elizabeth Page-Gould (@page_gould) and Michael Inzlict (minzlicht) - Creating and finding open science jobs

[15:30] Kristen Lane (@kristen_a_lane) and Heather Urry (@HeatherUrry) - Teaching reproducible research

[19:30] Jessica Flake (@JkayFlake) and Eiko Fried (@EikoFried)

[26:00] Michele Nuijten (@MicheleNuijten), Nick Brown (@sTeamTraen), and James Heathers (@jamesheathers) - checking yourself (before you wreck yourself), statistically

[38:00] Sanjay Strivastava (@hardsci), Alexa Tullett (@alexa_tullett), and Simine Vazire (@siminevazire)

[49:00] Wrap up

Music credit: Kevin MacLeod - Funkeriffic

freepd.com/misc.php

Episode 4 - Reproducibility Now

This week we dive into the Open Science Collaboration’s (2015) paper “Estimating the reproducibility of psychological science”

http://science.sciencemag.org/content/349/6251/aac4716

Highlights:

[1:00] This paper has all of the authors

[1:30] Direct vs conceptual replications

[4:30] PhD students running replications as the basis of extending a paradigm

[6:00] The 100 studies paper methods in brief

[8:00] everything’s available for this collaborative effort, and that is awesome

(https://osf.io/ezcuj)

[9:00] Reproducibility vs Replicability - what are we actually talking about

[9:30] Oxford summer school in reproducibiltiy

(https://www.eventbrite.co.uk/e/oxford-reproducibility-school-tickets-48405892327)

[11:00] paper discussing the computational reproducibility of papers

[15:00] Replication is not only about the p value folks!

[17:30] Sam brings up Bayes purely to be a douchebag

[19:30] A bayesian approach - Sophia gives us the paper (http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0149794) and we move on

[20:00] Replications as a method to diagnose problems in science. Are replications a viable problem solver?

[24:00] Psychology is only a teenager really

[26:00] If the original paper is trash, there’s probably no need to replicate it. Maybe just burn it down?

[27:00] Figure 1 - the average effect size halved in the replication attempt and most effects did not replicate.

[31:30] Do the results hint at more than publication bias? Are other QRPs involved?

[33:00] Comparing reproducibility across subfields of psychology. But, are these studies representative of an entire subfield

[35:30] Does journal impact factor mean anything?

[39:30] Are we actually being critical of previous research in general?

[41:00] “Our foundations have as many holes as a swiss cheese”

Music credit: Kevin MacLeod - Funkeriffic

freepd.com/misc.php