Hello. Below you’ll find your monthly injection of cynicism about the world of scientific research. Remember: it’s only by knowing about this stuff that we can do anything about it (but also it’s fun, in a true-crime kinda way). Here we go:

The links

• This would be a great article on bullshit business school “science” even without the twist ending.

• 7-17% of sentences in computer-science peer reviews are estimated to be written by AIs. Going by some of the slapdash reviews I’ve had in the past, I can’t help thinking this is in some ways a good thing…

  • But seriously: at this point letting an AI do the whole review is a dereliction of duty. But given how good they’re getting at spotting errors, doing your review without the “extra eyes” from an AI might soon be a dereliction of duty too.

• The least surprising RCT result ever: after years of fraud accusations and fines for misleading investors, Cassava Sciences finally released the results for its Alzheimer’s drug simufilam and… it doesn’t work at all. Obviously.

  • As ever, you have to wonder whether the field of Alzheimer’s research has a disproportionate level of bad science, or whether it’s just getting disproportionate attention and all fields are like that.

• I enjoyed this metaphor of the scientific literature being like an unsafe lab where you have no ability to label the faulty equipment to protect your colleagues.

• A very annoying aspect of papers getting retracted (for whatver reason) is that the retraction note doesn’t fully explain what went wrong with the article, and uses vague language about “errors” or similar. Here’s a rare instance of an inadequate retraction note getting rewritten.

• It’s still the case that effectively nobody publishes replication studies. “Based on these findings, we argue it would be premature to declare that psychology’s replication crisis is over”.

  • And on that same topic, a useful preprint on what scientists should do when they fail to replicate a result—should they try to replicate it again?

• Room-temperature superconductor fraudster finally fired. No, not that room-temperature superconductor fraudster. The other one. No no, not that one either! The one at Rochester!

• It’s been well over 100 years since the Piltdown Man hoax, so we were in need of another big scandal in palaeontology (or palaeoanthropology I guess). Here it is!

• Jesse is right here: Scientific American’s (now-departed) editor turned it into something not very scientific at all. Glad she’s gone.

  • The response when you point this out is often something like: “Scientific American isn’t a science journal! It’s just a pop magazine! Who cares if it publishes political stuff?”. But a lot of people look up to it, the same way they do to New Scientist here in the UK, which also isn’t a journal. It’s bad if stuff that’s associated with science becomes politicised, full stop.

• Sokal-style testing-the-boundaries hoax paper? The work of someone with psychosis that nevertheless got published in a “serious” journal? Whatever it is, you’ll surely join me in saying “WTF” at “Practice of neurosurgery on Saturn”.

• Coercive citation is a real thing: “As strongly requested by the reviewers, here we cite some references [[35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47]] although they are completely irrelevant to the present work.”

Subscribe now

P.S. The Studies Show

You can never predict what’ll be popular: this month people went crazy for our podcast episode on collider bias. So I guess we’ll do more stuff on weird statistical phenomena in future, and to make sure you get them in your inbox you should subscribe to The Studies Show!

P.P.S. “The other place”

Yes I know it’s cringe and all that. But it does seem like a lot of scientists have moved over from Twitter/X to Bluesky now, so to be able to keep tabs on stuff for this newsletter, I opened an account which you can follow if you want: stuartjritchie.bsky.social.

That’s it for November. Please do subscribe if you find this kind of thing useful, and I’ll see you back here in a month!

Image credit: Getty

With infinite apologies for it being a couple of days late (my excuse is that I’ve been moving house), here’s your update of all the maddest goings-on in the world of bad science for the month of October 2024.

Subscribe now

The links

• Whenever there’s a case of scientific fraud, you always feel terrible for the colleagues of the fraudster: innocent people who co-authored work where the data subsequently turned out to be fake. This article tells one such tragic tale.

• And you wonder how the colleagues of this guy feel: a chemist who’s been found to have faked data in 42 papers. If they all get retracted, he’ll end up as the 14th-most-retracted researcher of all time, which is a pretty decent showing.

• Then again, that’s nothing compared to what El Pais is calling “the biggest scandal in Spanish science”: 75 papers to be retracted from a computer scientist who seems to have obsessively and fraudulently manipulated the citation system for years. That’ll make him the 8th-most-retracted of all time.

• Latest “Jonathan Haidt vs. Social Media” update: Matt Jané, god bless him, isn’t content just to be a critic on the sidelines, and is setting up an open, transparent meta-analysis to try to provide some high-quality analyses on this question (“does social media cause mental health problems?”) at last.

• Researchers do a long-term study of the effects of puberty blockers, find that they aren’t beneficial to mental health, and then… decide not to publish the results in case they’re “weaponised”. Seems extremely bad, if you ask me.

• Do you remember when Daniel Kahneman made a big thing of having done an “adversarial collaboration” with another researcher who he disagreed with, for some research on whether money makes people happier? Turns out the article they wrote together was, er, not very good.

• Article about an intriguing new tool to spot papers that are at “high risk” of being fake (and the journals that publish a lot of them). A big “yikes” in these results for the mega-journal published by Springer Nature, Scientific Reports.

  • And while we’re on Scientific Reports, a bunch of data sleuths got together to try to give the editors a collective shake: this journal is absolutely chock-full of terribly flawed and probably fake research, and you’ve got to wonder if there’s any quality control happening at all.

  • I mean… just look at this example. Or this one. I remember when this journal first appeared, and everyone treated it like a serious outlet! Now it’s a joke.

• Remember the classic story of the scientific fraudster who coloured in the fur of a mouse with a black pen to try to claim he’d successfully done a skin graft? Well, here are some other scientists who seem to have gotten creative with a marker pen.

• If you’re a UK researcher interested in meta-science, you can now apply to get access to administrative data from the government’s main research funder.

• Couple of things about this news article in Nature about C-sections and feeding babies shit (yeah, you read that right):

  • One: why on Earth is Nature reporting on an n=31 “pilot” study that isn’t even published in readable form anywhere and was only presented at a conference?

  • Two: “Gendered language is used in this piece for clarity and to reflect the language used in the research, but Nature recognizes that not all people who give birth identify as mothers, and that not everyone who gives birth takes on the mother role”. What?!

• While we’re on the topic of woke Nature, you might’ve seen the most-ridiculed scientific article of the month: the one that urges Northern Hemisphere scientists to be more inclusive by avoiding saying things like “Summer Conference” or “Winter Meeting”, because the seasons are different below the Equator. Look, I know it’s boring to go on about this. But it’s funny that Nature seems to be stuck in an eternal 2020, long after everyone else has realised how cringe all this stuff is.

• “Can the retraction of articles endanger the mental health of researchers?”. Awww, did that big bad wetwaction hurt your widdle feewings? Try being someone with Alzheimer’s whose treatments were slowed down for years because fraudulent academics didn’t retract their fake results.

• “…we show that rhesus macaques, who have no knowledge about political candidates or their policies…”—just imagine writing that in a scientific article. Imagine it and then imagine just giving up being a “scientist” and instead getting a job that actually provides some value to society. Which is what the authors should immediately do in this case.

Subscribe now

P.S. The Studies Show

Our podcast episode about Philip Zimbardo (who died this month) and his Stanford Prison Experiment is one of our best yet, if I do say so myself. We also did a silly episode on psychic mediums for Halloween, as well as much more serious one on the science around surrogacy, and one on the Ig Nobel Prize-winning research debunking “Blue Zones”.

Farewell, friends. See you in the next one. Don’t forget to subscribe below to get these increasingly dyspeptic newsletters in your inbox:

Image credit: Getty

Hello. Every month that I send this newsletter out, I think “wow, a lot of stuff happened this month!”. That’s just because there’s a never-ending series of stories about frauds and sloppy researchers ruining science—and also of people doing their best to keep science honest.

If you want to keep up, the only way is to subscribe. You can do so below—and I’d also love it if you shared or forwarded this newsletter to someone you know who might also enjoy it.

Subscribe now

Share

The links

• You know how much I support the science reform movement—people doing replication studies, advocating pre-registration, all that kind of thing. It makes it all the more embarrassing, then, when prominent members of that movement collectively shit themselves in public.

  • The whole thing is such a catastrophic mess—a years-long process of bad study design, lack of transparency, backtracking, over-claiming, and eventually retraction—that you’ll probably have to read multiple articles to understand it. You can find a summary by Andrew Gelman here and one by Stephanie Lee here.

• This month’s massive new fraud story. This Alzheimer’s/Parkinson’s researcher was the head of the US National Institute on Aging’s Neuroscience Division, and is highly influential—“one of the most cited scientists in his field”. Now 132 (one hundred and thirty-two!) of his papers have been found to have suspicious images. As one interviewee put it: “There had to have been ongoing manipulation for years.”

  • And as Derek Lowe puts it: “Fraud, so much fraud”.

• More fraud, I’m afraid. An obstetrics and gynaecology researcher has had loads of his papers flagged for data integrity concerns—a “mere” 130 of them this time. And this time it’s for numbers that are suspiciously similar across multiple tables. As ever, fraudsters leave behind blatant evidence of their misconduct, if you know where to look. Or at least, the incompetent ones do. Don’t think too much about the ones who are good at covering up their misdeeds…

• This, however, is great news on the fraud front. You’ll recall that Harvard professor Francesca Gino tried to sue critics who pointed out fraud in her work. If she’d won that $25m case, it would’ve been very bad news for data sleuths and those who want to promote scientific integrity—you’d always be at risk of financial ruin if you dared to highlight fake findings. That’s why it’s fantastic news that her case against the critics was dismissed this month.

  • There’s a great summary of the story and a directly-reported account of the court hearing in this new New Yorker article.

• Relatedly, given he was one of Gino’s co-authors: “another shoe just dropped in the ongoing Dan Ariely scandal”. He seemed to think the whole thing was over; now his most-cited paper has an editorial expression of concern.

• This is cool: a database where you can match original findings from psychology with their replication attempts.

• Oh, and on that topic… yet another in a long line of attempts to discover the underlying psychological reasons why some people are liberal and some people are conservative—the idea that right-wingers have a “negativity bias”—fails to replicate.

• We knew about this one already, but the idea of “stereotype threat” on women’s maths performance doesn’t replicate either, and this has just been confirmed in a very high-quality study.

• Replication crisis “not just in psychology”, exhibit 981,345: there are big problems in the literature on using AI to solve partial differential equations (very important for understanding and modelling all kinds of physical phenomena from aircraft flight to the climate; in this case focused on fluid dynamics). Figure 1, showing reality vs. what we see in the scientific literature, is brutal.

• Famous study says black babies have better survival prospects when they’re cared for by black, as opposed to white, doctors. Re-analysis comes along and controls for low birth weight, which wasn’t included in the original analysis. The effect almost entirely goes away: there’s very little, or no, impact of the race of the doctor. More good news! The world is actually a bit less unfair and racist than we thought.

• Amusing (or would be, if it wasn’t so depressing) further investigation into the work of dodgy-as-hell researcher Nicholas Guéguen. It is unbelievable that people have to spend time debunking this obviously fake stuff posing as scientific research, when supposedly professional reviewers and editors couldn’t see it.

• You might’ve seen that there’s been an antitrust case filed against some of the biggest scientific publishers. Generally scientists hate the publishers, for many good reasons, so lots of people were saying “yeah! Awesome! Take them down!”. But Dorothy Bishop explains why the case doesn’t make sense—it attacks the publishers for doing stuff that isn’t actually bad.

• Newspapers and news websites get rightly criticised for doing “stealth corrections”—changing an article they’d previously published but without acknowledging it anywhere. Well, turns out scientific journals do exactly the same thing.

• A nice summary of the problems with science publishing and some of the tools to spot dodgy publications.

• This is bad: the Institute of Physics trying to push back against “unprofessional” reviews that hurt the feelings of the scientists who get them. Sure, it’s normally best to avoid being nasty for the sake of it, but I predict that many scientists will just use this as an excuse to shrug off genuine criticisms of their work.

  • Some of the “nasty” reviews they highlight aren’t even bad at all: “I do not like the paper as written”?! Are we supposed to feel sympathy for the poor soul (read: high-powered professor; Australia’s Chief Scientist) who received this horrifically insulting remark?

• This one is really baffling. There are dozens of papers out there that just carelessly use the wrong name for a drug used in pregnancy, replacing it with one that hasn’t been tested and could be dangerous. Who knows how many doctors have given their patients the wrong drug based on all the confusion?

• Feeling a bit lazy this month? Sorry to make you feel bad: here are the scientists who have “published at least a paper every two days in 2024”. What’s stopping you from engaging in this frankly absurd, literally unbelievable level of productivity?

Subscribe now

P.S. The Studies Show podcast

Quite a few people have found our podcast on the scientific publication system—how it works in theory, how it works in practice—quite useful. There are also episodes where we talk about Jonathan Haidt vs. social media, the effects of alcohol, and why I’m a toxoplasma truther.

That’s it for September. See you for the extra-spooooooky Halloween-themed edition next month! [disclaimer: may not be spooky or Halloween-themed]

Image credit: Getty

WHAT? It’s a list of links about bad science—fraud, error, poorly-designed studies—that I’ve collected over the past month.

WHY? If we don’t understand how bad things can get, we won’t be able to find realistic ways to make them better.

WHERE? It’s right below. Just scroll down a bit.

WHEN? In your own time. You’ve got T-minus one month before the next one.

WHO? I don’t really understand the question, but if you want to tell other people about this newsletter, I certainly won’t stop you. Look, there’s a special “share” button and everything. Go wild:

Share

Subscribe now

The links

• A meta-analysis found that in those experiments where people are made to stay off social media, there’s not much evidence of improvements in mental health. Jonathan Haidt then did a re-analysis, claiming that the experiments do in fact show a causal (negative) effect of social media on wellbeing. But, as Matt Jané convincingly shows here, Haidt messed up the stats, and his re-analysis shows nothing of the sort.

  • On that same topic, here’s another critique of a bad “social media effects” study.

• Richard Van Noorden’s stuff is always worth reading. This article, on the bizarre papers where up to 60% of the citations are to retracted studies, is no exception.

• Another useful survey study where scientists are asked if they’ve ever used any “questionable research practices” like coming up with their hypothesis after analysing the data or selectively reporting data. Predictably, large numbers say they have.

• Here’s a strange one. Turns out that a quarter of studies that use a scanning electron microscope misidentify the microscope—that is, they include the wrong manufacturer or model name (the study authors liken it to having done all your analysis in R, but writing in your paper “we did all our analysis in Python”). Why would this happen? Well, one possibility is that the studies were never done in the first place, so all the information in them is just garbled nonsense.

• Another example where—perhaps for political reasons—peer-reviewers criticise a study because of its results, not its methods. Doing work on racial discrimination is a total minefield, and it seems to me that basically every study should be a Registered Report (journal commits to publishing the study on the basis of the methods before any data are collected).

  • TBH, I think basically every study in every field should be a Registered Report, so I’m not making an exception here really.

• Part 2 of an ongoing series from James Heathers describing “the biggest screwup in the entire history of academic publishing”: how a big, “respected” publisher (Wiley) acquired a smaller publisher (Hindawi) for $298m… and didn’t notice for nearly two years that their journals were stuffed full of entirely fake, fraudulent papers.

  • They never learn: here’s a story of pretty much the exact same thing (though on a smaller scale) happening at Sage.

• More good stuff from Chris Said, this time a pithy little post on compensating scientific whistleblowers.

• In Vox, Kelsey Piper describes a case of scientific fraud from 2014 (which, I have to admit, I hadn’t heard of!) that might’ve killed many thousands of patients.

  • A nice reminder that it’s not just an epistemic insult when someone fakes research data: if those data start getting included in medical meta-analyses, the consequences become very real.

• Nice to see even more pushback against what might be called the “naive view” of misinformation (held by many highly-credentialed researchers!).

• A biology professor who was the head of a department at University of Maryland was found to have faked data in 13 papers and 2 grant applications. His punishment: he’s not allowed to work for the government or apply for grants for 8 years, and he has to retract the papers that haven’t already been retracted. Look, I don’t want to get into “cancel culture” or anything, but… fire this guy ASAP!

• Causal estimation methods are taking something of a beating this month: a paper on the many issues with using the classic instrumental variable of rainfall; a paper discussing the many issues with difference-in-difference analysis.

  • And this isn’t formally a regression discontinuity analysis, but it is kind of, and it’s still fun to read Andrew Gelman’s post where he calls it “absolute crap”.

• A paper complains that PubPeer comments about scientific misconduct are leaving universities with a “paralytic burden” of investigations. How about criticising the scientists, and not the people pointing out their fraudulent work?

• This is kind of a positive retraction story: some quantum physicists had their own paper retracted from Science because they found that they’d inadvertently made errors in the analysis, and were honest enough to admit it and correct the record. Big respect for that—but the retraction note that explains the errors is extremely terse and I feel like readers deserve a much more detailed explanation of what went wrong.

• And finally… if, like me, you enjoy the absurd things scientists write when their result doesn’t quite reach statistical significance, you’ll love this one.

Subscribe now

P.S. The Studies Show

Look, if you don’t subscribe to my podcast with Tom Chivers, you’re missing out. There have been multiple reports that a certain moment in our recent episode on the marshmallow test caused listeners to literally “LOL”. Loads more good stuff on the site, too.

August 2024? Completed it mate. See you next month!

Image credit: Getty

Look, I know you’re not interested in reading this first paragraph. You’re here for the links, right? You know it, I know it. So let’s just get to it: July’s best bad science links are below.

Subscribe now

The links

• “A new type of fraud”! Sneaking citations to your own previous papers into a new paper’s metadata - they don’t appear in the text but still cause your citation count to go up. You’ve almost got to respect the level of ingenuity here.

• “Can names shape facial appearance?”. I don’t wish to be dismissive, readers, but this is a result that just can’t possibly be true, and I knew that before reading beyond the title (it turns out to be full of totally unconvincing stats, so I was right to be sceptical). Still - it got into PNAS, so, uh, good on them…?

• We’ve had evidence of this before, but here’s a new paper that appears to close the book on the whole thing: “Marshmallow Test performance does not reliably predict adult outcomes”. Quite embarrassing for psychology, one might suggest!

• I can’t believe I’ve only just discovered the Twitter account of Mu Yang, another researcher with a great eye for spotting dodgy graphs in published scientific papers. I chuckled aloud at the impossible “back bend” in this one — and Yang has tweeted many many more examples.

• As more information appears about the Francesca Gino case, it allows data sleuths like the guys at Data Colada to be even more forensic with their fraud investigations. Remember that the authors of this post are—absurdly—being sued for defamation by Gino.

• Why zombie theories stick around: a nice summary/recap of the problems in the scientific system.

• Another glaringly obvious AI-generated scientific diagram in a published paper, raising questions about what on Earth the reviewers and editors at this seemingly-respectable journal were doing. Looking out the window?

  • And remember: this stuff is bad, but the real problem begins when AI-generated pictures are good enough to be indistinguishable from real ones. It won’t be long!

• Bizarre scenario where a scientist is sent a paper to review, and upon reading it notices that it’s “100% plagiarised”… from his own work. Several months pass, and then a respected journal in the field… publishes the plagiarised paper.

  • Worse, several other researchers pop up on Twitter to say the same thing happened to them. What on Earth is going on here?!

• Here’s a “monkey’s paw” scenario. You want scientists to calculate statistical power for their experiments? Well, lots of them do! But [monkey’s paw curls] they calculate it using a totally shite piece of software called G*Power which is based on clicking buttons rather than writing code - and thus the power calculations can only rarely be reproduced (and the default settings make them prone to errors).

• Paper claiming to do a “reversal of autism symptoms” in two kids goes viral. It’s in two kids. It doesn’t have a control group. It’s crap. It should be ignored.

• A controversy over a well-known set of experiments on honeybees and their little “waggle dance”. It looks like the papers are riddled with strange-looking data and what are euphemistically called “irregularities” in this Science article. The original author denies it all, of course.

  • As always, you’ve got to wonder how much more stuff like this is, ahem, buzzing around in the scientific literature.

• There’s a policy in the US (and elsewhere) called “ban the box”, where ex-convicts are helped to get jobs by stopping employers from requiring a “do you have a criminal record” checkbox on job applications. A prominent 2020 paper said this policy backfires and actually worsens the prospects of young black men (because employers just blanket-discriminate against them). A new reanalysis says no: actually, once you correct some errors in the 2020 paper, there’s no such backfire.

• And talking about re-analysis: here’s a detailed critique of one of my own papers from 10 years ago. As I’ve said many times, I agree with the “stay in school” message, I believe that education raises IQ (indeed, I wrote the meta-analysis confirming this), and I think the author might just misunderstand my views. Still, there are some very valid technical criticisms here.

• In genetics there’s the idea of “balancing selection” - genes for certain traits stick around even though they’re negative in some situations because they’re advantageous in others (e.g.: schizophrenia genes don’t disappear despite their obvious disadvantages, because they might cause creativity sometimes). Here’s a new and very strong critique of that idea for psychological traits.

• And to come back, full-circle, to manipulated citation counts: here’s an article about the world’s highest-cited cat, called Larry.

Subscribe now

P.S. The Studies Show

Our angriest podcast episode yet: on “misinformation”, and the shocking level of nonsense spoken about it (by people who should know better). That’s for paying subscribers; free listeners can hear some apocalyptic stuff on asteroids, nuclear winter, and air pollution.

Aaaaaand that’s us for another month. See you in August. If you don’t already subscribe, please do so below!

I have posted
the links
that are about
scientific integrity

and which
you had possibly
missed
during June

Forgive me
they are depressing
so grim
but so important

Subscribe now

The links

• I’ve long argued that we need much more serious punishment for scientific frauds. So I was very pleased to see Chris Said arguing “the case for criminalising scientific misconduct”. If a fraudster fakes research on say, cancer, that’s not just wasting money and effort (bad as it is to waste money and effort). It’s also slowing down progress towards a cure, meaning a lot of life-years are lost to sufferers. Lock them up!

  • And as if on cue, a scientist has been charged with defrauding the US Government out of millions of dollars after an FBI investigation. Innocent until proven guilty of course, but it’s nice to see this kind of thing being taken seriously.

• The largely bullshit field of “misinformation” research continues to take a very justified beating:

  • First, an article in Nature debunked three of the main beliefs of those who bang on about online misinformation (that the average person sees a lot of it; that it’s caused by algorithms, and that it causes polarisation);

  • Then, a prominent misinformation researcher who claimed that Meta put pressure on Harvard to fire her turns out to, perhaps, be spreading misinformation herself (described in a long article with a killer final section);

  • Then Matt Yglesias wrote a great article about how the studies tend to focus only on one narrow type of misinformation anyway;

  • Then, after the Trump-Biden debate this month, one of the most clownish adherents of the “misinformation” worldview descended into self-parody. These people are very silly and they deserve far more scrutiny!

• Another month, another major Alzheimer’s paper retracted for doctored images. This one was in Nature.

  • Incidentally, this will be great: Charles Piller, whose top-tier investigative journalism helped expose loads of fake Alzheimer’s research, has a book on the topic coming out next year.

• Talking about Nature retractions: here’s a stem cell paper that was published in Nature in 2002, had its integrity questioned in 2007, went on to be cited more than 4,400 times… and was only retracted in 2024. “This is fine”, as they say.

• Fake scientific images made by generative AI that can’t be detected as fake. This gave me that horrible “game over” feeling.

• My friend Ruben looks into more of the papers cited by Jonathan Haidt in his Anxious Generation book, and finds them wanting.

  • Same again, in a second thread.

• Professional org for trans healthcare commissions independent researchers to do a review of the evidence… and then won’t let them do their research without interference.

• Usually questions in article headlines are automatically answered with “no” - but here’s a rare exception. “Is science’s dominant funding model broken?”

• A big bust-up over a study that claimed that educational programmes in prison can reduce offending. The original debate is about whether the effect is causal, but Andrew Gelman adds to it, pointing out the uncertainty about the magnitude of the effect, too (it’s what he calls a “Type M error”).

• The “emotional dot probe” task is very commonly used in clinical psychology research, especially in people with anxiety. Turns out… well, you can probably guess how it turns out.

• Eternal sunshine of the… terrible study?

• A re-analysis of some of the work underlying the idea of “superforecasters” - people who are extremely good at predicting future events. A study from a decade ago appeared to find some of the reasons for why they were so capable - but this new analysis finds the whole thing to be a lot muddier.

• Good news, for a change: Stuart Buck from the Good Science Project writes about some important changes that are being suggested at the National Institutes for Health in the US (a major government research funder) that might make its funded research more reliable.

• We’ve all at some point written “casual” when we meant “causal”. But how many of us have left the typo in a published paper? Oh. Quite a lot of us.

  Subscribe now

P.S. The Studies Show

My favourite episode of the podcast from the last month is the paid one, on Viagra. Every other podcast seemed to be advertising mail-order penis pills, but we thought we’d do an episode on the science behind them. See also episodes on peanut allergy, addiction, and forensics.

If you’re already a subscriber to this newsletter, I’m very grateful. If you’re not, what are you waiting for? Put your email address in the box below:

Image credit: Getty.

You know how there’s loads of stuff on the internet about how science is cool, and great, and awesome? Well, this is the opposite of that. Here’s your monthly collection of links about scientific fraud, retracted research, and erroneous studies. Hope you “enjoy” it!

Subscribe now

The links

• Fraudulent anaesthesiologist Joachim Boldt is not only the most retracted researcher of all time, but has now hit the milestone of two hundred papers retracted. In fact, he’s exceeded it - he’s up to 210! Congratulations…? Imagine faking that many studies! Just imagine. You almost have to respect it.

• Talking about retractions: one particular Egyptian researcher in obstetrics and gynaecology has already had seven papers retracted, so some data-integrity researchers looked at all 263 papers he’s published (many within a very short period of time). They concluded that 43 had “impossible” results, 67 had “unlikely” results, and 11 had mistakes. Oh dear.

  • One amusing example: “…the authors claim that the mean age of the physicians surveyed was 42.6, and their mean number of years in practice was 26.4. Using these numbers, the average physician surveyed must have started practising at 16.2 years old.”

• There was a TikTok influencer called “Lab Shenanigans” who did videos about being a neuroscientist and who had 525,000 followers. Turns out he falsified loads of data in his research. I mean… are we surprised by this? The clue was in the name…

• I didn’t know there was a journal called Trial and Error that exists to publish null results. As with previous such things I expect it won’t solve the problem (i.e., that people are only interested in positive findings), but it’s a well-meaning idea. It’s covered in this Nature News article on negative results.

• To err is human. To set up an error-checking service for science, which I mentioned here a few months ago but which now has its first report, is divine.

  • There’s also a Nature article about the whole thing. More of this, please!

• You might’ve heard that there’s been a huge increase in maternal mortality in the US over the past couple of decades. Horrible! In such a rich country, too! Except, the whole thing is due to changes in recording and maternal mortality hasn’t actually increased at all.

• A meta-analysis on the effects of psilocybin (from magic mushrooms) on depression, published in the BMJ (formerly the British Medical Journal). Sounds good, right? Well, it looks like the authors mixed up the standard error and the standard deviation when coding some of the studies, leading them to grossly overestimate (like, by 500%) the effect of psilocybin. How can you not notice that you’re including effects that are bigger than basically anything else in the whole field of medicine?

• You know I love sharing articles about how “hard” sciences like physics are just as subject to the sorts of replication-crisis problems that are known in the social sciences. Well, here’s another one, on the increasing awareness of dodgy research in condensed matter research.

  • (Just to be clear: I don’t “love” this, really. It’s very bad and depressing. But it’s always good to be reminded that it doesn’t just happen in psychology).

• Useful Science “policy forum” article on the uncertainties surrounding the effects of preschool on longer-term outcomes. We need better research on this, ASAP.

  • See the letter at the bottom for an alternative perspective.

• Critics of antidepressants publish a review that questions the biological basis of depression. They are reprimanded by psychiatrists, including for not citing a meta-analysis of depression and tryptophan. Someone re-analyses that meta-analysis. It’s crap.

• Paper in the Journal of Environmental Psychology claims that the best place to relax is near water. A general finding about all humanity? Well… it turns out all the participants (and there were only 32 of them) were members of a swim team. Come on, man!

  • Adam Grant’s tweet where he promoted the study has, at the time of writing, 4.1m views, sadly not just because everyone is ridiculing it. Something something “truth gets its boots on” - you know the quote.

• Authors of a review on lithium and suicide make an objective, study-ruining error where they mix up rates and ratios. There’s no argument about this: you can’t combine those. The results are complete nonsense. The paper needs to be corrected or (more likely) retracted. But the journal editor doesn’t seem to realise, politely thanks the critics for their letter pointing out the problem, and… leaves the study in place and doesn’t do anything about it. Jolly good!

• The publisher Wiley just closed down nineteen journals (having already closed down four) because they’d become flooded with fake “paper mill” papers.

  • And, closer to home (for me), the Scottish Medical Journal retracted 13 papers at once because they were also the products of paper mills.

• You know “cupping”? The weird alternative-medicine treatment that leaves people covered in round suction bruises? A recent paper claimed that your psychological state could help heal those bruises quicker. But the whole thing is a nice lesson in unreliable statistics and flawed readings of the scientific literature, according to Andrew Gelman and Nick Brown.

• As if we wouldn’t already be highly sceptical of a paper that claims that a specific kind of Iranian honey is an “amazing preventive and therapeutic agent” against Alzheimer’s disease (that’s a quote from the title BTW), the paper had some strange-looking images and has now been retracted. Please enjoy this very funny exchange with the authors who describe the findings as “completely real” and thank critics for their “kindness and love”. Aww.

P.S. The Studies Show

Calling all podcast fans: this month you can hear me and Tom talking about Vitamin D, as well the effects of lead paint/pipes/petrol on your brain and on crime rates. And our paid episode this month was on the scientific writings of Johann Hari (you might be able to guess the conclusions we draw there).

Thanks for reading! If this kind of thing is valuable to you, please don’t forget to subscribe so that you’ll get the next one:

Image credit: Getty

Bad science links! Get your bad science links right here! As usual, over the month I’ve collected all the most interesting stories about meta-science, crap research, scientific fraud, the broken scientific publishing system, and related topics - and I’ve posted them below.

Enjoy. And please do subscribe and share the newsletter with your friends and colleagues if you haven’t done so already.

Subscribe now

Share

The links

• Another month, another Nobel prizewinner (this time from Stanford) who’s had to retract and correct several of his papers due to “errors” in the images. So it’s not just the ex-President of that university who was making these “errors”!

  • See also the note at the top: even more errors were found in the same scientist’s papers after the news article appeared. Oops!

• The data sleuth who noticed the problems in the above papers, Elisabeth Bik, successfully gets two other papers retracted… a mere nine years after she pointed out the problems. Science is self-correcting… sometimes, eventually.

  • I can actually go one better: here’s a paper (by the President and CEO of the Dana-Farber Cancer Institute, no less) that was retracted this month after concerns were raised about it ten years ago.

  • And in other Elisabeth Bik news, the BS harassment complaints thrown at her after she criticised the research of powerful French scientist Didier Raoult have, happily, been thrown out by a court.

• In last month’s newsletter I mentioned the disgraced superconductivity researcher Ranga Dias. Since then, journalists have seen the report his university (University of Rochester) commissioned into his misconduct. Fabrication, falsification, plagiarism - it’s all here.

  • Incidental detail for metascience fans: the investigations of Dias were kicked off by a complaint from Jorge Hirsch, the materials physicist who also—for better or worse—came up with the h-index.

• Oh, and talking about plagiarism - you’ll never guess what someone discovered in the work of Harvard prof and very-credibly-accused-fraudster, Francesca Gino. So it’s not just the ex-President of that university who was plagiarising!

  • Gino’s erstwhile colleague, Dan Ariely, this month posted an update to the effect that his university have investigated his research, given him a slap on the wrist for “faulty data”, but taken no further action. He’s “putting this matter behind him” now. Jolly good!

• One of those large-scale reproducibility attempts, covering 110 papers from economics and political science. The results are a mixed bag: a high proportion (85%) of papers had computationally-reproducible results (though, c’mon, it’s not that high when you think about it), but a quarter of papers have a substantial coding error, and just over half exaggerate their effect size. Lots more interesting results in there, too.

• Do more diverse companies do better financially? Consulting firm McKinsey says yes; these academics who’ve re-analysed the data say no.

  • From the re-analysis paper: “we commit to posting the data and code by 31 December 2026”. This is stupid. What if you get run over by a bus before that date? Just post it now!

• A refreshingly open and honest discussion from the editors of a reproductive health journal on their process of dealing with fraud allegations and retractions.

• Remarkable article by a scientist whose paper was retracted for entirely spurious reasons (in actual fact it seems to have been done at the behest of a publishing company the paper identified as a predatory outfit).

  • Seriously - read the retraction note. It’s Kafkaesque. The “regression analysis didn’t have a control group”? What are you talking about?

• Ruben Arslan finds a paper on subliminal priming (cited in Jonathan Haidt’s new book) where it was physically impossible that the stimuli were shown to the participants, even “subliminally”, because of the refresh rate of the monitors they used. And yet they still found effects! Psychology really is magic.

• “What is the prevalence of bad social science?” Dunno. But there are some interesting examples and discussions here.

• You know that thing where albums suddenly disappear from music streaming services, meaning that if you haven’t got the tracks downloaded to your computer, you can’t listen to them any more? Well, that - but for scientific journals.

• Rare example of a female (alleged!) scientific fraudster (though of course cf. Gino above): researcher at the University of Toronto is accused of fabricating data, as well as getting her husband to pretend to be participants in her psychology study. She’s had her PhD revoked, and seems to have lost a new job at Northwestern University, too.

• You think you’ve seen it all, and then along comes “co-authoring a paper with your own alter ego”.

Subscribe now

P.S. Invisible College

Are you aged 18-22? Are you around the UK on 26-31 August this year? You should consider attenting Invisible College, the residential programme in Cambridge run by Works in Progress magazine.

It’s all about Progress Studies - understanding how and why things get better over time. I’ll be giving a series of talks on the topics covered on this Substack: bad science, how to spot it, and what we can do about it. The week will also include loads of other interesting talks on topics like the history of technological progress and how public policy works.

If you’re interested (or you know someone who might be), you can find more details and the application form at this link. See you in August!

P.P.S. Metascience grants!

If you’re a researcher at a UK research org who’s interested in making science better, you need to take a look at these new meta-science grants (up to £300,000 each), co-funded by the UK Government and Open Philanthropy.

P.P.P.S. The Studies Show

More of my opinions on science-related matters, should you wish to hear them, are available on my podcast, The Studies Show. For understandable reasons, the biggest episode we’ve done recently was on the Cass Review and youth gender medicine. You’ll have to become a paid subscriber to The Studies Show for that one, but free subscribers can still listen to new episodes on microplastics, the history of probability, and whether depression really exists.

And we’re done. Hope you found something interesting above. To get the newsletter next month, make sure you’re subscribed. It’s free!

Image credit: Getty

One year! That’s how long I’ve now been doing these link-post newsletters on the Science Fictions Substack. I do hope you’re finding them interesting and/or useful.

If you know someone who might appreciate this newsletter—a monthly list of the best links to do with bad science, with my opinions left in—please do share it with them! Post the link in a group chat, tweet it out - whatever works to spread the word. I’d be very grateful indeed. Okay, here we go…

Share

Subscribe now

The links

• RIP Daniel Kahneman. Do I feel bad that over the years I trashed both of his pop-science books, the first (Thinking, Fast and Slow) for being packed full of uncritical citations of bad research, and the second (Noise) for being interminably boring? No. But do I think that overall he still had a positive effect on the public understanding of science? Also no.

  • His actual scientific work in the 1970s and 80s with Amos Tversky was genuinely great, though!

• A disappointing update on a study I shared here a few months back. It was authored by some of the leading lights in the Open Science movement, and apparently showed that techniques like preregistration improved the replicability of research. The authors claimed the study was itself preregistered… but now it looks like it wasn’t. This is extremely bad, and the way the authors have now said that they’re “looking for the preregistration document” is embarrassing for everyone concerned.

  • The lesson: trust no one. Not me, not your co-authors, not your Open Science heroes. No one.

• The Francesca Gino debacle continues. This month Harvard unsealed their 1,300-page report on their fraud investigation into her work. Here’s a brief summary, including mention of Gino trying to scapegoat another professor for falsifying the data.

• On a similar topic, the story of the “slow-motion downfall” of Didier Raoult (who you might remember as the hydroxychloroquine-for-COVID guy, but there’s so much more to it than that) is a must-read.

• …as is this article, also on a similar topic, about the superconductivity researcher Ranga Dias and the swirl of misconduct allegations around his lab and retractions of his research. After the last year, I feel really bad for all the scruplous, high-integrity superconductivity researchers out there - your field is getting a very bad reputation!

  • Here’s Derek Lowe’s additional commentary on the story, noting that too many senior researchers browbeat their students into doing bad science, and too many universities aren’t interested in getting to the bottom of it.

• Informative YouTube video on David Sinclair, the anti-ageing researcher with a history of massive claims… and unreplicable research.

  • Yes, the style of the channel is a bit too “I am hyper-optimising for YouTube” for my taste, but I promise the content is good.

• Paper comes out claiming that banning prositution in Sweden led to an increase in rape rates. Almost immediately, a re-analysis shows the result is not real, and just due to an error in the statistical analysis code. Oops!

• “A Columbia Surgeon’s Study Was Pulled. He Kept Publishing Flawed Data.” One of the top cancer surgeons in NYC has been accused of perpetrating a particularly lazy scientific fraud where he used the same images across multiple papers, claiming they were all from separate experiments.

• I’ve rarely read any pop-science article that makes as many dubious (or just plain ridiculous) statements as this one on “how the climate crisis affects our brains”.

  • The worst thing is finding out at the end that the author has apparently written an entire book on this stuff. Groan.

• Cool to see that the Bill & Melinda Gates Foundation is no longer putting up with the extortionate “article processing charges” required by Open Access journals, and is now mandating that research it funds be posted online as a free preprint by default.

• And more good news: the BMJ (formerly the British Medical Journal but it now just goes by the initials - what’s that all about?) is now mandating data and code sharing for all articles published there. Should’ve been the case a long time ago, but credit where due.

• A sudden and vast increase in the times the word “delve”—which happens to be a favourite of ChatGPT for whatever reason—is used in medical papers, starting in 2023. Pretty good prima facie evidence that scientists are using ChatGPT to help write their papers… and I really do hope it’s “help write” rather than “churn out filler text which they then thoughtlessly copy and paste”.

  • You know… like this. And this.

• “Given that it was arguably Beethoven’s skills as a musician and composer that made him an iconic figure in Western culture…”. The only place you’d see a mind-numbing sentence like that is, of course, in a scientific paper.

Subscribe now

P.S. The Studies Show

The podcasts continue apace. I was particularly pleased with our recent episode on the trouble with meta-analysis - discussing the pros and cons of a technique we perhaps rely on a little too much.

And we’re now doing short episodes in addition to the main, hour-long ones! The first one is on emotional intelligence. Please do give it a go, and subscribe on The Studies Show page if you like it!

That’s your lot. Thanks very much, as ever, for reading. See you in April!

Image credit: DALL-E

February might be the shortest month, but there’s been no shortage of bad science. Here are the most interesting links on that topic from the past month (and by the way, scroll to the bottom for a job ad that might just be of interest to readers of this Substack).

Enjoy - and don’t forget to become a subscriber if you aren’t already!

Subscribe now

February’s best bad science links

• Okay, let’s get it out of the way. You’ll only have missed this one if you happen to have had your internet cut off for the past fortnight. It is, of course, the AI-generated picture of a rat with an enormous penis, published in a supposedly serious peer-reviewed scientific journal. This went mega-viral, the paper rapidly got retracted, and it’s a portent of what we’re going to see much more of in the coming years: fake AI pictures, text, and data making it past scientific gatekeepers who are sleeping on the job.

• “Cognitive dissonance” is a great phrase, but it was also supposed to be an empirical discovery about people’s behaviour made by psychologists in the 1950s and 60s. How does that empirical work hold up? According to this very large new replication attempt, not particularly well.

• A fun preprint on how Google Scholar citations can be (and are being) gamed. The researchers develop the c-index, which is like the h-index, but about the number of papers that cite a given scientist multiple times. Most scientists are only cited once or twice in a given paper. But when there are at least 45 papers that cite you at least 45 times? Either you’ve founded your own field of study, or something funny’s going on.

• A feelgood study on how children who got vaccinated in India had higher future wages receives a pretty devastating critique: “the study is really measuring the effects of passage of time”.

• “Nature Medicine recently corrected a figure after my PubPeer comment and email. The ‘corrected figure’ contains a different version of the same error.” Remind me: what’s the point of scientific journals?

• You might’ve seen the study that claimed that women hunt in 79% of hunter-gatherer societies. If that was true, it would mean the general wisdom on the gender split in these societies is totally wrong. Well, it turns out the study was fatally flawed, so no need to update your views this time.

• Here’s a tip. If you want to get an article in Science where you demand that students be taught things for which there’s no good evidence, like transgenerational epigenetic inheritance, just make sure it’s in an article that’s filled with buzzwords like “equity”, “social justice”, “problematic”, and “power and privilege dynamics”.

  • Another way of getting an article in Science: write about how “indigenous knowledge” is “complementary” to science. The same way herbal medicine is “complementary” to actual medicine, I’m sure.

• Depressing story of how one guy’s fraudulent research on curcumin (a chemical compound that can dye things yellow-orange but probably has no health effects) has spawned a whole supplement industry - and even more fake research.

• Excellent paper on how to think about statistical power analysis and effect sizes in the research you’re planning. ~Everyone in psychology research, and many other fields besides, should read this.

• A weird one: the Dean of the School of Engineering at the University of Nevada, Reno runs his own pay-to-publish journal that’s full of complete gibberish. Imagine he was your boss! Andrew Gelman has a suitably baffled article on it.

  • The Retraction Watch article on the case notes that “the pages of the journal were also filled with articles from [the Dean’s] wife, his son, his students and the current editor-in-chief”. Help me understand why this exists!

• The Nobel Prize-winning medical scientist Gregg Semenza has just had his eleventh paper retracted due to data integrity concerns. Nothing to see here, everything fine, no big deal.

• Let’s end on a positive note: there’s a very cool new initiative to encourage people to spot and report errors in scientific papers. It’s like what I did for my book: if you find a mistake, you get paid - and bigger errors mean bigger payments. You can sign up to become an error-checker on their site. Do your bit for a slightly less error-strewn science!

Subscribe now

P.S. A job at GiveWell

GiveWell, the amazing organisation that rates charities as to their cost-effectiveness (and thus makes charitable donations much better at saving lives), is looking for a Head of Communications.

The job will involve the sorts of things that readers of this Substack enjoy, like sorting good scientific studies from bad ones, and writing and talking about (meta-)science to a wide audience. And that’s all in a job where you really know you’re making a direct, positive difference in the world. If you think you might be a good candidate, you should definitely apply - all the info is right here.

P.P.S. The Studies Show

Psychotherapy! Climate models! A big episode on whether phones and social media cause mental health problems! These are just some of the recent topics we’ve covered on my science podcast with Tom Chivers, The Studies Show. If you become a paid subscriber you can get access to the recent episodes on male and female brains and the Hans Eysenck affair, too.

And we’re done. Thanks to everyone who’s already subscribed to the Science Fictions newsletter. If you aren’t, you can sign up right here. See you next time!

Image credit: Getty