By Eyal Shahar at Brownstone dot org.
I think the answer is "yes," but here is the story for you to judge.
Each year the CDC publishes estimates of the effectiveness of the flu vaccine in the previous season. Reported by several networks, these estimates are based on a research design that is called a test-negative case-control study.
Over the years, the various authors have shared a similar analytical strategy: Early respiratory events were considered differently from later respiratory events (on the assumption that no effect is expected until immunity is built up). "Early" has typically been within two weeks post-vaccination.
Several months ago, I realized that this special handling of early events is a source of bias called "immortal time." To expose the bias, I used causal diagrams, formally called "directed acyclic graphs (DAGs)." DAGs were introduced in landmark publications from the 1990s and are widely recognized in epidemiology as a methodological tool.
I wrote a short paper with a scary title: "Immortal time bias in test-negative studies of the flu vaccine." I illustrated the bias by two simple DAGs that encoded the analytical approaches as described in those CDC-associated studies. The paper may be summarized in three bullet points:
Immortal time is an overlooked bias in test-negative, case-control studies of the flu vaccine.
The causal structure corresponds to misclassification bias or selection bias, depending on how early events were handled.
The bias can be avoided by considering all events and estimating built-up vaccine effectiveness by consecutive post-vaccination days.
I submitted the paper sequentially to three respected epidemiology journals. Surprisingly, the editor-in-chief of each journal rejected the paper within one week using boilerplate text. It was not sent for peer review. Why?
There are three possible reasons:
1. The message was not sufficiently important.
2. The paper was poorly written.
3. Soliciting peer review was not needed. The editor decided that the bias did not exist.
I can quickly eliminate the first reason. Exposing entrenched bias in studies of the annual flu vaccine is of utmost importance. I didn't need to compete with any "more important" papers.
Was it poorly written? I have published many scientific papers and two books. I am not a newcomer to epidemiology and even served as an associate editor for one of the three journals. So, that was not the reason.
We are left with the third reason. Three editors (possibly more in-house) decided that my analysis was obviously wrong. They did not need any external peer review. Was that the case indeed? How likely is it that three editors have reached a unanimous, independent verdict on my analysis without soliciting peer reviews?
I could have written a whole page on my expertise in the domains of causal diagrams and biases, but here is a short version of my record. I suggested a new taxonomy of biases based on DAGs and identified new types of bias. I authored about 20 DAG-related publications, including a book chapter. (Some of my articles were cited in the paper.) For sure, my methodological analysis was not "obviously wrong." More likely, it was obviously correct.
So, why was my paper rejected without peer review? May I speculate?
This paper has a startling message that discredits the CDC. All those studies about the effectiveness of the annual flu vaccine contained a mistaken analytical decision. The reported estimates are biased. Had my paper been published, it might have been picked up by some reporters. You can imagine the headlines.
The editors were dealing with a hot potato. They did not want to take the risk that peer-reviewers would endorse the paper, which would force them to publish it. An upfront rejection was a safe way out.
I tried once more. In my fourth submission to an epidemiology journal, I included the following cover letter:
"Dear Editor,
This short paper addresses immortal time in test-negative studies of the flu vac...