By Tomas Fürst at Brownstone dot org.

I welcome Eyal Shahar's call for a re-review of Covid vaccine papers. In fact, I started long before Eyal blew the whistle - even before the vaccines appeared.

At the end of the terrible year 2020, a highly influential paper appeared in Science. It made headlines in major media outlets around the world. The paper, titled "Inferring the effectiveness of government interventions against COVID-19," was soon used by governments across the globe to justify their increasingly authoritarian policies.

It attracted my attention because the last author was Czech mathematician Jan Kulveit. Together with my two colleagues, Ondřej Vencálek and Jakub Dostál, we wrote the following response:

"All models are wrong, but some are useful" goes a famous saying usually attributed to George Box. Today, he would perhaps say that all models are wrong, and some are even dangerous. This, in our opinion, is the case for the study "Inferring the effectiveness of government interventions against COVID-19" that appeared in Science and received widespread attention around the world.

The study aims at understanding the effectiveness of non-pharmaceutical interventions (NPIs) in controlling the Covid-19 pandemic. The authors analyze data on the total case counts and death counts from 41 (mostly European) countries between January and the end of May 2020. They produce an estimate of the effects of 8 different NPIs (such as limiting gatherings of people, closing schools, etc.) which were implemented in many countries during the studied period. The effect of each NPI is quantified by the reduction in the infection reproduction number R at the time of the NPI imposition in the respective country.

The results have been widely welcomed because they seem to show that all of the NPIs generally work, and the effect sizes seem to agree with the common sense (e.g. the more you restrict gatherings, the greater reduction of R you obtain). Governments across the world will be very happy to hear that the restrictions they imposed were justified. But were they?

In fact, we do not know, and this study does not help us to find out. We argue that there is a fatal flaw in the model which renders it useless. Looking at the only equation in the body of the paper (see the "Short model description" section), we see that the authors assume the underlying (unobservable) basic reproduction number R0,c to be constant in time for each country. This basic reproduction number is then multiplied by the effects of the NPIs and this is fitted to data. Thus, the model assumes that any change in the dynamic of the epidemic is due to the NPIs. This is deceptive because it is circular. If you want to quantify the effects of an intervention, you cannot assume that all the observed effects are due to the very intervention.

Also, this assumption of constant R0,c suggests why the authors chose to stop modeling once any NPI is lifted. The NPIs are usually lifted as the epidemic dwindles. Thus, the NPIs are present when R is high, and they are absent when R is low. With data from a longer time interval (including the summer period of low prevalence and relaxed NPIs), the simple model the authors used would learn a negative effect - that NPIs speed up the epidemic. This was clearly undesirable, so the authors chose not to use the data from the summer to fit the model. Such modeling strategy is highly questionable.

To make our point completely clear, we performed the following experiment. We took the original dataset and invented a new NPI that never existed. Let us say that from the imposition of this new NPI on, each citizen was required to wear a T-shirt with a "Stop-Covid" inscription, until this NPI was lifted.

We drew a random date uniformly from the period over which a particular country was modeled, and "imposed" this T-shirt NPI on the data (see reference [3] for the original dataset with the T-shirt NPI added). We did not change the numbers of cases and ...