Let’s check our assumptions before we do anything dumb.
Be like Hank, except inside
A widely followed model for projecting Covid-19 deaths in the U.S. is producing results that have been bouncing up and down like an unpredictable fever, and now epidemiologists are criticizing it as flawed and misleading for both the public and policy makers. In particular, they warn against relying on it as the basis for government decision-making, including on “re-opening America.”
“It’s not a model that most of us in the infectious disease epidemiology field think is well suited” to projecting Covid-19 deaths, epidemiologist Marc Lipsitch of the Harvard T.H. Chan School of Public Health told reporters this week, referring to projections by the Institute for Health Metrics and Evaluation at the University of Washington.
Others experts, including some colleagues of the model-makers, are even harsher. “That the IHME model keeps changing is evidence of its lack of reliability as a predictive tool,” said epidemiologist Ruth Etzioni of the Fred Hutchinson Cancer Center, who has served on a search committee for IHME. “That it is being used for policy decisions and its results interpreted wrongly is a travesty unfolding before our eyes.”
The IHME projections were used by the Trump administration in developing national guidelines to mitigate the outbreak. Now, they are reportedly influencing White House thinking on how and when to “re-open” the country, as President Trump announced a blueprint for on Thursday.
The chief reason the IHME projections worry some experts, Etzioni said, is that “the fact that they overshot” — initially projecting up to 240,000 U.S. deaths, compared with fewer than 70,000 now — “will be used to suggest that the government response prevented an even greater catastrophe, when in fact the predictions were shaky in the first place.”
That could produce misplaced confidence in the effectiveness of the social distancing policies, which in turn could produce complacency about what might be needed to keep the epidemic from blowing up again.
[…]
“This appearance of certainty is seductive when the world is desperate to know what lies ahead,” Britta Jewell of Imperial College and her colleagues wrote in their Annals paper. But the IHME model “rests on the likely incorrect assumption that effects of social distancing policies are the same everywhere.” Because U.S. policies are looser than those elsewhere, largely due to inconsistency between states, U.S. deaths could remain at higher levels longer than they did in China, in particular.
While other epidemiologists disagree on whether IHME’s deaths projections are too high or too low, there is consensus that their volatility has confused policy makers and the public.
See here and here for previous mentions of the IHME. As you may recall, I noted the very wide error bars on its numbers, which tends to be overlooked in the way the stories about this model were written. The IHME is the model that Greg Abbott has used for his reopening plan, so the fact that its projections can change significantly as new data comes in should be of concern. Plus, you know, the whole we are bad at testing thing. However you look at it, the data is noisy, and there’s evidence to suggest there are a lot more people out there with the virus than our level of testing has had the ability to detect. Which is to say again, there’s a lot we just don’t know yet. We shouldn’t rely on one view of the data for our understanding of what is happening, in the same way that we should not rely on any one single poll to understand what is happening in a given election.
Yes the IHME graph had a gray area around the line. The gray indicates the possible range, which goes from 30,000 to 160,000. Which is to say that anyone could make an estimate, science is not very accurate.
Jason,
To say that “science is not very accurate” is inaccurate.
The coronavirus is novel so it is new by definition. This little beast of a virus is still being figured out and understood.
And the more important point is the impact of the behavior of humans (some incredibly stupid) on the case and death toll.
The fundamental problem here is that the what’s being “predicted” is a function of mass social and health/hygene behavior. The people being studied can and do alter their behavior in response to public communication (incl. the model projections and prescriptions as to what and what not to do, whether they are compulsory or only exhortative).
It’s not like studying the course of a hurricane or a flood event, natural phenomena. Nobody could know to what extent compliance would be achieved, and the mobility restriction rules also vary, and even change over time, across jurisdictions.
Any aggregate predictions (or date-based metrics) for the US as a whole additionally involve a compositional problem, i.e. they represent the aggregate sum of what happens in many different jurisdictions (success and failure) with different mobility rules, population densities, and other relevant factors. In NY, for example, the subway system may have played a key rule in the spread. Many other jurisdictions don’t have a subway and individual traffic predominates (over public transport) in cities such as Houston.
For purposes of tracking pandemic progression, it does not even make much sense to look at the number (or estimates) for the US as a whole. Additionally, different parts of the country are at different points of the curve.
That said, you have to rely on the best model the experts can come up with, and adjust it as you go along, and as better data becomes available that allows for the assumptions (in lieu of data) to be revised.
What’s the alternative? Reading tea leaves?
brad, I think that I agree with you, but just a semantic distinction. I mean that everyone says “reopening” must be based on “science.” However, “science” is giving a very big projected range. As you wrote, there is a lot to learn about this virus.