https://www.science.org/doi/10.1126/sciadv.adn0671
https://archive.is/nnElw
Epidemic outcomes following government responses to COVID-19: Insights from nearly 100,000 models
Eran Bendavid (Department of Medicine, Stanford University, Department of Health Policy)
Chirag J. Patel (Department of Biomedical Informatics, Harvard)
Abstract
Government responses to COVID-19 are among the most globally impactful events of the 21st century. The extent to which responses—such as school closures—were associated with changes in COVID-19 outcomes remains unsettled. Multiverse analyses offer a systematic approach to testing a large range of models. We used daily data on 16 government responses in 181 countries in 2020–2021, and 4 outcomes—cases, infections, COVID-19 deaths, and all-cause excess deaths—to construct 99,736 analytic models. Among those, 42% suggest outcomes improved following more stringent responses (“helpful”). No subanalysis (e.g. limited to cases as outcome) demonstrated a preponderance of helpful or unhelpful associations. Among the 14 associations with P values < 1 × 10-30, 5 were helpful and 9 unhelpful.In summary, we find no patterns in the overall set of models that suggests a clear relationship between COVID-19 government responses and outcomes. Strong claims about government responses’ impacts on COVID-19 may lack empirical support.
DISCUSSION
In this study, we perform a multiverse analysis of nearly 100,000 ways of probing the relationship between COVID-19 government responses and outcomes in 181 countries. The goal is to create a multiverse of plausible analyses and assess the sensitivity of the results to these choices. Exploring the multiverse for a question of high importance may be useful where there is no consensus. In this study, we found no clear pattern in the overall set of analyses or in any subset of analyses. We are left to conclude that strong claims about the impact of government responses on the COVID-19 burden lack empirical support.
Inferences from this analysis deserve careful consideration, including a clear understanding of what this study cannot illuminate. First, none of the models tested can tell the extent to which any government response could have improved COVID-19 outcomes. Perhaps with another virus, other implementation strategies, or different populations, school closures could have extinguished transmission. Nor can we learn from this study what COVID-19 outcomes would have been like in the absence of these responses. Second, our analysis is global in scope and examines government responses and COVID-19 outcomes at the level of countries. This is suitable for inferring global patterns and trends but cannot exclude patterns at state, district, community, or even neighborhood levels.
Third, and perhaps most importantly, we cannot conclude that there is compelling evidence to support the notion that government responses improved COVID-19 burden, and we cannot conclude that there is compelling evidence to support the notion that government responses worsened the COVID-19 burden. The concentration of estimates around a zero effect weakly suggests that government responses did little to nothing to change the COVID-19 burden.
This conclusion departs meaningfully from many scientific studies of government responses. For example, a highly cited study on this topic notes that “Our results show that major non-pharmaceutical interventions—and lockdowns in particular—have had a large effect on reducing transmission” (9).Such conclusions are common in the scientific literature (table S1), but our analysis—extensive in scope and outcomes—suggests that such strong claims lack empirical justification.
…
In sum, this comprehensive analysis of government responses and COVID-19 outcomes fails to yield clear inferences about government response impacts. This suggests that strong notions about the effectiveness or ineffectiveness of government responses are not backed by existing country-level data, and scientific modesty is warranted when learning from the responses to the COVID-19 pandemic.