But randomized trials may not always be available. Or they may be too cumbersome to perform, or not ethical. Most doctors can relate to recent comments by the Food and Drug Administration director Stephen Hahn in last week’s congressional pandemic hearing. “In a rapidly moving situation like we have now with Covid-19,” he said, decisions are made “based on the data that’s available to us at the time.”
So, until the trials arrive, here are some ideas for how natural experiments could help:
Timing of treatment
Patients hospitalized with critically ill Covid-19 infection in the days before the Recovery trial results were announced would be expected to have worse outcomes than otherwise similar patients hospitalized in the days afterward, assuming doctors suddenly started using more dexamethasone (which they almost certainly have). Consistent with the trial results, we would expect to see an effect only in Covid-19 patients who were critically ill.
Based on the trial’s results, it’s also reasonable to think that once patients are critically ill, earlier treatment with dexamethasone might lead to better outcomes (the steroid has not been shown to be effective with patients who are not on respiratory support). This hypothesis could be tested by evaluating whether mortality rates were lower for patients hospitalized with severe Covid-19 infection in the one to two days before the Recovery announcement compared with otherwise similar patients hospitalized in the week prior — who would, by chance, be getting the drug later in their disease course.
Staggered changes in protocols
Changes over time in practice patterns within hospitals could also be used to better estimate the effectiveness of Covid-19 treatments. Hospitals have varied considerably in their treatment protocols for Covid-19, and these protocols have changed in the months since the pandemic began. The staggered change in treatment patterns across hospitals could allow researchers to estimate the effectiveness of Covid-19 treatments by using each hospital, at a different period in time, as its own control.
For example, if dexamethasone is indeed effective, we would expect that hospitals that quickly incorporated the drug into treatment protocols after the Recovery trial announcement would experience earlier reductions in Covid-19 deaths than hospitals adopting it later. The key to this natural experiment would be to verify that the average characteristics of coronavirus patients within a hospital would be unchanged in the short interval before protocols were changed versus afterward.
Just above or below a threshold
The sometimes-arbitrary thresholds that hospitals use to decide which patients receive specific treatments could be used to better estimate the effectiveness of Covid-19 treatments. Treatment decisions often rely on clinical cutoffs, such that patients immediately above or below a threshold have very different likelihoods of treatment despite being otherwise similar.
For example, if a hospital decides that every person needing more than six liters per minute of oxygen will go on a ventilator, patients using six liters will go on a ventilator, but those using five liters will not — even though they are much the same.