BOISE — Understanding the blizzard of data points that go into Idaho’s response to the coronavirus pandemic, and how those play into decisions on whether the state’s met its criteria to move into each stage of reopening the economy, can be challenging — especially when the numbers seem to change or sometimes seem not to add up.
“We continue to make the choice to report the data as soon as we get it,” state Health & Welfare Director Dave Jeppesen told the Idaho Press. That means sometimes numbers are posted, and then they change the next day or the day after.
So this reporter sat down, virtually, with Jeppesen and state Epidemiologist Christine Hahn on Friday to talk data and criteria.
“I understand the data is not easy to understand,” Hahn said. “We’re not hiding anything. We are professionals, we’re trying to show the data as it is. … I’m very proud of our metrics. I think we’ve made them workable.” She added, “We can keep tweaking them and working on them.”
Jeppesen noted that when numbers change, the department gets questions, with some concerned that changing numbers are evidence of “some conspiracy or something.”
For example, on Wednesday, the state was reporting a cumulative total of 792 cases in Ada County, but on Thursday, it was reporting 791 — one fewer. The reason: One of the cases that was counted for Ada proved, on further investigation, to be from a different county.
Another example, from last Monday: The total number of cases reported for the day, 2,455, was one more than the figure people got if they added up the total number listed for each county, which came to 2,454.
A reader who did the math and contacted the Idaho Press wrote, “While the numbers are only off by 1, is this indicative of a greater pattern of faulty data reporting, or a faulty program that is not calculating correctly? This is simple math and it is wrong.”
The answer: “Because we’re trying to get the data up as quickly as possible, cases are initially reported by where we think they live,” Hahn said, typically based on a lab test report. On that day, officials knew they had one additional positive case, but didn’t know which county to assign it to — so it was included in the total statewide numbers, but not assigned to any county.
“We hear about cases through lab reports,” Hahn said. “Lab reports are amazingly lacking in detail. They don’t ask you a lot of questions. … We don’t get a lot of information that we really need, especially with some of these drive-thru clinics.” Sometimes they don’t even collect a full address, she said, just a phone number. “But we want to report it.”
After receiving the report, epidemiologists contact the individual to gather the additional information, in a process that can take several days. The information is continually being updated and reconciled.
Hahn said the state could add an asterisk to the by-county chart, noting that for one case on that date, for example, the county was still unknown; she added that to a list of changes in the works as the Idaho Department of Health & Welfare continues to add to and refine the data it’s publicly reporting on the state website, coronavirus.idaho.gov.
CONFIRMED VS. PROBABLE
There also are different numbers being reported, even for such basics as the number of new cases each day, by the state’s seven public health districts and by various news organizations and others around the state that are compiling and analyzing those figures each day. The differences come down to two things: Confirmed versus probable cases, and timing.
The state’s official figures combine lab-confirmed and “probable” cases, and have since April 9.
“This is a really challenging area,” Hahn said. “The only way you can be a confirmed case is because you have a positive lab test. But the epidemiologists, say they have a confirmed case, and then it turns out the person has three other people in their household and they all have similar symptoms. Because of the limited testing that’s been available, we’ve not always been testing those people. … So there’s a national case definition for probable cases — similar symptoms and a close link to somebody that has the (confirmed) case. We think it’s important to show both.”
Idaho started reporting both when the Council of State and Territorial Epidemiologists came out with the definition of probable cases and the Centers for Disease Control and Prevention adopted it. “So we started tracking those,” Hahn said. “Almost all of these are household members of confirmed cases — that’s what most of the probable cases are.”
Said Jeppesen, “It’s a pretty high bar on the probable. It’s a really tight definition.”
TIMING OF REPORTS
Each day, each of Idaho’s seven public health districts publicly reports the number of new cases it’s tallied, in real time. Then they submit the figures to the state, and they upload overnight into the state’s data system. This is where timing comes in: The state’s figures are a day behind the health district figures. But it’s the state’s figures that Idaho is relying on to determine whether it meets the criteria to move into the next stage of reopening the economy or not.
“We’ve tried to be a place where the live data (reported on the state website each day, and updated at 5 p.m.) is the data,” Jeppesen said. “We then take the snapshot from that data to do the calculations.”
“We use our own numbers,” Hahn said.
“It’s all timing, but the data’s the same,” Jeppesen said. “We recognize that this creates these types of questions, but we’d rather do that than wait a day or a week to get everything settled.”
Some of the data that goes into determining whether Idaho’s meeting the criteria doesn’t have those timing issues. The number of emergency room visits for COVID-like illness and the number of those that lead to hospital admissions, for example, is captured immediately electronically.
“We know every ER visit in almost real time,” Hahn said, thanks to an electronic data system that covers about 91% of all ER visits in the state. “One or two small facilities don’t report to us,” she said.
As a result, the time frame for the 14-day figures that are submitted to the governor for ER visits and hospital admissions are from a slightly more up-to-date window than the new-case numbers.
Another timing issue that can be confusing is weekend reporting. Idaho stopped posting new reports on Sundays in early May, largely because few reports were coming in that day and Hahn said it wasn’t worth having a state employee work seven days just to enter those. However, the daily new case numbers that the state is posting on Mondays now reflect only the new cases from Sunday to Monday, while the cumulative case number reported on Mondays reflects all new cases since Saturday — leading to numbers that don’t appear to add up.
Hahn noted that on the chart that shows new cases by date on the coronavirus.idaho.gov website, which includes a toggle to move back and forth between confirmed and probable cases, figures are shown for all days, including Sundays. “It’s in the graph,” she said. “And that graph is what we use.”
“We’re not losing the data,” Jeppesen said.
CRITERIA TO REOPEN
The criteria, and metrics, that Idaho is using to decide whether it’s meeting standards to move to the next stage of reopening the economy fall into three categories, all three of which must be met for the state to move ahead.
The three are:
- Syndromic: ER visits for COVID-like symptoms and hospital admissions.
- Epidemiologic: New cases and testing data.
- Health care: Hospital capacity and supplies and test results for infections among health care workers.
To move from any reopening stage to the next, in the first category, Idaho must show a downward trend over the most recent reported 14-day period for ER visits for COVID-like symptoms, or fewer than 20 of those visits per day on average during that period; and also, for hospital admissions resulting from those visits, a downward trend over the most recent reported 14-day period or fewer than two admissions per day on average over that period. Currently, Idaho is easily meeting all of those standards.
For the second category, Idaho must show a downward trend over the most recent reported 14-day period — which may be slightly different than the 14-day period for ER visits and hospital admissions, because of reporting delays — in new reported cases, or fewer than 20 per day on average; or testing data that shows either a downward trend in positive test results over 14 days or a percentage-positive per day on average of less than 5%.
The state hasn’t necessarily been showing that downward trend in new cases — on Friday, new cases spiked to 61 —or falling below 20 per day; nor has the percentage of positive test results been falling recently, though it’s now well below its level a month ago. But the percentage of positive tests is falling well below 5% in recent weeks; between May 10 and May 16, for example, 5,207 tests were completed, and just 3.5% were positive for COVID-19. That’s enough to meet the second category’s criteria.
For the third category, health care, Idaho must be able to treat all patients without using crisis standards of care, which has been true throughout the outbreak; and meet specific targets for available hospital capacity and equipment: At least 50 available ventilators and intensive care unit beds; and at least a 10-day available supply of personal protective equipment, including N95 masks, surgical masks, face shields, gowns and gloves. It must also meet targets to show a “robust COVID-19 testing program in place for at-risk health care workers.”
The first half of that is being easily met, and has been for weeks. The second half, which requires a downward trend over the most recent reported 14-day period for new infections among health care workers, or fewer than two per day on average over the same 14-day period, is a much closer call, though it’s improved in recent days. There were four new cases reported among health care workers on May 16, for example; two on May 18 (a Monday); two on May 19; and then none Wednesday, one Thursday and none Friday.
“The third criteria is super important, the burden on hospitals,” Hahn said. “We’re in really good shape on bed capacity.”
As for health care workers, “We hope to see that number fall to zero — we don’t want to see health care workers get infected,” she said. But many were at the beginning of the outbreak, particularly in the Wood River Valley.
“The goal here is that we are testing health care workers,” Jeppesen said. “Our current metric says downward trend or less than two a day, and we’re actually doing quite well on those, because most of our health care workers were early in the process.” But to ensure that that metric really demonstrates a robust testing system for those workers, he said, “We may adjust this one as we move forward, but that objective won’t change.”
Hahn said the same numbers reported on the state website each day for new cases among health care workers are the ones that go into this metric.
“All data is problematic, it has limitations,” she said. “That’s why we don’t look at any single measure. … The big picture is there are a lot of cases out there that are never diagnosed, and we know that.”
Whether it’s data on how many people die of heart attacks across the country each year, which leaves out cases in which the coroner doesn’t specify or another contributing condition, like pneumonia, is cited as the cause of death, data, she said, is “always squishy. But what we’re looking at is trends.”
Dr. Kathryn Turner, deputy state epidemiologist and chief of the Bureau of Communicable Disease Prevention for the Idaho Division of Public Health, said, “We think that the data that we’re displaying is equal to if not better than the amount of data being displayed by other state websites. In fact, Idaho has one of the more robust dashboards across the country.”
Hahn said no one’s quantified that by studying all 50 states’ data pages, but she’s personally looked at surrounding states’ and at Florida’s, which was renowned for being among the most complete in the nation prior to a scandal involving the firing of its architect.
“I looked at what they had up there, and I said, ‘We have all that up there,’” she said, with one major exception: Recent trend lines for some of the state’s larger counties, to show where in the state the newest cases are occurring. That’s something the state is working to add, she said.
On the state’s rebound.idaho.gov staged reopening website, PDF documents are posted showing the snapshots of data that were submitted to the governor for the moves into Stage 1 and Stage 2 of reopening.
“When we get to the decision point, we take a snapshot,” Jeppesen said. “It’s presented slightly differently, but it’s the same data that’s on our live website.”