COVID-19 Data Dispatch

Tag: Testing

  • Florida is no longer sending tests to Quest Diagnostics

    This past Tuesday, the Florida Department of Health (DOH) announced that the department would stop working with Quest Diagnostics. Quest is one of the biggest COVID-19 test providers in the nation, with test centers and labs set up in many states. The company claimed in a statement to the Tampa Bay Times that it has “provided more COVID-19 testing on behalf of the citizens of Florida than any other laboratory.”

    So, why is Florida’s DOH cutting ties? Quest Diagnostics failed to report the results of 75,000 tests to the state department in a timely manner. Most of these results were at least two weeks old, and some were as old as April. As all the old results were logged at once on Monday night, Florida’s test and case counts both shot up: nearly 4,000 of those tests were positive.

    Such a reporting delay skews analysis of Florida’s testing capacity over time, especially as many of the backlogged tests were reportedly conducted during the peak of the state’s outbreak in June and July. This delay also likely means that, while the people tested with this batch of tests still received their results in a timely manner (according to Quest), contact tracers and other public health workers were unable to track or trace the nearly 4,000 Floridians who were diagnosed. Such an error may have led to many more cases.

    According to Florida Governor Ron DeSantis, such an error is tantamount to violating state law:

    “To drop this much unusable and stale data is irresponsible,” DeSantis said in a statement Tuesday. “I believe that Quest has abdicated their ability to perform a testing function in Florida that the people can be confident in. As such I am directing all executive agencies to sever their COVID-19 testing relationships with Quest effective immediately.”

    But is cutting all ties with Quest the correct response? Florida’s testing capacity already is below recommended levels. According to the Harvard Global Health Institute, the state has conducted 124 tests per 100,000 people over the past week (August 30 to September 5), with a positivity rate of 13.2%. This per capita rate is far below the state’s suggested mitigation target of 662 tests per 100,000 people, and this test positivity rate is far above the recommended World Health Organization rate of 5%.

    Florida will be able to send many of its tests to state-supported, public testing sites, the Tampa Bay Times reports. Still, this switch will take time and cause additional logistical hurdles at a time when Florida should not be putting the breaks on testing.

    more testing data
  • Three different units for COVID-19 tests

    Three different units for COVID-19 tests

    Colorado is one of six states currently reporting its testing in “test encounters,” a new metric that has appeared in recent weeks. Screenshot of Colorado’s dashboard taken on September 5.

    A few weeks ago, one of my coworkers at Stacker asked me: how many people in the U.S. have been tested for COVID-19?

    This should be a simple question. We should have a national dataset, run by a national public health department, which tracks testing in a standardized manner and makes regular reports to the public. The Department of Health and Human Services (HHS) does run a national testing dataset, but this dataset only includes diagnostic, polymerase chain reaction (PCR) test results, is not deduplicated—a concept I’ll go into more later—and is not widely publicized or cited.

    Meanwhile, 50 state public health departments report their local testing results in 50 different ways. Different departments have different practices for collecting and cleaning their test results, and beyond that, they report these results using different units, or the definitive magnitudes used to describe values.

    You might remember how, in a high school science class, you’d get a point off your quiz for putting “feet” instead of “meters” next to an answer. Trying to keep track of units for COVID-19 data in the U.S. is like that, except every student in the class of 50 is putting down a slightly different unit, no teacher is grading the answers, and there’s a mob of angry observers right outside the classroom shouting about conspiracy theories.

    Naturally, the COVID Tracking Project is keeping track anyway. In this issue, I’ll cite the Project’s work to explain the three major units that states are using to report their test results, including the benefits and drawbacks of each.

    Much of this information is drawn from a COVID Tracking Project blog post by Data Quality Lead Kara Schechtman, published on August 13. I highly recommend reading the full post and checking out this testing info page if you want more technical details on testing units.

    (Disclaimer: Although I volunteer for the COVID Tracking Project and have contributed to data quality work, this newsletter reflects only my own reporting and explanations based on public Project blog posts and documentation. I am not communicating on behalf of the Project in any way.)

    Specimens versus people

    Last spring, when the COVID Tracking Project’s data quality work started, state testing units fell into two main categories: specimens and people.

    When a state reports its tests in specimens, their count describes the number of vials of human material, taken from a nose swab or saliva test, which are sent off to a lab and tested for the novel coronavirus. Counts in this unit reflect pure testing capacity: knowing the number of specimens tested can tell researchers and public health officials how many testing supplies and personnel are available. “Specimens tested” counts may thus be more precise on a day-to-day basis, which I would consider more useful for calculating a jurisdiction’s test positivity rate, that “positive tests divided by total tests” value which has become a crucial factor in determining where interstate travelers can go and which schools can reopen.

    But “specimens tested” counts are difficult to translate into numbers of people. A person who got tested five times would be included in their state’s “specimens tested” count each time—and may even be included six, seven, or more times, as multiple specimens may be collected from the same person during one round of testing. For example, the nurse at CityMD might swab both sides of your nose. Including these double specimens as unique counts may artificially inflate a state’s testing numbers.

    When a state reports its tests in people, on the other hand, their count describes the number of unique human beings who have been tested in that state. This type of count is useful for measuring demographic metrics, such as what share of the state’s population has been tested. In most cases, when states report population breakdowns of their testing counts, they do so in units of people; this is true for at least four of the six states which report testing by race and ethnicity, for example.

    Reporting tests in units of people requires public health departments to do a process called deduplication: taking duplicate results out of the dataset. If a teacher in Wisconsin (one of the “people tested” states) got tested once back in April, once in June, and once this past week, the official compiling test results would delete those second two testing instances, and the state’s dataset would count that teacher only once.

    The problem with such a reporting method is that, as tests become more widely available and many states ramp up their surveillance testing to prepare for school reopening, we want to know how many people are being tested now. As recent COVID Tracking Project weekly updates have noted, testing seems to be plateauing across the country. But in the states which report “people tested” rather than “specimens tested,” it is difficult to say whether fewer tests are actually taking place or the same people are getting tested multiple times, leading them to not be counted in recent weeks’ testing numbers.

    Test encounters

    So, COVID-19 testing counts need to reflect the numbers of people tested, to provide an accurate picture of who has access to testing and avoid double-counting when two specimens are taken from one person. But these counts also need to reflect test capacity over time, by allowing for accurate test positivity calculations to be made on a daily or weekly basis.

    To solve this problem, the COVID Tracking Project is suggesting that states use a new unit: test encounters. The Project defines this unit as the number of people tested per day. As Kara Schechtman’s blog post explains, though this term may be new, it’s actually rather intuitive:

    Although the phrase “testing encounters” is unfamiliar, its definition just describes the way we talk about how many times people have been “tested for COVID-19” in everyday life. If an individual had been tested once a week for a month, she would likely say she had been tested four times, even if she had been swabbed seven times (counted as seven tests if we count in specimens), and even though she is just one person (counted as one test if we count in unique people). In this case, that commonsense understanding is also best for the data.

    To arrive at a “testing encounters” count, state public health departments would need to deduplicate multiple specimens from the same person, but only if those multiple specimens were taken on the same day. “Testing encounters” counts over time would accurately reflect a state’s testing capacity, without any artificial inflation of numbers. And, as a bonus, such counts would align with public understanding of what it’s like to get tested for COVID-19—making them easier for journalists like myself to explain to our readers.

    What is your state doing?

    The COVID Tracking Project currently reports total test encounters for five states—Colorado, Rhode Island, Virginia, New York, and Washington—along with the District of Columbia. Other states may report similar metrics, but have not yet been verified to match the Project’s definition.

    You can find up-to-date information about which units are reported for each state on a new website page conveniently titled, “How We Report Total Tests.” The page notes that the Project prioritizes testing capacity in choosing which state counts to foreground in its public dataset:

    Where we must choose a unit for total tests reporting, we are prioritizing units of test encounters and specimens above people—a change which we believe will provide the most useful measure of each jurisdiction’s testing capacity.

    Also, if you’ve visited the COVID Tracking Project’s website recently, you might have noticed that the state data pages have seen a bit of a redesign, in order to make it clear exactly which units each state is using. Each state’s data presentation now includes all three units, with easy-to-click definition popups for each one:

    Screenshot of the COVID Tracking Project page for New York.

    I recommend checking out your state’s page to see which units your public health department is using for COVID-19 tests, as well as any notes on major reporting changes (outlined below the state’s data boxes). You can read more about the site redesign here.

    When my coworker asked me how many people in the U.S. have been tested for COVID-19, I wasn’t able to give him a precise answer. The lack of standards around testing units and deduplication methods, as well as the federal government’s failure to be a leader in this work, have made it difficult to comprehensively report on testing in America. But if people—and I mean readers like you, not just data nerds like me—make testing units part of their regular COVID-19 conversations, we can help raise awareness on this issue. We can push our local public health departments to standardize with each other, or at least get better about telling us exactly what they’re doing to give us the numbers they put up on dashboards every day.

    more testing data

  • What’s up with testing in Texas?

    The COVID Tracking Project published a blog post this week in which three of our resident Texas experts, Conor Kelly, Judith Oppenheim, and Pat Kelly, describe a dramatic shift in Texas testing numbers which has taken place in the past two weeks.

    On August 2, the number of tests reported by Texas’s Department of State Health Services (DSHS) began to plummet. The state went from a reported 60,000 tests per day at the end of July to about half that number by August 12. Conor, Judith, and Pat explain that this overall drop coincides with a drop in tests that DSHS classifies as “pending assessment,” meaning they have not yet been assigned to a county. Total tests reported by individual Texas counties, meanwhile, have continued to rise.

    Although about 85,000 “pending assessment” tests were logged on August 13 to fill Texas’s backlog, this number does not fully add up to the total drop. For full transparency in Texas, DSHS needs to explain exactly how they define “pending assessment” tests, how tests are reclassified from “pending” to being logged in a particular county, and, if tests are ever removed from the “pending” category without reclassification, when and why that happens. As I mentioned in last week’s issue, DSHS has been known to remove Texans with positive antigen tests from their case count; they could be similarly removing antigen and antibody tests reported by counties from their test count.

    If you live in Texas, have friends and family there, or are simply interested in data issues in one of the country’s biggest outbreak states, I highly recommend giving the full post a read. For more Texas test reporting, check out recent articles from Politico and the Texas Tribune.

    more testing data
  • Antigen tests: fast, cheap, and almost diagnostic

    Antigen tests: fast, cheap, and almost diagnostic

    COVID-19 tests conducted in Texas as of August 8. Screenshot retrieved from the Texas Tests and Hospitals dashboard.

    So far in this pandemic, there have been two main players for determining who has been infected with SARS-CoV-2, the virus which causes COVID-19.

    There are polymerase chain reaction (PCR) tests, or molecular tests, which identify viral genetic material in a patient’s ear, nose, and throat cells. And there are antibody tests, or serology tests, which identify cells produced by a patient’s immune system response in their bloodstream. PCR tests are also called “diagnostic” tests, because they are used to conclusively diagnose patients with COVID-19.

    If you get a positive PCR test result, you know that you currently have the disease; you should begin self-isolating and should tell anyone with whom you recently had in-person contact to do the same. Antibody tests, on the other hand, are not diagnostic: they identify patients who have built up an immune response to COVID-19, likely (but not certainly) because they were infected with it. If you get a positive antibody test result, your local public health department would likely count you as a “probable” or “suspected” case.

    In May, however, a new type of testing came on the scene. The Food and Drug Administration (FDA) authorized its first antigen test on May 9, and its second antigen test on July 6. By the end of July, both types of antigen tests had been distributed to hundreds of nursing homes across the country.

    What are antigen tests? Antigen tests, like PCR tests, involve putting a swab up a patient’s nose. The swab takes a sample of potentially infected cells; the sample is then placed in a special chemical solution that breaks down the cells and flags the presence of antigens, unique pieces of the SARS-CoV-2 virus which normally live on the outside of the virus’ structure and are a key piece of immune system response. The testing process can be done in about fifteen minutes, and does not require the complex equipment needed to perform a PCR test.

    This ten-minute video from Medmastery gives a detailed explanation of how antigen tests work. You can also see a brief overview of how antigen tests compare to PCR and antibody tests here:

    Proponents of antigen tests suggest that these tests may one day become so readily available and so easy to use that high-risk workers and those in outbreak areas could test themselves before leaving the house. FDA leaders point out in their May 9 statement about the first authorized antigen test:

    Antigen tests are also important in the overall response against COVID-19 as they can generally be produced at a lower cost than PCR tests and once multiple manufacturers enter the market, can potentially scale to test millions of Americans per day due to their simpler design, helping our country better identify infection rates closer to real time.

    However, while antigen tests are technically diagnostic—they can tell you if you have COVID-19 right now—they do not meet the epidemiological gold standard for accurate testing. Antigen tests have a high specificity, meaning that they do not identify many false positives; if you receive a positive COVID-19 antigen test result, you can be pretty sure your result is correct. But they have a lower sensitivity than PCR tests, meaning that these tests may miss identifying people who are, in fact, infected with SARS-CoV-2. If you receive a negative COVID-19 antigen test result, but you have symptoms that match the disease or recently came into contact with someone who was infected, an epidemiologist would advise you to check your result by getting a PCR test.

    Antigen tests are useful in quickly identifying COVID-19 patients who may be isolated and begin receiving treatment. And, once these tests are more readily available, they will useful in determining infection rates in a broad population. But because of that low test sensitivity, someone with a positive antigen test result cannot be considered a “confirmed case of COVID-19” by public health departments.

    Who is conducting antigen tests? As I covered in last week’s issue of this newsletter, nursing homes are doing antigen tests, big time. On July 14, the Trump Administration and the Department of Health and Human Services (HHS) announced that COVID-19 antigen tests would be distributed to nursing homes in hotspot areas. On July 31, the CDC’s Dr. Robert Redfield claimed in the congressional subcommittee hearing on national coronavirus response that nearly one million of these test kits had already been distributed.

    The Associated Press reported on August 4 that, according to HHS’s Admiral Brett Giroir, this distribution program is on track to get 2,400 antigen test machines and test kits to go with them out to nursing homes by mid-August. However, the HHS’s supply only includes enough tests for most nursing homes to test all of their residents once, and all of their staff twice. Many nursing home administrators will need to make their own deals with suppliers, or get support from state public health departments, in order to continue doing antigen testing after their federal supplies run out.

    Antigen tests are also gaining prominence in Texas, where high case rates have put testing in high demand. In an analysis for the Houston Chronicle, published on August 2, Matt Dempsey, Stephanie Lamm, and Jordan Rubio estimated that tens of thousands of COVID-19 cases had been identified by antigen tests across the state. This analysis was based on data from the 11 Texas counties that published independent antigen test counts as of August 2. Texas’ Department of State Health Services (DSHS) only includes cases confirmed by PCR tests in its official total case count—a decision which may be more epidemiologically valid, but has caused confusion at the local level:

    On July 16, DSHS removed almost 3,500 cases from Bexar County’s case totals, saying the cases were “probable” and not confirmed because they were from antigen test results.

    San Antonio officials pushed back.

    “To be clear, this is not an ‘error’ in Metro Health’s reporting,” said Colleen Bridger, San Antonio’s interim director of public health, in a press release. “This is a disagreement over what should be reported in total counts.”

    On August 8, DSHS began reporting antigen tests on its Texas Tests and Hospitals dashboard. The August 8 numbers include about 14,000 total antigen tests, with about 2,000 positive results. Based on the Houston Chronicle’s analysis, this is likely a significant undercount—but at least Texas is starting to publish some numbers.

    How are antigen test results being reported nationally? Outside of Texas, antigen test numbers are hard to come by. As of the time I send this newsletter, only two other states report official antigen test counts: Kentucky and Utah. Kentucky reports 459 antigen tests as of August 8 (they do not report how many of these tests were positive). Utah reports about 5,000 people tested with antigen tests as of August 4, with about 500 of those people receiving a positive result.

    At the COVID Tracking Project, we have an important procedure: when folks on the data entry team notice that something new is happening with COVID-19 data—say, a new type of test gets approved by the FDA, or hospitals undergo a major change in their reporting protocol—we ask our outreach team, a group of reporters affiliated with the project, to write to every state public health department and ask them how they’re dealing with the change. Most states public health departments have now received questions about antigen testing (and pool testing, but that’s the subject for another newsletter). Answers generally fall in the range of, “We’re not doing antigen testing,” “We’re not doing it at the state level but some commercial labs are,” and “We’re starting to monitor it and include positive antigen tests as probable cases.”

    Pennsylvania is one example of the third approach:

    https://twitter.com/sarambsimon/status/1292225693795876864

    It’s not bad that states are including positive antigen tests as probable cases—as I said earlier, antigen tests are not accurate enough to confirm a case of COVID-19. But when states combine results from different test types in a single count, it is difficult to accurately calculate test positivity rates, testing rates per population, and other important metrics. COVID Tracking Project founders Alexis Madrigal and Rob Meyer explained this issue in detail back in May, when some states (and the CDC) were combining PCR and antibody test results. The same basic principle still applies: each test is used for a different purpose and has a different level of accuracy, and so its results should be reported separately.

    And what about those thousands of antigen tests that were distributed to nursing homes? As I reported in last week’s issue, the national Nursing Home COVID-19 Public File does not specify what types of tests nursing homes are using to identify cases, nor do state-reported datasets on COVID-19 in nursing homes. A FAQ document put out by the Center for Medicare and Medicaid Services (CMS) states that nursing homes are required to “report the results of the COVID-19 tests that they conduct to the appropriate federal, state, or local public health agencies.” This includes, presumably, state public health departments and the HHS. But it is unclear whether either HHS’s or CMS’s datasets will be adjusted to include antigen test counts. I reached out to CMS’s press office asking about these results, and have yet to receive a response.

    This is likely only the beginning for antigen tests. Politico reported earlier today that Admiral Giroir “hopes to have 20 million rapid point-of-care tests available per month by September.” Scientists quoted in a recent New York Times article cite antigen tests as a key technology for improving America’s testing speed. Both manufacturers producing FDA-approved antigen tests, Quidel and BD, cite supply issues which will make it difficult for them to meet demand from nursing homes and local public health departments. Still, the federal government has made antigen tests a priority, and I predict that their prevalence will only grow. COVID-19 data producers must adjust their reporting accordingly.

    Support the COVID Data Dispatch

    more testing data

  • Featured Sources, Aug. 2

    These sources, along with all others featured in previous weeks, are included in the COVID-19 Data Dispatch resource list.

    • Public health departments, underfunded and under threat: This week, Kaiser Health News (KHN) data reporter Hannah Recht released the dataset behind KHN and The Associated Press’s recent feature on how local public health departments in the U.S. have been left unprepared to face COVID-19. The dataset includes six files examining spending and staffing at public health departments across the country.
    • COVID-19 testing sites: The healthcare company Castlight has built a comprehensive database of COVID-19 testing sites in the U.S., down to the ZIP Code level. Castlight’s Tableau dashboard allows users to explore this database by county and compare the number of available test sites with current case counts. This dataset was cited in a recent 538 article on testing disparities.
    • The CoronaVirusFacts Alliance Database: Since the start of the pandemic, Poynter’s International Fact-Checking Network has connected fact-checkers in over 70 countries working to correct COVID-19 misinformation. The results of these fact-checkers’ work are compiled in a database, which you can search by country, fact rating, and topic.
    All featured sources