Two Abbott BinaxNOW antigen tests used by Betsy and her sister during their trip, demonstrating single-line negative results.
As I mentioned in last week’s newsletter, I went on vacation during the last week of August. I traveled to California to visit my younger sister, who was living there over the summer for an internship. We spent a couple of days in the Bay Area, then drove to Southern California, hit a few different destinations, and finally flew out of Los Angeles International Airport (LAX).
Both my sister and I are fully vaccinated, of course, but we still took a number of precautions to make our trip as safe as possible in light of the current COVID-19 surge. I wanted to share what we did in case it’s useful for others thinking about travel right now.
Key precautions included:
Limited exposure and tests before traveling: For ten days before my flight out to California, I avoided all unmasked indoor activities, including small gatherings with fully vaccinated friends. Closer to my flight, I also avoided other activities outside my home (grocery shopping, taking the subway, and the like). I got a PCR test at a NYC Health site two days before my flight, and didn’t leave my apartment between that test and heading out to the airport.
Rapid testing every two days during the trip: I carried three packs’ worth of Abbott BinaxNOW rapid antigen tests in my suitcase, and bought a couple more packs in California. Starting two days after I arrived, my sister and I tested ourselves every other day. While antigen tests like these are more likely to give you a false negative than PCR tests, testing frequently increases your chances of catching a case—and catching it early in your disease progression, when you’re most likely to infect others. The tests were easy to incorporate into our routines, especially once we got used to the swabbing process.
Sticking to outdoor activities: Our trip included a lot of outside time: hiking in a redwood forest, hitting up Southern California beaches, camping for a night at Joshua Tree National Park, and lots of outdoor dining. The riskiest two activities of the week, we agreed, were an afternoon at Monterey Bay Aquarium and a day at Universal Studios in LA. But both of those establishments did a great job of reminding all patrons (including the vaccinated) to keep their masks on inside—following current mask regulations in the Bay Area and LA County.
Double-masking indoors: On both my flights and whenever I needed to be inside for a longer period of time, I wore two masks: a surgical mask and a fitted, multi-layer cloth mask. Studies suggest that cloth masks on their own are not the best option for protecting against Delta, but layering a cloth and surgical mask can increase protection. I especially like to layer masks because surgical masks typically don’t fit well on my face; when I add a cloth mask on top, the fit becomes much more snug.
Precautions when visiting relatives: My sister and I visited a few relatives over the course of our trip. For these visits, we basically followed the precautions that health experts recommended last Thanksgiving: eat outside whenever possible, mask up in common areas inside, open windows for extra ventilation, etc. We also organized the trip so that no visits occurred after Universal Studios, which we had deemed the riskiest activity of the week.
Extra testing after traveling: Flying out of LAX proved to be incredibly stressful, as neither the airport nor our airline put in effort to enforce mask guidance. My sister and I even gave extra masks to two fellow travelers at the check-in desk, because the airline apparently did not have any extras available. Because of this extra COVID-19 exposure risk, I did some extra testing upon my return to the East coast: daily rapid tests for two days in a row, followed by a PCR test.
hi @JetBlue! your mask availability and enforcement is apparently so bad that my sister and I had to literally GIVE TWO OF OUR OWN EXTRA MASKS to fellow travelers at the (very crowded) LAX check-in desk. this is extremely irresponsible in the face of the current covid surge.
Personally, this will probably be my last major trip for a while. Conferences that I’d planned to attend in the fall have been moved to all-virtual formats, and I’m lucky to live close enough to my parents that I don’t need a flight to see them for the holidays. If I were to travel again, though, I’d likely follow these same precautions—with adjustments based on COVID-19 infection rates at my destination. Also, I would probably avoid LAX.
Antigen test kit image via Dronepicr // Wikimedia Commons.
This week, I had a story on COVID-19 testing published in Slate’s Future Tense vertical. The piece explores how testing will change in the next few months as more Americans become vaccinated and rapid tests become more widely available, with a practical focus: how should you interpret the test numbers on your local COVID-19 dashboard?
Overall, I found, we will need to keep getting tested even post-vaccination. But the purpose of testing may shift, for many of us in the U.S., from diagnostic testing—a test to figure out if you are currently sick with COVID-19—to more screening and surveillance testing—tests to identify case trends and stomp out outbreaks in a broader community. This shift may be aided by the rise of rapid, at-home tests, which are becoming much more widely available thanks to investment from the federal government. Just this week, USA Today reported that at-home tests will soon be sold at national pharmacies CVS, Walgreens, and Walmart.
In the CDD today, I’m excited to share one of the interviews I conducted for the piece, with Dan Larremore, a statistician at the University of Colorado and long-time advocate for the potential of rapid tests. We talked on April 2, just days after two major developments in the testing space: the FDA gave Emergency Use Authorization to several rapid tests for over-the-counter use, and the CDC and NIH announced a massive study to investigate how well these tests work for population-level screening. (One more piece of context: when we talked, case numbers were rising at a more concerning rate than they are now.)
Larremore and I talked about his reaction to the rapid test news, how to interpret testing numbers, other new test types that may come on the market, and more.
The interview below has been lightly edited and condensed for clarity.
Betsy Ladyzhets: First question is, since a lot of this piece is meant to be about the numbers of testing, what would you consider the most useful metric or metrics right now to how successful testing is, whether that’s test positivity or other things?
Dan Larremore: I think about testing for three different reasons. One reason is information about the trajectory of the pandemic, which is things like test positivity rates, number of new cases. We test to kind-of get our bearings in the movement of the pandemic. The second reason that we test is more at the individual level, but it’s still that [same] information. And that is, I would test because I want to know, am I sick? Might I give the disease to somebody else? Or, can my current symptoms be explained by being infected? So the first two are informational.
The third reason that we might test is specifically just to break transmission chains, which is more like testing as an answer, not as a question. And so, for at-home testing, for serial testing, for the regular kind of testing that we have to do to be on campus here [at the University of Colorado]… To me, that’s much more about an intervention to slow down transmission than it is about gaining information.
So, that’s a roundabout way of answering your question. But I think, in terms of what numbers to look at, it really matters what the intention of the testing is. So if people continue to take the pandemic seriously, and continue to, like, get tested regularly, or get tested when they feel sick, then those sort of daily case numbers will remain useful and interesting. And therefore the test positivity rate information will continue to tell us something about the trajectory of the pandemic. Does that answer your question?
BL: Yeah, I think that does answer my question. Because I think that was one of the big kind of questions I had going into this story is like, is test positivity still useful if maybe, as people get vaccinated, they stopped thinking they need to get tested or as other dynamics change. But yeah, I had another person who I’ve interviewed for this story also had kind of a hierarchy description [of testing], so that definitely is a useful thing to think about.
DL: Knowing why people come in to get tested just gives you so much more interpretability of like, what the numbers mean. Here on campus, if I want to be here on campus, I need to get tested weekly. I’ve been vaccinated, one dose, dose number two soon. But nevertheless, I still need to spit in the tube every week, and they test it. So, the test positivity rates here on campus are minuscule. Because with high compliance, everybody gets tested, so the denominator in that positivity rate is huge. At a drive up site, or at a doctor’s office, where people are coming in because they feel sick, the test positivity rates are going to be a lot higher.
And I think, as people get more and more vaccinated—regardless of the case counts—as the pandemic feels like it’s winding down, I think people may be less likely to get tested. And so you can imagine test positivity rate being driven by, not just what the virus is doing, but a lot of the human behavior as well… I guess, the way that you can put it is, that you can see changes in the test positivity that are driven entirely by human behavior with respect to getting tested at all. And not so much about changes in the virus.
BL: That makes sense. And I think it’s also about access, right? Are people able to go and get tested if they feel sick? Or if you’re thinking about schools and workplaces, is their employer having everyone get tested once a week? Is that something we’re going to see more of now that we have—like, literally earlier this week, the FDA gave EUA to a bunch of antigen tests, right?
DL: Yeah, for at-home use.
BL: Right. Do you think we’re gonna finally see that massive use of rapid tests that experts have wanted to see since, like, the summer?
DL: A lot of people, myself included, have been excited about the possibility of at-home repeated antigen testing, as a way to really take community transmission levels and push them down. Because we know that asymptomatic transmission occurs, we know that getting people results rapidly is absolutely critical. Because four-day-old information is totally useless if you have infected people in those four days.
I don’t know how useful those test kits are going to be right away, like, right now, given that we just now have an EUA for that kind of at home use that we’ve been hoping for for a long time. But at the same time, cases are shooting up due to these new variants around the U.S. and more importantly, around the world. So, I think these tools are still going to be useful, especially this fall, as we get a lot more kids in school. And we start bringing people together, temperatures starting to cool off, more people are indoors. I think that having the rapid test as a screening tool will still be valuable, particularly if we see limited uptake of the vaccine.
BL: That makes sense… Another question around rapid tests is, that I know in the U.S., it’s really hard to get comprehensive data on them. I volunteered at the COVID Tracking Project, and I wrote [a blog post] about the problem of antigen test data. So I was curious as to how we will know how well the tests are working. And if there are any specific studies that you’re watching or data sources that you recommend, in terms of, like, knowing if people are actually using these at home tests.
DL: I am excited about… On the 30th, there was a press release about the NIH and CDC rolling out at-home testing to two cities.
BL: I saw that, yeah!
DL: The work that Michael Mina and I did last year was showing that, at the individual level, the trade off between test sensitivity and turnaround time, should really tell us that turnaround time is critical. Like, theoretically, at the individual person level, the rapid test idea is really, really good. What we’ve not yet seen—outside of Slovakia—is the ability to flood the zone with tests, as Fauci put it, and just catch as many positives as possible and drive the epi curve downward, just because of the rapid tests.
My feeling is that, really, the proof is in the pudding. If people can, at a community level, use a boatload of rapid tests regularly for a few weeks, and we can watch the new cases spike as we find those positives and then crash as we break all the transmission chains. That’s the key thing. That’s the key thing that I’m looking out for in these new trials.
The Slovakia folks showed some of the limitations of this [strategy]. There’s a paper in Science where they wrote up their results. And basically what they found was the rapid testing worked really well, but the problem was on the isolation side. [Not everyone who tests positive can truly and effectively isolate.] In the short term, while they still had the supplies, these three waves of everybody in the country getting tested, worked like they were supposed to work, worked like the theory said. However, once you stop testing, you take your foot off the brakes, things re-accelerate. The second thing is that a lot of folks in particularly rural areas were like, okay, well, I’m positive, but I live with my family. How am I going to isolate? So unlike on a college campus here, where when somebody tests positive, we have a separate dorm set up for them for two weeks—in real life, that isolation stuff is going to be harder.
BL: So it’s kind-of like, you need to pair it with the social services aspect, or some other way to help people out. I know, in New York, there’s a hotel room program, where if you test positive, you can contact the City Department of Health, and they’ll like, put you up in a hotel for two weeks. I don’t know how much it gets used, but it definitely seems like something that should be around in more places.
DL: I mean, even if it’s just supporting people, by telling them like, this is gonna be awkward, but wear your mask at home, and don’t hang out with your family. Go watch TV in the basement, or, you know, otherwise keep distance from people. Whatever these interventions are, they can help. And we know that what we call the secondary attack rate is actually not that high. So, even among people who live in the same household, even among spouses, if one person is sick, that doesn’t mean that the other person definitely gets it. It’s only [around 20%] chance that they do. We looked at pairs of roommates here on campus, and studied [transmission between] them. And even among roommates, the secondary attack rate is not that high [20-28%].
BL: Well, that brings me to another question I wanted to ask you, which is how public health communication around testing either is changing or should be changing in order to express like, okay, maybe you’ve been vaccinated, but you still need to get tested, or what needs to be communicated about these at home tests, or any other messages that you think are important to be conveying.
DL: One thing that’s important is that we know that we need to keep our eye on the variants. And evidence is emerging… that there are some breakthrough variants, they are less well handled by the vaccine, even though the vaccine works really well… So, testing is going to remain important, even as, more broadly, the vaccine protects people from the most severe disease.
We’re seeing a really interesting split right now, right? Where like, cases are going up, and we expect hospitalizations to then go up, and then mortality to go up. But I wonder if we’re gonna see that in the U.S. this time. Like, week on week, mortality continues to go down. And the question is, are we gonna get it again?
BL: Is it gonna go up again?
DL: Or did we vaccinate enough of the high-risk people that the mortality stays flat even while cases go up?
BL: I definitely think there’s going to be kind-of a demographic aspect of it. Like,I’ve seen charts where people do, with the HHS hospitalization data, they publish it by age. So, you can see that hospitalizations are going way up in people ages, like, 18 to 30, but not so much in seniors. Although, kind-of tangentially, one data gap that I get annoyed by is that there’s very little demographic data for testing. Like, if you look at race data, for example, there are maybe five states that publish testing data by race and ethnicity. And there’s not a lot of it by age. So it’s kind-of hard to track patterns there.
DL: I mean, I don’t know what the right messaging is around testing, other than, [if you feel like you’re sick], if you have the symptoms, you’ve got to go get tested. It doesn’t matter if you’re vaccinated or not. If you feel sick, you should go get tested. If you are going to be around somebody who you know is really vulnerable, if testing is available, go get tested.
I still think it’s a valuable intervention, especially in places where vaccine uptake is low or vaccine availability is low. But I think the question is, like, really, how long is—what’s the expiration date on recommendations about testing? If 70% of the US is vaccinated by July, let’s say, does it still make sense to recommend a huge amount of testing? And I don’t know the answer to that. What I would like to see is people equipped for this fall, if there are spikes in cases, or if there are variants that are circulating even among vaccinated people, it would be amazing if, like, a local public health authority could tell everybody, this Sunday, I want everybody to use your rapid at-home test. Report your results anonymously to this number. And, if you test positive, take it seriously, take precautions. Measures like that could preserve privacy, while still collecting that key surveillance data and crashing the epidemic curve.
BL: Right, that would be really cool to see. Are there any other types of COVID tests or surveillance methods that you think might become more useful and more prevalent in the next few months or heading into the fall?
DL: Yeah, there are two kinds. So, one key point about at-home tests is that privacy-preserving aspect. Like, I trust local public health here. But I tend to vote left of center and generally trust the government. I live in Colorado, and not everybody feels that way. There’s definitely a strong libertarian independent streak. And I feel like one of the key advantages of at-home tests is that they appeal to that kind of person. They empower a person and their family to make health decisions, and they give you the information. But they don’t necessarily get recorded by something like COVID Tracking Project or HHS.
BL: Yeah.
DL: There’s less visibility for authorities, but for some folks who want information that they can act on to protect themselves and others, then that’s going to be fine. And so, as much as I would like to know exactly what is happening with the pandemic, if the trade-off for lower cases is that we don’t know about a lot of cases among folks who would rather not report their data, I think that’s a fair trade-off.
BL: Yeah, that makes sense.
DL: But it’s sort of like uncomfortable for me, who likes the data to say that. You know what I mean?
BL: It reminds me of the conversation around exposure notification apps. I talked to someone who works on those apps recently, and she kind-of said the same thing, that she would rather have everyone using the app than really good data from a tiny subset of the population that’s okay with their privacy being violated.
DL: A big thing to me is that, we know that the pandemic has been political. And I don’t see any reason why we can’t have solutions that work for the person who votes left of center, the person who votes in the middle, and the person who votes right of center.
But the other kind of tests that I’m excited about, only because they’re extremely cheap and really easy, are anosmia screens. We know that loss of sense of smell is highly specific to COVID. If you don’t have a stuffy nose, and you [suddenly] can’t smell things, you probably have COVID. So, there are companies that produce, like, a little card with a scratch-and-sniff quiz. You don’t know what’s behind the panel, but you scratch, pull up a smartphone app, and then say what you think the smell is from a multiple choice test.
One of the cool things is that anosmia only occurs in around 40% of people [with COVID] if you ask them to self-report. But if you give them one of these objective quizzes, the prevalence of anosmia as a symptom goes up to [around 75 or 80%], depending on which study you look at. The important thing is that those cards cost 25 cents apiece, and multiple people can use the same card. It’s literally a scratch-and-sniff with an online quiz.
BL: That’s incredible. Do you know if there are tests like that that are up for EUA?
DL: Yeah. Roy Parker, Michael Mina, and I collaborated with a great team to write this paper last year on typical COVID testing [PCR, LAMP, and antigen testing], test sensitivity and frequency and turnaround time. Then, Roy and I teamed up with Derek Toomre at Yale School of Medicine, and took the same idea and said, well, what if we use frequent, repeated anosmia screening tests?
One of the things that I like about those is that they’re cheap. But another thing that I like about them is that nobody thinks that [a smell test is] the same thing as a proper COVID test. You can’t get them confused. [This is important because one of the questions with rapid tests is how people may interpret a negative test—they might be infected, but the virus is at a low level. So if they get a negative rapid test and then go to the gym, the test could actually have an unintended effect. But if an anosmia test tells you that you still have your sense of smell… People understand that just because you still have your sense of smell doesn’t mean that you’re COVID-free. There’s lower risk of unintended consequences.]
Anyway, I think the messaging around those [anosmia] tests is easy. They’re easy to use, you can do them at home. And they can print them for, like, a quarter apiece. So the modeling suggests that they could be pretty effective and really cheap. You could literally mail somebody a stack of 10 of these things to everybody in the U.S.
BL: And do it once a week!
DL: Yeah, do it once a week. So, that’s another kind of test that I would like to see out there. The company that Derek started, that makes those [tests], just won one of the XPRIZEs for COVID testing. So, I think that’s cool. It’s a more creative kind of test, and it’s inexpensive.
BL: Sweet. So, that was all the questions that I had. Is there anything you think is important on this topic that I should know for this article?
DL: I just think it’s really important to keep a global perspective… As with vaccines, we know there are inequities within the U.S. But there are definitely global inequities. And while we might feel like we’re on a glide path to herd immunity through vaccination here in the States, things look very different in the rest of the world. And so, the use of these tests may become more important this fall, we may get more variants globally, even as the U.S. cases go down. I think it’s an ongoing story, even if hospitalizations and deaths continue to drop here in the States.
BL: And those tests you’re talking about that are cheaper and easier to use are useful in many places, not just here.
DL: Yeah, that’s the hope… I feel generally optimistic about [the state of the pandemic], but like, hesitant.
BL: I feel you. It’s definitely weird to see everyone getting very excited about the summer when I’m kind-of sitting here in my COVID reporting bubble, like, ahhh, not there yet.
DL: Well, one of the hardest parts, I think, for public health officials is going to be, if cases are going up, but mortality and hospitalizations are flat or going down. If COVID is only making people sick, but it’s not hospitalizing and killing them. Then, like, do we just reopen everything? Do we open up the schools? That’s tough when we’ve been acclimated to keep our foot on the brakes as much as possible.
Since our main stories this week focused on NYC, here are a couple of updates from the federal public health agencies.
CDC releases demographic vaccination data: This past Monday, the CDC published a MMWR report on characteristics of Americans vaccinated during the first month of our vaccine rollout, December 14 to January 14. Race/ethnicity data are missing for almost half of the vaccinations recorded here, but the existing data show white Americans getting vaccinated at higher rates than Black and Hispanic/Latino Americans.
CDC vaccination dashboard now includes time series: As of Thursday, the CDC’s COVID Data Tracker now includes a page called “Vaccination Trends,” which shows daily counts of total doses administered, people receiving 1 dose, and people receiving 2 doses. The doses are assigned to the date they’re administered, not the date they’re reported, so they may not match time series on other dashboards. Time series aren’t yet available for individual states.
U.S. Department of Education surveying COVID-19’s impact for students: On Friday, the Department of Education announced that the Institute of Education Sciences would collect “vital data on the impact of COVID-19 on students and the status of in-person learning.” Data gathered in the survey will include the share of America’s schools that are open in-person vs. remote, enrollment by various demographic metrics, attendance rates, and information on the logistics of in-person and remote instruction. Notably, the survey does not promise to collect data on COVID-19 cases, hospitalizations, and deaths associated with K-12 schools.
At-home tests are coming: This week, the White House announced that the new administration has contracted with diagnostic company Ellume and six other suppliers to produce tens of millions of at-home tests by the end of the summer. Ellume’s test is an antigen test, and, though the other test suppliers haven’t yet been announced, we can assume they also make antigen or rapid nucleic acid amplification tests; neither test type is currently reported by the federal government. If these tests do become a significant part of America’s COVID-19 response—and some experts are skeptical that this will happen—a major change in test reporting practices will be needed.
HHS funds new COVID-19 tests: On October 31, HHS and the Department of Defense announced a $12.7 million contract with InBios International, a Seattle-based diagnostic testing company. The contract aims to help InBios increase its production capacity for two COVID-19 tests: a rapid antigen test called the SCoV-2 Ag Detect Kit and an antibody test called the SCoV-2 Detect IgM/IgG Food & Drug Administration (FDA).
HHS distributes antigen tests to HBCUs: At the end of September, the Trump administration announced that Historically Black Colleges and Universities (HBCUs) would be one category of priority sites for the distribution of Abbott BinaxNOW antigen tests, of which the administration has purchased 150 million. This promise is now coming to fruition; HHS announced on October 31 that 389,000 BinaxNow tests have been distributed to 83 HBCUs in 24 states, at no cost to the schools. How these schools will use the tests and report their testing data, however, remains to be seen.
FDA reminds antigen test providers to use them properly: The FDA issued a letter to clinical laboratory staff and health care providers on November 3, reminding them that antigen tests may incur false positives when the instructions for these tests’ use are not correctly followed. FDA recommendations include using antigen tests for symptomatic individuals, handling tests correctly, and using PCR tests to confirm results in low incidence counties. As I’ve discussed in this newsletter before, incorrect use of antigen tests may lead to misleading results that waste clinical resources or instill false confidence in people who receive false negatives.
HHS needs better testing oversight and data: Two new articles in STAT News this past week have discussed COVID-19 test regulation and reporting. An investigation by Kathleen McLaughlin finds that laboratory developed tests, diagnostic tools developed by and for specific facilities, fall in a “regulatory gray area” which makes it easy for innacuracies to slip past the FDA and HHS. Meanwhile, an op-ed by OB-GYN Joia Crear-Perry points out the public health danger in allowing demographic data on testing to be lost when rapid tests are not incoporated into reporting pipelines.
Long-term readers might remember that I devoted an issue to antigen testing back in August. Antigen tests are rapid, diagnostic COVID-19 tests that can be used much more quickly and cheaply than their polymerase chain reaction (PCR) counterparts. They don’t require samples to be sent out to laboratories, and some of these tests don’t even require specialized equipment; Abbott’s antigen test only takes a swab, a testing card, and a reagent, and results are available in 15 minutes.
But these tests have lower sensitivity than PCR tests, meaning that they may miss identifying people who are actually infected with COVID-19 (what epidemiologists call false negatives). They’re also less accurate for asymptomatic patients. In order to carefully examine the potential applications of antigen testing, we need both clear public messaging on how the tests should be used, and accessible public data on how the tests are being used already. Right now, I’m not seeing much of either.
When I first covered antigen testing in this newsletter, only three states were publishing antigen test data. Now, we’re up past ten states with clear antigen test totals, with more states reporting antigen positives or otherwise talking about these tests in their press releases and documentation. Pennsylvania, for example, announced that the governor’s office began distributing 250,000 antigen test kits on October 14.
Meanwhile, antigen tests have become a major part of the national testing strategy. Six tests have received Emergency Use Authorization from the FDA. After Abbott’s antigen test was given this okay-to-distribute in late August, the White House quickly purchased 150 million tests and made plans to distribute them across the country. Context: the U.S. has done about 131 million total tests since the pandemic began, according to the COVID Tracking Project’s most recent count.
Clearly, antigen testing is here—and beginning to scale up. But most states are ill-prepared to report the antigen tests going on in their jurisdictions, and federal public health agencies are barely reporting them at all.
I’ve been closely investigating antigen test reporting for the past few weeks, along with my fellow COVID Tracking Project volunteers Quang Nguyen, Kara Schechtman, and others on the Data Quality team. Our analysis was published this past Monday. I highly recommend you give it a read—or, if you are a local reporter, I highly recommend that you use it to investigate antigen test reporting in your state.
But if you just want a summary, you can check out this Twitter thread:
today, @COVID19Tracking published an analysis of antigen testing by me and @quangpmnguyen, featuring annotations by @karaschechtman. we asked: how are states reporting their antigen tests? the answer is: largely, they aren't. https://t.co/2plOKfaehy (1/11)
First: state antigen test reporting is even less standardized than PCR test reporting. While twelve states and territories do report antigen test totals, nine are combining their antigen test counts with PCR test counts, which makes it difficult to analyze the use of either test type or accurately calculate test positivity rates. The reporting practices in sixteen other states are unclear. And even among those states with antigen test totals, many relegate their totals to obscure parts of their dashboards, fail to publish time series, report misleading test positivity rates, and engage in other practices which make the data difficult for the average dashboard user to interpret.
Second: antigen tests reported by states likely represent significant undercounts. Data reporting inconsistences between the county and state levels in Texas, as well as a lack of test reporting from nursing homes, suggest that antigen tests confuse data pipelines. While on-site test processing is great for patients, it cuts out a lab provider which is set up to report all COVID-19 tests to a local health department. Antigen tests may thus be conducted quickly, then not reported. The most damning evidence for underreporting comes from data reported by test maker Quidel. Here’s how the post explains this:
Data shared with Carnegie Mellon University by test maker Quidel revealed that between May 26 and October 9, 2020, more than 3 million of the company’s antigen tests were used in the United States. During that same period, US states reported less than half a million antigen tests in total. In Texas alone, Quidel reported 932,000 of its tests had been used, but the state reported only 143,000 antigen tests during that same period.
Given that Quidel’s antigen test is one of six in use, the true number of antigen tests performed in the United States between late May and the end of September was likely much, much higher, meaning that only a small fraction are being reported by states.
Again: this is for one of six tests in use. America’s current public health data network can’t even account for three million antigen tests—how will it account for 150 million?
Abbott ID NOW tests, pictured at a health clinic in Williamsburg (my photo).
Two weeks ago, I went to my COVID-19 testing site of choice for a post-Maine trip screening. I walked the now-familiar ten blocks to the site, a private clinic in Williamsburg, and waited at the now-familiar counter for one of the receptionists to be available to check me in.
“Do you want a rapid test?” the receptionist asked.
“No, PCR, please,” I replied.
I had assumed that the “rapid test” she offered was an antigen test. Antigen tests, as I’ve described in this newsletter before, have a low sensitivity, meaning that they may miss identifying people who are actually infected with the novel coronavirus. (These missed cases are called false negatives.) Evidence also suggests that antigen tests will return more false negatives for patients who are asymptomatic. As I was not exhibiting COVID-19 symptoms, and rather wanted to get tested because I had recently traveled out of the state, I was interested in receiving a more accurate test.
But confusion quickly ensued: the rapid test that my clinic offered, as it turned out, was not an antigen test. It was a nucleic acid amplification test—a model manufactured by Abbott called the ID NOW. Like PCR (polymerase chain reaction) tests, this test uses gene amplification to identify genetic material associated with the novel coronavirus in a patient’s mucus sample. But unlike PCR tests, the ID NOW test does not require clinics to send samples out to faraway labs for processing. This test is distributed with small, in-house DNA amplification machines that can provide patients with their results in 15 minutes. I got the result of my ID NOW test later that same afternoon. (And then I got the results of a second test, this one a PCR test which I had asked the clinic to request at a lab, several days later. Both tests were negative.)
I hadn’t heard of Abbott ID NOW tests before last week. But they’re in the news now, connected to what has become America’s most infamous COVID-19 outbreak: President Trump’s White House relied on Abbott ID NOW tests. And they used these tests incorrectly.
No test type specification in molecular testing data
Rapid testing has been the White House’s defense against critiques of COVID-19 carelessness, explains Slate reporter Daniel Politi. Each day, staffers would get tested with Abbott ID NOW tests. Upon receiving negative results, staffers would be cleared to take off their mask and act without consideration for social distancing. Boxes of the ID NOW tests used for this screening have been pictured at the White House since March.
But how accurate are those negative results? If you’re asymptomatic: not very. The ID NOW test is only authorized for use in people with symptoms. A guidance from the Food and Drug Administration (FDA), updated on September 17, specifies:
The ID NOW COVID-19 is authorized for use with respiratory specimens collected from individuals who are suspected of COVID-19 by their healthcare provider within the first seven days of the onset of symptoms.
The majority of White House staffers who were tested with these tests had not been coughing and wheezing, nor had they been suspected of COVID-19 by a healthcare provider. In asymptomatic patients like these, as many as one-third of negatives returned by the ID NOW test may be false negatives. It’s no surprise, then, that the virus spread quickly through an indoor environment where staffers were using rapid tests—but doing little else.
White House staff are not the only people who used the wrong type of test to check their COVID-19 status. I shouldn’t have gotten an Abbott ID NOW test either. But when a nurse at my local clinic, which I saw as a site of trusted health expertise, offered one to me, I didn’t ask too many questions. It’s PCR, I thought. It’ll be accurate.
But first of all: the Abbott ID NOW test is not a PCR test. It’s in the same family as PCR tests (molecular-based, DNA amplification), but it operates on a different paradigm. And second of all, the health care workers at my clinic gave me no indication of how accurate this rapid test was, especially for my asymptomatic self. (Note: I have no hard feelings toward this clinic or any of the workers there. I’m simply using my own experience as an example of how poorly information about test types is conveyed to the public in general.)
What if my PCR test, sent out to a lab, had returned a positive result? I would have traversed Brooklyn, gotten groceries, grabbed coffee at a neighborhood cafe, and attended a protest in Queens that weekend, buoyed by a negative result yet unknowingly spreading the coronavirus to my fellow New Yorkers. And both of my tests would have been accounted for on New York’s testing dashboard in one single tick mark. New York reports tests in encounters, so my two specimens would have been deduplicated as simply “one person tested on September 30.”
I say “would have been” because I am not, in fact, sure that my Abbott ID NOW test made it into the New York Public Health Department’s database at all.
Here’s a line from the press release that Abbott put out on October 2, the day that Trump tested positive:
While we don’t know the details on who was tested and on which devices, we know that more than 11 million Americans have taken the ID NOW test, helping to stop the spread of the virus.
11 million is a pretty big number: if these tests were all included in the national count reported by the COVID Tracking Project, they would make up about 10% of the total. But are the ID NOW tests included in the COVID Tracking Project’s count? It is difficult to say. The majority of state public health departments, the sources for the COVID Tracking Project’s data, simply report total counts of nucleic acid-based tests, if they specify their test type at all.
State COVID-19 dashboards fail to separate out test counts by method or manufacturer. Some states, such as Maine and Illinois, acknowledge that their test counts include all “nucleic acid-based tests,” not only the PCR tests which fall into this category. Other states, such as Oklahoma and Florida, report detailed data about the testing capacity of individual labs and point-of-care facilities, but fail to supply the testing models used at each location. South Carolina acknowledges a small number of “unknown” tests, for which the public health department is investigating possible categorizations. The Department of Health and Human Services, meanwhile, only reports PCR tests, absent counts of any other molecular test type.
And, returning to Abbott’s press release: the manufacturer admits that they “don’t know the details on who was tested and on which devices.” This leaves a lot of open questions about how rapid testing data are being collected; Abbott seems to imply that even the manufacturer itself does not have access to specific information about where and how their tests are in use. If I had to guess, I’d say that 11 million figure comes from bulk test orders sent out by facilities like my local clinic.
It’s great for healthcare responses that Abbott tests can be processed quickly on-site, in facilities like a healthcare clinic or a major political site. But public health departments don’t have relationships with clinics—they have relationships with testing laboratories. When tests aren’t sent out to a lab, it’s easy for them to fall through gaps in a data pipeline which relies on complex electronic reporting systems. (This is also a problem for antigen tests.)
So, the problem stands: Abbott ID NOW tests are in use across the country. 11 million of them have been distributed since March. But where are the tests being used, how many of them have returned positive results, and are they being used correctly—to quickly diagnose patients with COVID-19 symptoms? Without these data, it is difficult to determine whether Abbott’s test should be part of America’s testing strategy going forward.
Conflating antigen and PCR tests
I can’t discuss the data pitfalls of rapid COVID-19 testing without devoting a few paragraphs to antigen tests.
Five days before his positive COVID-19 test was made public, President Trump announced a plan to deploy 150 million antigen tests across the country. 100 million tests will go to state and territory public health departments, while the remaining 50 million will go to facilities supporting vulnerable populations such as nursing homes, hospice care facilities, and historically Black colleges. Trump discussed how useful antigen tests could be for screening, suggesting that teachers could be tested regularly.
The tests Trump praised are rapid antigen tests manufactured by Abbott, which received FDA Emergency Use Authorization at the end of August. Abbott’s antigen tests are cheap—each one costs $5. And, like the ID NOW tests, they’re fast—patients receive results in 15 minutes. But, also like the ID NOW tests, antigen tests are more effective for symptomatic people.
Here is how Alexis Madrigal and Rob Meyer explain it, in an article for The Atlantic published this past week:
If distributed en masse and used to screen asymptomatic people, these antigen tests will deliver hundreds of thousands—if not millions—of false results, they say. False negatives could lead to reckless behavior by people who don’t know they’re sick. False positives can also put people at risk: If a virus-free nursing-home resident with a false positive is placed in a COVID-19 ward, that person could become infected.
This is not to say that antigen tests are useless; their utility is still being debated in epidemiological and public health circles. The advantages of these cheap, fast tests may outweigh the dangers of their low sensitivity. But to truly understand this problem, we need access to better antigen test data—not just 60 tests from Nevada nursing homes (the sample size upon which that state’s decision was made).
If 11 million Abbott ID NOW tests are a data concern, 150 million Abbott antigen tests are a data minefield. For the past month, I’ve been working on an analysis for the COVID Tracking Project which covers how states are reporting—and failing to report—antigen test results. This analysis is set to be released in the next week, in all its detailed glory. But I can tell you now that the majority of states are not reporting antigen tests in separate counts from DNA-based tests, just as the majority of states are not reporting other types of DNA-based tests as separate from PCR tests. In fact, several states now specifically report that their testing counts combine PCR and antigen tests in one figure.
When two different test types are conflated in a single total figure, it is difficult to analyze the progression of how either test is being used. I can’t tell you how many antigen tests are being put to use across the country, or how effective they are at containing COVID-19 in a given community, if these test numbers are lumped in with another test type. Test lumping also presents a(nother) challenge for test positivity calculations, as antigen and PCR tests have very different sensitivity levels.
And even the few separate antigen test counts that states do report are likely significant undercounts of the actual scale of antigen testing going on in the U.S. As I mentioned earlier, no data reporting pipelines are set up for tests that occur rapidly in point-of-care settings. The Center for Medicare & Medicaid Services (CMS) does not report the number of antigen tests that have occurred in nursing homes, for example.
So far, it appears that state and federal public health agencies alike are unprepared to report the results of antigen tests. Before the White House deploys its 150 million Abbott antigen tests, I hope to see that change.
Utah was one of the first states to begin reporting antigen tests back in early August. The state is also one of only three to report an antigen testing time series, rather than simply the total number of tests conducted. However, the format in which Utah presents these data is… challenging.
Rather than reporting daily antigen test counts—or daily PCR test counts, for that matter—in a table or downloadable spreadsheet, Utah requires users to hover over an interactive chart in an extremely precise fashion. Interactive charts are useful for visualizing data, but far from ideal for accessibility.
Hot tip for anyone interacting with this chart: you can make your life easier by clicking “Compare data on hover,” toggling the chart to show all four of its daily data points at once. (Sad story: I did not learn this strategy until I’d already spent an hour carefully zooming in and around the chart to record all of Utah’s antigen test numbers.)
In related news: keep an eye out for a COVID Tracking Project blog post on antigen testing, likely to be published in the coming week.
The COVID Tracking Project published a blog post this week in which three of our resident Texas experts, Conor Kelly, Judith Oppenheim, and Pat Kelly, describe a dramatic shift in Texas testing numbers which has taken place in the past two weeks.
On August 2, the number of tests reported by Texas’s Department of State Health Services (DSHS) began to plummet. The state went from a reported 60,000 tests per day at the end of July to about half that number by August 12. Conor, Judith, and Pat explain that this overall drop coincides with a drop in tests that DSHS classifies as “pending assessment,” meaning they have not yet been assigned to a county. Total tests reported by individual Texas counties, meanwhile, have continued to rise.
Although about 85,000 “pending assessment” tests were logged on August 13 to fill Texas’s backlog, this number does not fully add up to the total drop. For full transparency in Texas, DSHS needs to explain exactly how they define “pending assessment” tests, how tests are reclassified from “pending” to being logged in a particular county, and, if tests are ever removed from the “pending” category without reclassification, when and why that happens. As I mentioned in last week’s issue, DSHS has been known to remove Texans with positive antigen tests from their case count; they could be similarly removing antigen and antibody tests reported by counties from their test count.
If you live in Texas, have friends and family there, or are simply interested in data issues in one of the country’s biggest outbreak states, I highly recommend giving the full post a read. For more Texas test reporting, check out recent articles from Politico and the Texas Tribune.
So far in this pandemic, there have been two main players for determining who has been infected with SARS-CoV-2, the virus which causes COVID-19.
There are polymerase chain reaction (PCR) tests, or molecular tests, which identify viral genetic material in a patient’s ear, nose, and throat cells. And there are antibody tests, or serology tests, which identify cells produced by a patient’s immune system response in their bloodstream. PCR tests are also called “diagnostic” tests, because they are used to conclusively diagnose patients with COVID-19.
If you get a positive PCR test result, you know that you currently have the disease; you should begin self-isolating and should tell anyone with whom you recently had in-person contact to do the same. Antibody tests, on the other hand, are not diagnostic: they identify patients who have built up an immune response to COVID-19, likely (but not certainly) because they were infected with it. If you get a positive antibody test result, your local public health department would likely count you as a “probable” or “suspected” case.
In May, however, a new type of testing came on the scene. The Food and Drug Administration (FDA) authorized its first antigen test on May 9, and its second antigen test on July 6. By the end of July, both types of antigen tests had been distributed to hundreds of nursing homes across the country.
What are antigen tests? Antigen tests, like PCR tests, involve putting a swab up a patient’s nose. The swab takes a sample of potentially infected cells; the sample is then placed in a special chemical solution that breaks down the cells and flags the presence of antigens, unique pieces of the SARS-CoV-2 virus which normally live on the outside of the virus’ structure and are a key piece of immune system response. The testing process can be done in about fifteen minutes, and does not require the complex equipment needed to perform a PCR test.
This ten-minute video from Medmastery gives a detailed explanation of how antigen tests work. You can also see a brief overview of how antigen tests compare to PCR and antibody tests here:
Proponents of antigen tests suggest that these tests may one day become so readily available and so easy to use that high-risk workers and those in outbreak areas could test themselves before leaving the house. FDA leaders point out in their May 9 statement about the first authorized antigen test:
Antigen tests are also important in the overall response against COVID-19 as they can generally be produced at a lower cost than PCR tests and once multiple manufacturers enter the market, can potentially scale to test millions of Americans per day due to their simpler design, helping our country better identify infection rates closer to real time.
However, while antigen tests are technically diagnostic—they can tell you if you have COVID-19 right now—they do not meet the epidemiological gold standard for accurate testing. Antigen tests have a high specificity, meaning that they do not identify many false positives; if you receive a positive COVID-19 antigen test result, you can be pretty sure your result is correct. But they have a lower sensitivity than PCR tests, meaning that these tests may miss identifying people who are, in fact, infected with SARS-CoV-2. If you receive a negative COVID-19 antigen test result, but you have symptoms that match the disease or recently came into contact with someone who was infected, an epidemiologist would advise you to check your result by getting a PCR test.
Antigen tests are useful in quickly identifying COVID-19 patients who may be isolated and begin receiving treatment. And, once these tests are more readily available, they will useful in determining infection rates in a broad population. But because of that low test sensitivity, someone with a positive antigen test result cannot be considered a “confirmed case of COVID-19” by public health departments.
Who is conducting antigen tests? As I covered in last week’s issue of this newsletter, nursing homes are doing antigen tests, big time. On July 14, the Trump Administration and the Department of Health and Human Services (HHS) announced that COVID-19 antigen tests would be distributed to nursing homes in hotspot areas. On July 31, the CDC’s Dr. Robert Redfield claimed in the congressional subcommittee hearing on national coronavirus response that nearly one million of these test kits had already been distributed.
The Associated Press reported on August 4 that, according to HHS’s Admiral Brett Giroir, this distribution program is on track to get 2,400 antigen test machines and test kits to go with them out to nursing homes by mid-August. However, the HHS’s supply only includes enough tests for most nursing homes to test all of their residents once, and all of their staff twice. Many nursing home administrators will need to make their own deals with suppliers, or get support from state public health departments, in order to continue doing antigen testing after their federal supplies run out.
Antigen tests are also gaining prominence in Texas, where high case rates have put testing in high demand. In an analysis for the Houston Chronicle, published on August 2, Matt Dempsey, Stephanie Lamm, and Jordan Rubio estimated that tens of thousands of COVID-19 cases had been identified by antigen tests across the state. This analysis was based on data from the 11 Texas counties that published independent antigen test counts as of August 2. Texas’ Department of State Health Services (DSHS) only includes cases confirmed by PCR tests in its official total case count—a decision which may be more epidemiologically valid, but has caused confusion at the local level:
On July 16, DSHS removed almost 3,500 cases from Bexar County’s case totals, saying the cases were “probable” and not confirmed because they were from antigen test results.
San Antonio officials pushed back.
“To be clear, this is not an ‘error’ in Metro Health’s reporting,” said Colleen Bridger, San Antonio’s interim director of public health, in a press release. “This is a disagreement over what should be reported in total counts.”
On August 8, DSHS began reporting antigen tests on its Texas Tests and Hospitals dashboard. The August 8 numbers include about 14,000 total antigen tests, with about 2,000 positive results. Based on the Houston Chronicle’s analysis, this is likely a significant undercount—but at least Texas is starting to publish some numbers.
How are antigen test results being reported nationally? Outside of Texas, antigen test numbers are hard to come by. As of the time I send this newsletter, only two other states report official antigen test counts: Kentucky and Utah. Kentucky reports 459 antigen tests as of August 8 (they do not report how many of these tests were positive). Utah reports about 5,000 people tested with antigen tests as of August 4, with about 500 of those people receiving a positive result.
At the COVID Tracking Project, we have an important procedure: when folks on the data entry team notice that something new is happening with COVID-19 data—say, a new type of test gets approved by the FDA, or hospitals undergo a major change in their reporting protocol—we ask our outreach team, a group of reporters affiliated with the project, to write to every state public health department and ask them how they’re dealing with the change. Most states public health departments have now received questions about antigen testing (and pool testing, but that’s the subject for another newsletter). Answers generally fall in the range of, “We’re not doing antigen testing,” “We’re not doing it at the state level but some commercial labs are,” and “We’re starting to monitor it and include positive antigen tests as probable cases.”
Pennsylvania is one example of the third approach:
It’s not bad that states are including positive antigen tests as probable cases—as I said earlier, antigen tests are not accurate enough to confirm a case of COVID-19. But when states combine results from different test types in a single count, it is difficult to accurately calculate test positivity rates, testing rates per population, and other important metrics. COVID Tracking Project founders Alexis Madrigal and Rob Meyer explained this issue in detail back in May, when some states (and the CDC) were combining PCR and antibody test results. The same basic principle still applies: each test is used for a different purpose and has a different level of accuracy, and so its results should be reported separately.
And what about those thousands of antigen tests that were distributed to nursing homes? As I reported in last week’s issue, the national Nursing Home COVID-19 Public File does not specify what types of tests nursing homes are using to identify cases, nor do state-reported datasets on COVID-19 in nursing homes. A FAQ document put out by the Center for Medicare and Medicaid Services (CMS) states that nursing homes are required to “report the results of the COVID-19 tests that they conduct to the appropriate federal, state, or local public health agencies.” This includes, presumably, state public health departments and the HHS. But it is unclear whether either HHS’s or CMS’s datasets will be adjusted to include antigen test counts. I reached out to CMS’s press office asking about these results, and have yet to receive a response.
This is likely only the beginning for antigen tests. Politico reported earlier today that Admiral Giroir “hopes to have 20 million rapid point-of-care tests available per month by September.” Scientists quoted in a recent New York Times article cite antigen tests as a key technology for improving America’s testing speed. Both manufacturers producing FDA-approved antigen tests, Quidel and BD, cite supply issues which will make it difficult for them to meet demand from nursing homes and local public health departments. Still, the federal government has made antigen tests a priority, and I predict that their prevalence will only grow. COVID-19 data producers must adjust their reporting accordingly.
COVID-19 Data Dispatch 