COVID-19 Data Dispatch

Tag: interview

  • A new tracker highlights the racial disparities—and the missing data—in America’s COVID-19 outbreaks

    A new tracker highlights the racial disparities—and the missing data—in America’s COVID-19 outbreaks

    Screenshot of the Health Equity Tracker showing which states are missing race and ethnicity data for COVID-19 cases.

    Two weeks ago, a major new COVID-19 data source came on the scene: the Health Equity Tracker, developed by the Satcher Health Leadership Institute at Morehouse School of Medicine.

    This tracker incorporates data from the CDC, the Census, and other sources to provide comprehensive information on which communities have been hit hardest by COVID-19—and why they are more vulnerable. Notably, it is currently the only place where you can find COVID-19 race/ethnicity case data at the county level.

    I featured this tracker in the CDD the week it launched, but I wanted to dig more into this unique, highly valuable resource. A couple of days ago, I got to do that by talking to Josh Zarrabi, senior software engineer at the Satcher Health Leadership Institute—and a fellow former volunteer with yours truly at the COVID Tracking Project.

    Zarrabi has only been working on the Health Equity Tracker for a couple of months, but he was able to share many insights into how the tracker was designed and how journalists and researchers might use it to look for stories. We talked about the challenges of obtaining good health data broken out by race/ethnicity, communicating data gaps, and more.

    The interview below has been lightly edited and condensed for clarity.

    Betsy Ladyzhets: Give me the backstory on the Health Equity Tracker, like how it got started, how the different stakeholders got involved.

    Josh Zarrabi: At the beginning of the pandemic, the Satcher Health Leadership Institute at Morehouse School of Medicine saw the lack of good COVID data in the country, and especially the lack of racial data. The COVID Tracking Project kind-of tried to solve that as well with the Racial Data Tracker. 

    Morehouse wanted to do something similar. And so they applied for a Google.org grant… After about nine months, the tracker just got released. It went through a couple of different iterations, but what it is now is, it’s a general health equity tracker, so it tracks a couple of different determinants of health. And it really has a focus on equity between races and amplifying marginalized races as much as possible.

    Probably the most innovative thing it does is, it shows COVID rates by race down to the county level. We think that’s relatively hard to find anywhere else. (Editor’s note: It is basically impossible to find anywhere else.)  So that’s probably like the main feature that it has that people care about, but it does track other health metrics. We also have poverty, health insurance, and we try to track diabetes and COPD, but there’s not great data on that, unfortunately, in the United States. We’re planning to add more metrics in the future.

    BL: How does this project build on the COVID Racial Data Tracker? And I know, like APM has a tracker for COVID deaths by race. And there are a couple other similar projects. So what is this one doing that is taking it to the next level?

    JZ: A couple of things. We’re using the CDC restricted dataset. Basically what the dataset looks like is, it’s like a very large CSV file where every single line is an individual COVID case. So we’re able to break it down basically however we want. So we were able to break that down to the county level, state level and national level.

    And what we do is we allow you to compare that [COVID rates] to rates of poverty, and rates of health insurance in different counties. We think that’s pretty innovative, and we’re gonna allow you to compare it to other things in the future. So that’s one thing that we do. And I mean, the second thing that I would say is like, probably makes us stand out the most I would say is our real focus on racial equity, and showing where the data gaps are and how that affects health equity. So what you’ll notice, if you go to our website, we very prominently display the amount of unknown… 

    BL: Yeah, I was gonna ask you about that, because I know the COVID Racial Data Project had similar unknown displays. Why is it so important to be highlighting those unknowns? And what do you want people to really be taking away from those red flag notes?

    JZ: We really try to do our best to display the data in context as much as possible. First of all, the most important thing, I think, is just showing the high percentage of unknown race and ethnicity of COVID cases in the United States. For something like 40% of cases, we don’t know the race and ethnicity of the person who had COVID.

    We want people to really think about that when they look at, for example, you’ll notice that it looks like Black Americans are affected to the exact level of their population. Black Americans look like 12% of the population and 11% of cases. But we don’t know the race of 40% of people who have COVID. And so we really wanted people to think about that when they look at these numbers. And it’s the same for American Indian/Alaskan Native populations. It doesn’t look like they’re that heavily affected in the United States. But that’s why we allow you to break down into the county level, where race is not being reported. And so we really want people to look and say, like, oh, wow, like in Atlanta, 60% of cases are not being counted for race and ethnicity.

    We’re not doing any extrapolation. We’re not multiplying, we’re not like trying to guess the races of unknowns, or anything like that. We really want people to think about that, when they’re saying like, oh, wow, it looks like Native American people are not really heavily affected by COVID. It’s like, no, we just don’t know. We don’t know their races, or those people are just not being reported properly by the health agencies.

    And if you look at places that have high percentages of Black Americans and high percentages of American Indian/Alaskan Natives, you’ll see that those places are the same places that are not reporting the race and ethnicity of the people who had COVID.

    We had a team of about 20 health equity experts advising us throughout the entire project. That’s where those red flags that you see come from. It’s explaining, for example, if you look into deaths for Native American and Alaska Natives, there’s an article about how a lot of American Indian/Alaskan Native people who died are not, are improperly categorized racially, and they’re often categorized as white. And so we have that kind of stuff to really try to put the numbers in context.

    We were only able to do that, because we had this large team of racial equity experts and health equity experts advising us throughout the entire time. And so we really had diverse representation on the project as we were building it, and people who really knew what they were talking about.

    BL: What can public health agencies and also researchers and journalists do to push for better data in this area?

    JZ: The good thing is we are seeing [data completion] get better over time. And so we’ve seen, for example, the percentage of race and ethnicity for cases improved from about 50% to about 60% over the last couple of months.

    And, I mean, really, all you can do is—it’s really a thing that goes down to the county level. So, everybody’s just got to call their county representatives. I’d be like, hey, could you please report the race and ethnicity of the county’s COVID cases to the CDC? Unfortunately, a lot of that work might be too late, because [data were submitted months ago]. But we have seen it get better. And so we’re hoping that, you know, these health agencies are able to do the work and really, like, properly report these cases to the CDC… 

    BL: ‘Cause a lot of it comes from the case identification point, where if you’re not asking on your testing form, what race are you, then you just might not have that information. Or you might be, like, guessing and getting it wrong or something, right?

    JZ:  Yeah, there’s guessing. There’s two different categories of unknown cases—there’s unknown and there’s missing. The vast majority of these cases have filled out unknown [in the line file], which means that the person who’s filling out the data form literally puts “unknown” as the race. We don’t really know exactly what that means in every case. But it could be they didn’t ask, it could be the person didn’t feel comfortable saying it, just said, “I don’t want to tell you my race.” Or it could just be that they just didn’t make an effort to figure out what their race is.

    (Editor’s note: For more on the difficulties of collecting COVID-19 race data, I recommend this article by Caroline Chen at ProPublica.)

    BL: Do you have a sense of how that 60% known cases compares to what the COVID Racial Data Tracker had in compiling from the states?

    JZ: Yeah, I think the COVID Racial Data Tracker was a bit higher [in how many cases had known race/ethnicity]. But the thing is, as far as I understand, the COVID Racial Data Tracker was using aggregate numbers.

    BL: We were looking at the states and then kind-of like, synthesizing their data to the best of our ability, which was pretty challenging because every state had slightly different race and ethnicity categories. There were some states that had almost no unknown cases, but there were some where almost all cases or almost all deaths were unknown. New York, I don’t know if they ever started reporting COVID cases by race.

    JZ: They do to the CDC, I don’t think they report—

    BL: They don’t report it on their own, state public health site.

    JZ: Let me actually check that… Yeah, so New York is not great. They have a 60% unknown rate. [Race and ethnicity is only reported to the CDC for 40% of cases.] Not great. Actually, New York City is pretty good. But the rest of New York State is not doing a good job reporting the race and ethnicity of cases.

    BL: Because I’ve gotten tested here, I know that New York City is good about collecting that [race and ethnicity] from everybody.

    JZ: I was one of those cases in New York City, actually. When [I got called by a contact tracer], I was kind of chatting with them about this. They asked me about my race—I actually became a probable case for COVID, like, the day after I started this job. And [NYC Health] called me, they were like, “What’s your race?” I was like, “Oh, that’s kind of funny, I just started working on this racial data project.” And—this is totally anecdotal. But she told me, most people just refuse to report their race. 

    And then for deaths… 40% of COVID deaths in New York state, they don’t know the race, which is not great. New York is not good compared to the rest of the states. It’s one of the worst states for unknowns.

    BL: Could you tell me more about the process of getting the [restricted] case surveillance data from the CDC and how you’ve been using that?

    JZ: The process of getting it’s not that hard. You just apply, and then they give you access to a GitHub repo, and then you can just use it. Using the data itself is pretty hard because the data files are so large. We were lucky enough to have a team of Google engineers working on this project, they wrote a bunch of Python scripts that analyze the data and aggregate it in a way that the CDC isn’t doing.

    The reason why they restrict the use is because it’s line-by-line data. [Each line is a case.] And the CDC does suppress some of the data because they think it would make those cases identifiable. Still, you’re not allowed to just, like, release the data into the wild, because they want to know who else has track of it. So, we wrote some Python to aggregate the data, in exactly the way you see on the website. We aggregate it to the amount of cases, deaths and hospitalizations per county, per race, essentially. 

    The CDC has been extremely helpful, like, we’ve had a couple of meetings with them. We think we were one of the heaviest users of the data at the beginning, because we pointed out a couple of problems with the data that they actually fixed. So, that’s cool.

    BL: That’s good to hear that they were responsive.

    JZ: Yeah, definitely. We meet with them every couple of weeks. They’re really good partners in this.

    BL: And they update that [case surveillance] dataset once a month?

    JZ: They started doing it every two weeks now. Every other Monday, they update the dataset.

    BL: Could you talk more about the feature of the tracker that lets you compare COVID to other health conditions and insurance rates? I thought that was really unique and worth highlighting.

    JZ: We wanted to really provide the [COVID] numbers in context. And so that’s one way that we thought that we could do that and really show how… These numbers don’t happen, like a high rate of COVID for race doesn’t happen in a vacuum. There are political determinants of health.

    For example, you’ll see everywhere that Hispanic Americans are just by far the most impacted by COVID case-wise. In California especially. And we provide those numbers in context—Hispanic Americans are also much less likely to be insured than white Americans, for example, and much more likely to be in poverty. And, you know, it’s not a crazy surprise that they would also be more likely to have contracted COVID at some point.

    [The comparison feature] was a way that we thought, we would just allow people to really view numbers in context and get a better understanding of what the political situation is on the ground with where these high numbers are happening.

    BL: What are the next conditions that you want to add to the tracker?

    JZ: I want to be careful, because we can’t make any promises. But we’re talking about adding smoking rates, maybe. [The challenge is] where we can find data that we can aggregate correctly.

    BL: Right. Are you looking specifically for data that’s county level as opposed to state level?

    JZ: Hopefully… It depends. I was pretty surprised by the lack of quality in, for example, COPD and diabetes data, where like, if you look at [the dataset], like it’s state level—but in most states, there’s not a statistical significance for most races.

    BL: Wow.

    JZ: For example, we use the BRFSS survey. [The Behavioral Risk Factor Surveillance System.] It’s a CDC survey. And as far as we can tell, it’s the gold standard for diabetes [data] in the country.

    And if you look at, say, diabetes, for most states… There’s only, like, four states where Asian people are statistically significant in the survey to make any sort of guess about how many people have diabetes, which is pretty atrocious. But that [data source] is the best we could do, you know. Ideally, we would like to find places that do go down to the county level, but it’s hard.

    For as paltry as the COVID data is, it’s much better than—as far as I’ve seen, like, the fact that there’s like a line-by-line database that the CDC provides, that you can really make all these breakdowns of, is a huge step ahead [compared to other health data]. I’m not like a data expert on this kind of stuff, I’ve just been working on this project for two and a half months. But as far as I’ve seen, that’s what the situation is.

    BL: Yeah, I mean, that kind of lines up with what I have seen as well. And I bet a lot of it is a case where, like, a journalist could FOIA [the data] from a county or from a state. But that’s not the same as getting something that is comprehensive, line-by-line, from the CDC.

    JZ: And we [the Satcher institute] don’t want to be a data collection agency, like the COVID Tracking Project or the New York Times is. I mean, we want this to be a sustainable project. And the COVID Tracking Project was not a sustainable project.

    BL: Yeah, totally. I was there doing the [data entry] shifts twice a week, that’s not something we could have done forever.

    JZ: Yeah, I was there, too. I always think, like, the COVID Tracking Project could only exist when there’s an army of unemployed people who are too afraid to leave their house.

    BL: And volunteers who were like, yeah, sure, I’ll do this on my evenings and weekends.

    JZ: Who, you know, you don’t want to leave, you’re too afraid to go, like talk to people. You want to stay home in front of your computer all day, and feel useful.

    I’m sure you could find all the diabetes data by going to county and state health department websites, but it’s too much work. So we really want everything to come from federal sources, basically, that’s our goal.

    BL: How are you finding that people have used the tracker so far? Like, do you know of any research projects that folks are doing?

    JZ: We released it a couple weeks ago, and we haven’t really heard of any yet… But we hope people are looking at it. And we have a couple of meetings lined up with some interesting research groups and stuff like that. So hopefully, they’ll like it.

    BL: Are there any specific statistics or comparisons or anything else you found in working on it that you would want to see explored further? Are there any stories that you want to see come out of it?

    JZ: The high rates of unknown data in a lot of places, that really needs to be looked into. Because it’s just hard to make any conclusions about what’s going on if—I mean, in some states like New York, over 50% of cases are unknown. That’s a huge problem. And that’s definitely something that needs to be looked into, like, why that’s happening. And if there’s anything that can be done to change that [unknown rate.] The reason why I do think that it can get better is because the COVID Tracking Project racial data had higher completeness rates. And so they [the states] probably do know the races of people who got sick, but they’re just not reporting it for whatever reason.

    And for me, something that’s really stuck out was the extremely high rates of COVID for Hispanic and Latino people, especially in California. If you look at them and compare them to white rates, it’s, like, the exact opposite pattern. So it kind of does look like Hispanic and Latino people were kind-of shielding white people from getting COVID, if you compare the numbers. That’s something I would look into, too, like, why that happened.

    (Editor’s note: This story from The Mercury News goes into how the Bay Area’s COVID-19 response heightened disparities for the region’s Hispanic/Latino population.)

    BL: And another question along the same lines, is there a specific function or aspect of the tracker that you would encourage people to check out?

    JZ: The unknowns. Just, like, look into your county and see what percentage of cases in your county have reported race and ethnicity at all. I think you can really see how good of a job your county has done at reporting that data. I know I was kind-of shocked by that rate for the county like I grew up in, like, I know that they have the resources to [report more data], but they’re just not doing a very good job.

    BL: How would you say this experience with tracking COVID cases might impact the world of public health data going forward, specifically health equity data, and how do you see the tracker project playing a role in that?

    JZ: We really want this project to show the importance of tracking racial health data down to the county level or even lower than that. County is the best we can do right now, but we’d love to see city level or something like that. And again, I kind-of said this before—as much as was missing for the COVID data, it’s still better than the data that there is for most other diseases and other determinants of health. So we would like to see, like, more things able to be filled out on the tracker. We would like to be able to get more granular on more different determinants of health, so that we can see, for example, how poverty impacts health, or a lack of health insurance, or how diabetes and COVID are related down to the county level. You can’t really do that right now… 

    We want people to see that, A, there’s a lot of data missing. But B, even with the data that we have, we can see that there’s like a huge problem. And so we would like to be able to fill out the data more to really get a better picture of what’s going on. If we can see there’s a problem, we can make better policy to help and make these disparities not as stark.

    Support the COVID Data Dispatch
  • The short-term future of COVID-19 testing

    The short-term future of COVID-19 testing

    Antigen test kit image via Dronepicr // Wikimedia Commons.

    This week, I had a story on COVID-19 testing published in Slate’s Future Tense vertical. The piece explores how testing will change in the next few months as more Americans become vaccinated and rapid tests become more widely available, with a practical focus: how should you interpret the test numbers on your local COVID-19 dashboard?

    Overall, I found, we will need to keep getting tested even post-vaccination. But the purpose of testing may shift, for many of us in the U.S., from diagnostic testing—a test to figure out if you are currently sick with COVID-19—to more screening and surveillance testing—tests to identify case trends and stomp out outbreaks in a broader community. This shift may be aided by the rise of rapid, at-home tests, which are becoming much more widely available thanks to investment from the federal government. Just this week, USA Today reported that at-home tests will soon be sold at national pharmacies CVS, Walgreens, and Walmart.

    In the CDD today, I’m excited to share one of the interviews I conducted for the piece, with Dan Larremore, a statistician at the University of Colorado and long-time advocate for the potential of rapid tests. We talked on April 2, just days after two major developments in the testing space: the FDA gave Emergency Use Authorization to several rapid tests for over-the-counter use, and the CDC and NIH announced a massive study to investigate how well these tests work for population-level screening. (One more piece of context: when we talked, case numbers were rising at a more concerning rate than they are now.)

    Larremore and I talked about his reaction to the rapid test news, how to interpret testing numbers, other new test types that may come on the market, and more.

    The interview below has been lightly edited and condensed for clarity.

    Betsy Ladyzhets: First question is, since a lot of this piece is meant to be about the numbers of testing, what would you consider the most useful metric or metrics right now to how successful testing is, whether that’s test positivity or other things?

    Dan Larremore: I think about testing for three different reasons. One reason is information about the trajectory of the pandemic, which is things like test positivity rates, number of new cases. We test to kind-of get our bearings in the movement of the pandemic. The second reason that we test is more at the individual level, but it’s still that [same] information. And that is, I would test because I want to know, am I sick? Might I give the disease to somebody else? Or, can my current symptoms be explained by being infected? So the first two are informational.

    The third reason that we might test is specifically just to break transmission chains, which is more like testing as an answer, not as a question. And so, for at-home testing, for serial testing, for the regular kind of testing that we have to do to be on campus here [at the University of Colorado]… To me, that’s much more about an intervention to slow down transmission than it is about gaining information.

    So, that’s a roundabout way of answering your question. But I think, in terms of what numbers to look at, it really matters what the intention of the testing is. So if people continue to take the pandemic seriously, and continue to, like, get tested regularly, or get tested when they feel sick, then those sort of daily case numbers will remain useful and interesting. And therefore the test positivity rate information will continue to tell us something about the trajectory of the pandemic. Does that answer your question?

    BL: Yeah, I think that does answer my question. Because I think that was one of the big kind of questions I had going into this story is like, is test positivity still useful if maybe, as people get vaccinated, they stopped thinking they need to get tested or as other dynamics change. But yeah, I had another person who I’ve interviewed for this story also had kind of a hierarchy description [of testing], so that definitely is a useful thing to think about.

    DL: Knowing why people come in to get tested just gives you so much more interpretability of like, what the numbers mean. Here on campus, if I want to be here on campus, I need to get tested weekly. I’ve been vaccinated, one dose, dose number two soon. But nevertheless, I still need to spit in the tube every week, and they test it. So, the test positivity rates here on campus are minuscule. Because with high compliance, everybody gets tested, so the denominator in that positivity rate is huge. At a drive up site, or at a doctor’s office, where people are coming in because they feel sick, the test positivity rates are going to be a lot higher.

    And I think, as people get more and more vaccinated—regardless of the case counts—as the pandemic feels like it’s winding down, I think people may be less likely to get tested. And so you can imagine test positivity rate being driven by, not just what the virus is doing, but a lot of the human behavior as well… I guess, the way that you can put it is, that you can see changes in the test positivity that are driven entirely by human behavior with respect to getting tested at all. And not so much about changes in the virus.

    BL: That makes sense. And I think it’s also about access, right? Are people able to go and get tested if they feel sick? Or if you’re thinking about schools and workplaces, is their employer having everyone get tested once a week? Is that something we’re going to see more of now that we have—like, literally earlier this week, the FDA gave EUA to a bunch of antigen tests, right?

    DL: Yeah, for at-home use.

    BL: Right. Do you think we’re gonna finally see that massive use of rapid tests that experts have wanted to see since, like, the summer?

    DL: A lot of people, myself included, have been excited about the possibility of at-home repeated antigen testing, as a way to really take community transmission levels and push them down. Because we know that asymptomatic transmission occurs, we know that getting people results rapidly is absolutely critical. Because four-day-old information is totally useless if you have infected people in those four days.

    I don’t know how useful those test kits are going to be right away, like, right now, given that we just now have an EUA for that kind of at home use that we’ve been hoping for for a long time. But at the same time, cases are shooting up due to these new variants around the U.S. and more importantly, around the world. So, I think these tools are still going to be useful, especially this fall, as we get a lot more kids in school. And we start bringing people together, temperatures starting to cool off, more people are indoors. I think that having the rapid test as a screening tool will still be valuable, particularly if we see limited uptake of the vaccine.

    BL: That makes sense… Another question around rapid tests is, that I know in the U.S., it’s really hard to get comprehensive data on them. I volunteered at the COVID Tracking Project, and I wrote [a blog post] about the problem of antigen test data. So I was curious as to how we will know how well the tests are working. And if there are any specific studies that you’re watching or data sources that you recommend, in terms of, like, knowing if people are actually using these at home tests.

    DL: I am excited about… On the 30th, there was a press release about the NIH and CDC rolling out at-home testing to two cities.

    BL: I saw that, yeah!

    DL: The work that Michael Mina and I did last year was showing that, at the individual level, the trade off between test sensitivity and turnaround time, should really tell us that turnaround time is critical. Like, theoretically, at the individual person level, the rapid test idea is really, really good. What we’ve not yet seen—outside of Slovakia—is the ability to flood the zone with tests, as Fauci put it, and just catch as many positives as possible and drive the epi curve downward, just because of the rapid tests.

    My feeling is that, really, the proof is in the pudding. If people can, at a community level, use a boatload of rapid tests regularly for a few weeks, and we can watch the new cases spike as we find those positives and then crash as we break all the transmission chains. That’s the key thing. That’s the key thing that I’m looking out for in these new trials.

    The Slovakia folks showed some of the limitations of this [strategy]. There’s a paper in Science where they wrote up their results. And basically what they found was the rapid testing worked really well, but the problem was on the isolation side. [Not everyone who tests positive can truly and effectively isolate.] In the short term, while they still had the supplies, these three waves of everybody in the country getting tested, worked like they were supposed to work, worked like the theory said. However, once you stop testing, you take your foot off the brakes, things re-accelerate. The second thing is that a lot of folks in particularly rural areas were like, okay, well, I’m positive, but I live with my family. How am I going to isolate? So unlike on a college campus here, where when somebody tests positive, we have a separate dorm set up for them for two weeks—in real life, that isolation stuff is going to be harder.

    BL: So it’s kind-of like, you need to pair it with the social services aspect, or some other way to help people out. I know, in New York, there’s a hotel room program, where if you test positive, you can contact the City Department of Health, and they’ll like, put you up in a hotel for two weeks. I don’t know how much it gets used, but it definitely seems like something that should be around in more places.

    DL: I mean, even if it’s just supporting people, by telling them like, this is gonna be awkward, but wear your mask at home, and don’t hang out with your family. Go watch TV in the basement, or, you know, otherwise keep distance from people. Whatever these interventions are, they can help. And we know that what we call the secondary attack rate is actually not that high. So, even among people who live in the same household, even among spouses, if one person is sick, that doesn’t mean that the other person definitely gets it. It’s only [around 20%] chance that they do. We looked at pairs of roommates here on campus, and studied [transmission between] them. And even among roommates, the secondary attack rate is not that high [20-28%].

    BL: Well, that brings me to another question I wanted to ask you, which is how public health communication around testing either is changing or should be changing in order to express like, okay, maybe you’ve been vaccinated, but you still need to get tested, or what needs to be communicated about these at home tests, or any other messages that you think are important to be conveying.

    DL: One thing that’s important is that we know that we need to keep our eye on the variants. And evidence is emerging… that there are some breakthrough variants, they are less well handled by the vaccine, even though the vaccine works really well… So, testing is going to remain important, even as, more broadly, the vaccine protects people from the most severe disease.

    We’re seeing a really interesting split right now, right? Where like, cases are going up, and we expect hospitalizations to then go up, and then mortality to go up. But I wonder if we’re gonna see that in the U.S. this time. Like, week on week, mortality continues to go down. And the question is, are we gonna get it again?

    BL: Is it gonna go up again?

    DL: Or did we vaccinate enough of the high-risk people that the mortality stays flat even while cases go up?

    BL: I definitely think there’s going to be kind-of a demographic aspect of it. Like,I’ve seen charts where people do, with the HHS hospitalization data, they publish it by age. So, you can see that hospitalizations are going way up in people ages, like, 18 to 30, but not so much in seniors. Although, kind-of tangentially, one data gap that I get annoyed by is that there’s very little demographic data for testing. Like, if you look at race data, for example, there are maybe five states that publish testing data by race and ethnicity. And there’s not a lot of it by age. So it’s kind-of hard to track patterns there.

    DL: I mean, I don’t know what the right messaging is around testing, other than, [if you feel like you’re sick], if you have the symptoms, you’ve got to go get tested. It doesn’t matter if you’re vaccinated or not. If you feel sick, you should go get tested. If you are going to be around somebody who you know is really vulnerable, if testing is available, go get tested.

    I still think it’s a valuable intervention, especially in places where vaccine uptake is low or vaccine availability is low. But I think the question is, like, really, how long is—what’s the expiration date on recommendations about testing? If 70% of the US is vaccinated by July, let’s say, does it still make sense to recommend a huge amount of testing? And I don’t know the answer to that. What I would like to see is people equipped for this fall, if there are spikes in cases, or if there are variants that are circulating even among vaccinated people, it would be amazing if, like, a local public health authority could tell everybody, this Sunday, I want everybody to use your rapid at-home test. Report your results anonymously to this number. And, if you test positive, take it seriously, take precautions. Measures like that could preserve privacy, while still collecting that key surveillance data and crashing the epidemic curve.

    BL: Right, that would be really cool to see. Are there any other types of COVID tests or surveillance methods that you think might become more useful and more prevalent in the next few months or heading into the fall?

    DL: Yeah, there are two kinds. So, one key point about at-home tests is that privacy-preserving aspect. Like, I trust local public health here. But I tend to vote left of center and generally trust the government. I live in Colorado, and not everybody feels that way. There’s definitely a strong libertarian independent streak. And I feel like one of the key advantages of at-home tests is that they appeal to that kind of person. They empower a person and their family to make health decisions, and they give you the information. But they don’t necessarily get recorded by something like COVID Tracking Project or HHS.

    BL: Yeah.

    DL: There’s less visibility for authorities, but for some folks who want information that they can act on to protect themselves and others, then that’s going to be fine. And so, as much as I would like to know exactly what is happening with the pandemic, if the trade-off for lower cases is that we don’t know about a lot of cases among folks who would rather not report their data, I think that’s a fair trade-off.

    BL: Yeah, that makes sense.

    DL: But it’s sort of like uncomfortable for me, who likes the data to say that. You know what I mean?

    BL: It reminds me of the conversation around exposure notification apps. I talked to someone who works on those apps recently, and she kind-of said the same thing, that she would rather have everyone using the app than really good data from a tiny subset of the population that’s okay with their privacy being violated.

    DL: A big thing to me is that, we know that the pandemic has been political. And I don’t see any reason why we can’t have solutions that work for the person who votes left of center, the person who votes in the middle, and the person who votes right of center.

    But the other kind of tests that I’m excited about, only because they’re extremely cheap and really easy, are anosmia screens. We know that loss of sense of smell is highly specific to COVID. If you don’t have a stuffy nose, and you [suddenly] can’t smell things, you probably have COVID. So, there are companies that produce, like, a little card with a scratch-and-sniff quiz. You don’t know what’s behind the panel, but you scratch, pull up a smartphone app, and then say what you think the smell is from a multiple choice test.

    One of the cool things is that anosmia only occurs in around 40% of people [with COVID] if you ask them to self-report. But if you give them one of these objective quizzes, the prevalence of anosmia as a symptom goes up to [around 75 or 80%], depending on which study you look at. The important thing is that those cards cost 25 cents apiece, and multiple people can use the same card. It’s literally a scratch-and-sniff with an online quiz.

    BL: That’s incredible. Do you know if there are tests like that that are up for EUA?

    DL: Yeah. Roy Parker, Michael Mina, and I collaborated with a great team to write this paper last year on typical COVID testing [PCR, LAMP, and antigen testing], test sensitivity and frequency and turnaround time. Then, Roy and I teamed up with Derek Toomre at Yale School of Medicine, and took the same idea and said, well, what if we use frequent, repeated anosmia screening tests?

    One of the things that I like about those is that they’re cheap. But another thing that I like about them is that nobody thinks that [a smell test is] the same thing as a proper COVID test. You can’t get them confused. [This is important because one of the questions with rapid tests is how people may interpret a negative test—they might be infected, but the virus is at a low level. So if they get a negative rapid test and then go to the gym, the test could actually have an unintended effect. But if an anosmia test tells you that you still have your sense of smell… People understand that just because you still have your sense of smell doesn’t mean that you’re COVID-free. There’s lower risk of unintended consequences.]

    Anyway, I think the messaging around those [anosmia] tests is easy. They’re easy to use, you can do them at home. And they can print them for, like, a quarter apiece. So the modeling suggests that they could be pretty effective and really cheap. You could literally mail somebody a stack of 10 of these things to everybody in the U.S.

    BL: And do it once a week!

    DL: Yeah, do it once a week. So, that’s another kind of test that I would like to see out there. The company that Derek started, that makes those [tests], just won one of the XPRIZEs for COVID testing. So, I think that’s cool. It’s a more creative kind of test, and it’s inexpensive.

    BL: Sweet. So, that was all the questions that I had. Is there anything you think is important on this topic that I should know for this article?

    DL: I just think it’s really important to keep a global perspective… As with vaccines, we know there are inequities within the U.S. But there are definitely global inequities. And while we might feel like we’re on a glide path to herd immunity through vaccination here in the States, things look very different in the rest of the world. And so, the use of these tests may become more important this fall, we may get more variants globally, even as the U.S. cases go down. I think it’s an ongoing story, even if hospitalizations and deaths continue to drop here in the States.

    BL: And those tests you’re talking about that are cheaper and easier to use are useful in many places, not just here.

    DL: Yeah, that’s the hope… I feel generally optimistic about [the state of the pandemic], but like, hesitant.

    BL: I feel you. It’s definitely weird to see everyone getting very excited about the summer when I’m kind-of sitting here in my COVID reporting bubble, like, ahhh, not there yet.

    DL: Well, one of the hardest parts, I think, for public health officials is going to be, if cases are going up, but mortality and hospitalizations are flat or going down. If COVID is only making people sick, but it’s not hospitalizing and killing them. Then, like, do we just reopen everything? Do we open up the schools? That’s tough when we’ve been acclimated to keep our foot on the brakes as much as possible.

  • Privacy-first from the start: The backstory behind your exposure notification app

    Privacy-first from the start: The backstory behind your exposure notification app

    New Jersey reports data on how people are using the state exposure notification app, COVID Alert NJ. Screenshot taken on March 28.

    Since last fall, I’ve been fascinated by exposure notification apps. These phone applications use Bluetooth to track people’s close contacts and inform them when a contact has tested positive for COVID-19. As I wrote back in October, though, data on the apps are few and far between, leaving me with a lot of questions about how many people actually have these apps on their phones—and how well they’re working at preventing COVID-19 spread.

    This week, I put those questions to Jenny Wanger, co-founder of the TCN Coalition and Director of Programs at the Linux Foundation of Public Health. TCN stands for Temporary Contact Numbers, a privacy-first contact tracing protocol developed by an international group of developers and public health experts. As a product manager, Wanger was instrumental in initial collaboration between developers in the U.S. and Europe, and now helps more U.S. states and countries bring exposure notification apps to their populations.

    Wanger originally joined the team as what she thought would be a two-week break between her pandemic-driven layoff and a search for new jobs. Now, as the TCN Coalition approaches its one-year anniversary, exposure notification apps are live on 150 million phones worldwide. While data are still scarce for the U.S., research from other countries has shown how effective these apps may be in stopping transmission.

    My conversation with Wanger ranged from the privacy-first design of these apps, to how some countries encouraged their use, to how this project has differed from other apps she’s worked on.

    The interview below has been lightly edited and condensed for clarity.

    Betsy Ladyzhets: To start off, could you give me some background on how you got involved with the TCN colatition and what led you to this role you’re in now?

    Jenny Wanger: My previous company did a very large round of layoffs with the beginning of the pandemic because the economics changed quite dramatically, and I was caught in that crossfire. And a couple of days later, a friend reached out and asked whether I was available to help—he was like, “I need a product manager for this thing, we’re trying to launch these apps for the pandemic. It should be, like, two weeks, and then you can go back to whatever.” So I signed up for that. I thought, sure, I’m not gonna be getting a job in the next two weeks. 

    A lot of what we were trying to do, the person who brought me on, was to convince people to use the same system and be interoperable with each other, to have more collaboration across projects. As opposed to all of these different apps being built, none of which would be able to work with each other. We found that there was somebody doing the same thing over on the European side, which was Andreas [Gebhard].

    We scheduled a meeting with all of the people we were trying to convince to do something interoperable and all of their people, and out of that meeting came the TCN Coalition. Andreas suggested the name TCN Coalition pretty much on a whim, which we’ve learned, never try to name a project in a meeting with other people there, because it will haunt you for a long time.

    That’s what we ended up with… TCN Coalition was formed, and we started trying to get everybody to build an interoperable standard and protocol and share that kind-of thing together. It was probably a week or two later that Apple and Google announced that they were going to be having APIs available to use. We weren’t totally sure what to do with that, so we kept moving forward, waiting for more information from them, and then also coaching everybody, like let’s make this interoperable with Apple and Google, that fixes a lot of problems that we weren’t able to fix otherwise.

    We kept growing, we started building out some relationships with public health authorities. And meanwhile, somebody started poking around in our area from the Linux Foundation… Eventually, it became clear that we were not gonna be able to grow to the degree that we wanted without a business model, and Linux Foundation brought that piece of the puzzle. So we merged our community to seed the Linux Foundation Public Health, and Linux Foundation Public Health brought in a business model and some funding that allowed us to keep doing the work that we were doing. We were also getting to the point where a bunch of our volunteers were saying that they needed to go back to having jobs… There was a lot of early momentum, and that slowed down over time, understandably.

    So yeah, that’s how TCN ended up merging in with LFPH. That man who was poking around TCN way back at the beginning was a guy—his name is Dan Kohn, he unfortunately passed away from cancer at the beginning of November. With that, I ended up taking on more of a leadership role in LFPH than I’d anticipated. We eventually got a new executive director at this point, and I’ve been part of the leadership team throughout. That’s sort-of the high level story.

    BL: Thank you. So, how did your background—you do product management stuff, right, how did that lead into connecting coders and running this coalition?

    JW: As a product manager, I’ve always been focused on how to get something built that actually meets the needs of a certain population, and is actually useful. There’s two sides to that. One is the project management side, of like, okay, we need to get this done.

    But much more relevant has been, on the product side, we need to make sure that we’re building things that—there are so many different players in the space, with an exposure notification app or now as we’re looking at vaccine credentials. You’ve got the public health authority, who is trying to achieve public health goals. You’ve got the end user, who actually is going to have this product running on their phone. You have Apple and Google, or anybody else who is controlling the app stores, that have their own needs. You’ve got the companies that are actually building these tools out, building out these products who are trying to hit their own goals. It’s a lot of different players, and I think where my background as a product manager has really helped has been, I’ve got frameworks and tools of how to balance all these different needs, figure out how to move things forward and get people working together, get them on the same page, to actually have something go to market that does what we think it’s supposed to do.

    BL: Right. To talk about the product itself now, can you explain how an exposure notification app works? Like, how would you explain it to someone who’s not very tech savvy.

    JW: The way I explain exposure notification is essentially that your phone uses Bluetooth to detect whether other phones are nearby. They do this by broadcasting random numbers, and the other phones listen for these random numbers and write them down in a database.

    That’s really all that’s happening—your phone shouts out random numbers, they’re random so that they don’t track you in any way, shape, or form, they’re privacy-preserving. You’ve got that cryptographic security to it. The other phones write down the numbers, and they can’t even tell, when they get two numbers, whether they’re from the same phone or different phones. They just know, okay, if I received a number, if I wrote it down, that means I was close enough to that phone in order to be at a distance, being at risk of COVID exposure.

    Then, let’s say one of those phones that you were near, the owner of that phone tests positive. They report to a central database, “Hey, I tested positive.” When this happens, all of the random numbers that that phone was broadcasting get uploaded to a central server. And what all the other phones do is, they take a look at the list on the central server of positive numbers, and they compare it to the list that’s local on their phone. If there’s a match, they look to see, like, “How long was I in the vicinity of this phone? Was it for two minutes, five minutes, 30 minutes?”

    If it goes over the threshold of being near somebody who tested positive for enough time that you’re considered a close contact, then you get a notification on your phone saying, “Hey, you were exposed to COVID-19, please follow these next steps.”

    The nice thing about this system is, it’s totally privacy-preserving, there’s pretty much no way for anybody to look at these random numbers and tell who’s tested positive or who hasn’t. They can’t tell who anybody else has been by. So it’s a really privacy-first system.

    And what we’re now seeing, which is really exciting, is that it’s effective. There’s a great study that just came out of the U.K. about a month ago, showing that for every additional one percent of the population that downloaded the NHS’s COVID-19 app, they saw a reduction in cases of somewhere between 0.8 and 2.3 percent.

    BL: Oh, wow.

    JW: The more people that adopt the app, it actually has had a material impact on their COVID-19 cases. The estimates overall are as many as 600,000 cases were averted in the U.K. because of this app.

    Editor’s note: The study, by researchers at the Alan Turing Institute, was submitted for peer review in February 2021. Read more about the research here.

    BL: That goes into something else I was going to ask you, which is this kind-of interesting dynamic between all the code behind the apps being open source, that being very public and accessible, as opposed to the data itself being very anonymized and private—it’s this tradeoff between the public health needs, of we want to use the app and know how well it’s working, versus the privacy concerns.

    JW: The decision was made from the beginning, since the models showed higher levels of adoption of these apps was going to be critical in order for them to be successful. The more people you could get opting into it, the better. Because of that, the decision was made to try and design for the lowest common denominator, as it were. To make sure that you’re designing these apps to be as acceptable to as many people as possible, to be as unobjectionable as possible in order to maximize adoption.

    With all of that came the privacy-first design. Yes, a lot of people don’t care about the privacy issues, but we were seeing that enough people cared about it that, if we were to launch something that compromised somebody’s privacy, we were going to see blowback in the media and we were going to see all sorts of other issues that tanked the success of the product.

    Yes, it would be nice to get as much useful information to public health authorities as possible, but the goal of this was not to supplant contact tracing, but to supplement it. The public health authorities were going to be getting most of the data that we were able to provide via they know who’s tested positive. They’re already getting contact tracing interviews with them. It wasn’t clear what we could deliver to the public health authority system that wasn’t already being gathered some other way.

    There could’ve been something [another version of the app] where it gave the exposure information, like who you’ve been with, to the public health authority, and allowed them to go and contact those people before the case investigations did. But there were so many additional complications to that beyond just the privacy ones, and that wasn’t what—we weren’t hearing that from the public health authorities. That wasn’t what they needed. They were trying to figure out ways to get people to change behavior.

    We really pressed forward with this as a behavior change tool, and to get people into the contact tracing system. We never wanted it to replace the contact tracing that the public health authorities were already spinning up.

    BL: I suppose a counter-argument to that, almost, is that in the U.S., contact tracing has been so bad. You have districts that aren’t able to hire the people they need, or you have people who are so concerned with their privacy that they won’t answer the phone from a government official, or what-have-you. Have you seen places where this system is operating in place of contact tracing? Or are there significant differences in how it works in U.S. states as opposed to in the U.K., where their public health system is more standardized.

    JW: Obviously, none of us foresaw the degree to which contact tracing was going to be a challenge in the U.S. I think, though, it’s very hard—the degree to which we would’ve had to compromise privacy in order to supplant contact tracing would have been enormous. It’s not like, oh, we could loosen things just a little bit and then it would be a completely useful system. It would have to have been a completely centralized, surveillance-driven system that gave up people’s social graphs to government agencies.

    We weren’t designing this, at any point in time, to be exclusively a U.S. program. The goal was to be a global program that any government could use in order to supplement their contact tracing system. And so we didn’t want to build anything that would advance the agenda—we had to think about bad actors from the very beginning. There are plenty of people just in the U.S. who would use these data in a negative way, and we didn’t want to open that can of worms. And if you look at more authoritarian or repressive governments, we didn’t want to allow them a system that we would regret having launched later.

    BL: Yeah. Have you seen differences in how European countries have been using it, as compared to the U.S.?

    JW: There have been some ways in which it’s been different, which has more to do with attitudes of the citizenry than with government use of the app itself. The NHS [in the U.K.] has a more unique approach.

    The U.K. and New Zealand both ended up building out a QR code check-in system, where if you go to a restaurant or a bar… You have a choice, either you write your name and phone number in a ledger that the venue keeps at their front door. So if there’s an outbreak later, they can call you, reach out and do the case investigation. Or you scan a QR code on your phone that allows you to check into that location and figure out where you’re moving. If there’s an alert [of an outbreak] there, you get a notification saying, you were somewhere that saw an outbreak, here’s your next steps.

    One of the big advantages of the U.K. choosing to do that is essentially that—every business had to print out a QR code to post at their front door. Something like 800,000 businesses across England and Wales printed out these QR codes. And that means anyone who walks into one of those venues gets an advertisement for their app, every single time they go out. It was very effective in getting good adoption.

    We’ve also seen a very big difference in how different populations think about the app and use it. For instance, Finland has had very good compliance with their app. What we mean by that is, if you test positive and you get a code that you need to upload, in Finland, there’s a very high likelihood that you actually go through that process in your exposure notification app. That’s something that I think a lot of jurisdictions have been struggling with in the U.S. and other countries—once you get the code, making sure that somebody actually uploads it.

    It makes sense, because getting a positive diagnosis for COVID is a very stressful thing. It’s a very intense moment in your life. And you might not be thinking immediately, “Oh, I should open my app and upload my code!”

    BL: Right, that’s not the first thing you think of… This relates to another question I have, which is how you’ve seen either U.S. states or other countries adapting the technology for their needs. You talked about the U.K. and New Zealand, but I’m wondering if there are other examples of specific location changes that have been made.

    JW: There have been some mild differences. Like, this app will allow you to see data about how each county is performing in your jurisdiction, so you can also go there to get your COVID dashboard. I’ve seen some apps where, if you get a positive exposure notification, that jumps you to the front of a line for a test. You can schedule a test in the app and you can get a free test as opposed to having to pay for it.

    I’ve seen things like that, but overall, at least with the Google/Apple exposure notification system, it’s been small changes to that degree. Where you see more dramatic changes is where countries have built their own system. You can look at something like Singapore, where people who don’t have phones get a dongle that they can use to participate in the system. It’s entirely centralized, and so they are able to do things like, a lot of contact tracing actually from the information they get with the app. There are places where it’s more aggressive in that sense.

    For the most part, though, I’d say it’s been pretty consistent… The one-year anniversary of the TCN Coalition isn’t until April 5, but if you think about how far we’ve come from this just being an idea in a couple of people’s heads to, last I heard, the GAIN [Google/Apple] exposure notification apps are on 150 million phones worldwide.

    BW: Wow! Is that data publicly available, on say, how many people in a certain country have downloaded apps?  I know, one state that I’ve found is publishing their data is New Jersey, they have a contact tracing pane on their dashboard. I was curious if you’d seen that, if you have any thoughts on it, or if there are any other states or countries that are doing something similar.

    JW: I wish there was more transparency. Switzerland has a great dashboard on the downloads and utilization of their app. DC, Washington state, also publicly track their downloads. I’m sure a few others do but I don’t know off the top of my head who makes the data public.

    I do wish it were the default for everybody to make that data public… There’s a lot of concern by states where there’s not good adoption, that by making the data public they’re opening up a can of worms and are going to get negative press and attention for it, so they don’t want to. So it’s been a mix in that way.

    BL: I think part of that is also an equity concern. How do you know that you have a good distribution of the population that’s adopting it, or even that the people who need these apps the most, say essential workers, people of color, low-income communities—how do you know that they’re adopting it when it’s all anonymous?

    JW: It’s actually—if you’re going to have low adoption, what’s much more effective is if you have high adoption in a certain community. There is a health equity question, but it’s not necessarily about equal distribution of the app, but rather—and this is where some states have been successful, is that they haven’t gotten high adoption across the board but they’ve decided on a couple of high-need communities that are the ones they’re going to target for getting adoption of the app. They’ve gone after those instead, and that, for many of the states, has been a more effective way to drive use.

    BL: I live in New York City, and I know I’ve seen ads for the New York one, like, in the subways and that sort of thing, which I have appreciated.

    Is there a specific state or country that you’d consider a particularly successful example of using these apps?

    JW: NHS, England and Wales, definitely. I think Ireland has done a pretty good job of it, and Ireland is—we’re particularly fond of them because they were one of the first to open source their code, and make it available. They open-sourced with LFPH to make it available for other countries, and so that is the code that powers the New York app as well. New York, New Jersey, Pennsylvania, Delaware, and then a couple of other countries globally, including New Zealand. It’s the most used code, besides the exposure notification express system that Apple and Google built for getting these apps out.

    I also mentioned Finland before, I think they got the messaging right such that they have very high buy-in on their app.

    BL: Are you collecting user feedback, or do you know if various states and countries are doing this, in order to improve the apps as they go?

    JW: Usually as a product manager, you’re constantly wanting to improve the UI [user experience] of your app, getting people to open it, and all that. These are interesting apps in that they’re pretty passive. Your only goal is to get people not to delete them. They can run in the background for all of eternity. As long as the phone is on and active, that’s all that’s needed.

    BL: As long as you have your Bluetooth turned on, right?

    JW: As long as you have your Bluetooth turned on. So the standard for the success of these apps is a completely different beast. We at LFPH have not been monitoring the user feedback on this, but a lot of states and countries are. Most of them have call centers to deal with questions about the app.

    Some jurisdictions are improving it, but most improvements are focused on the risk score, which is the settings about how sensitive the app should be.

    BL: Like how far apart you need to be standing, or for how long?

    JW: Right. How to translate the Bluetooth signal into an estimate of distance, and how likely should it be—how willing are you to send an alert to somebody, telling them that they’ve been exposed, based on your level of confidence about whether they actually were near somebody or not. There’s a decent amount of variance there in terms of how a state thinks about that, but that’s been much more on the technical side, where people are trying to tweak the system, than on the actual app. There have been some language updates to clarify things, to make it easier for people to know what to do next, but it’s not been the core focus of the app designs like it would be if this were a more traditional system.

    BL: What does your day-to-day job actually look like, coordinating all of these different systems?

    JW: We’re [LFPH/TCN] really an advisor to the jurisdictions. It’s not a coordinating thing but rather, I spend a lot of my time on calls with various states saying, “Here’s what’s happening with the app over in this place, here’s what this person is doing, have you considered this, do you want to talk to that person.” I’m trying to connect people, trying to provide education about how these systems work, and for the states that are still trying to figure out whether to launch or not, convincing them to do it and sharing best practices.

    Also, with Linux Foundation Public Health, we’re working on a vaccination credentials project. So I’m splitting my time between those, as well as just running the organization and keeping financials, board relationships, networking, fundraising, keeping all of those things together.

    BL: Sounds like a lot of meetings.

    JW: It’s a fair number of meetings, this is true.

    BL: So, that’s everything I wanted to ask you. Is there anything else you’d like folks to know about the system?

    JW: Ultimately, the verdict is, now that we’re seeing it’s effective [from the U.K. study], I think that adds to the impetus to download and use the system. Even before that, though, the verdict was—this is extraordinarily privacy-preserving, there’s no reason not to do it. That continues to be our message. There’s no harm in having this on your phone, it doesn’t take up much battery life, so turn it on!

  • How one biostatistics team modeled COVID-19 on campus

    How one biostatistics team modeled COVID-19 on campus

    Screenshot of a modeling dashboard Goyal worked on, aimed at showing UC San Diego students the impact of different testing procedures and safety compliance.

    When the University of California at San Diego started planning out their campus reopening strategy last spring, a research team at the school enlisted Ravi Goyal to help determine the most crucial mitigation measures. Goyal is a statistician at the policy research organization Mathematica (no, not the software system). I spoke to Goyal this week about the challenges of modeling COVID-19, the patterns he saw at UC San Diego, and how this pandemic may impact the future of infectious disease modeling.

    Several of the questions I asked Goyal were informed by my Science News feature discussing COVID-19 on campus. Last month, I published one of my interviews from that feature: a conversation with Pardis Sabeti, a computational geneticist who worked on COVID-19 mitigation strategies for higher education. If you missed that piece, you can find it here.

    In our interview, Goyal focused on the uncertainty inherent in pandemic modeling. Unlike his previous work modeling HIV outbreaks, he says, he found COVID-19 patterns incredibly difficult to predict because we have so little historical data on the virus—and what data we do have are deeply flawed. (For context on those data problems, read Rob Meyer and Alexis Madrigal in The Atlantic.)

    Paradoxically, this discussion of uncertainty made me value his work more. I’ve said before that one of the most trustworthy markers of a dataset is a public acknowledgment of the data’s flaws; similarly, one of the most trustworthy markers of a scientific expert is their ability to admit where they don’t know something.

    The interview below has been lightly edited and condensed for clarity.

    Betsy Ladyzhets: I’d love to hear how the partnership happened between the university and Mathematica, and what the background is on putting this model together, and then putting it into practice there.

    Ravi Goyal: Yeah, I can give a little bit of background on the partnership. When I did my PhD, it was actually with Victor De Gruttola [co-author on the paper]. We started using agent-based models back in 2008 to sort of understand and design studies around HIV.  And in particular in Botswana, for the Botswana Combination Prevention Project, which is a large random cluster study in Botswana.

    So we started using these kinds of [models] to understand, what’s the effect of the interventions? How big of a study has to be rolled out to answer epidemiological questions? Because, as you would imagine, HIV programs are very expensive to roll out, and you want to make sure that they answer questions.

    I’ve been working with [De Gruttola] on different kinds of HIV interventions for the last decade, plus. And he has a joint appointment at Harvard University, where I did my studies, and at the University of California in San Diego. And so when the pandemic happened, he thought some of the approaches and some of the stuff that we’ve worked on would be very applicable to helping think about how San Diego can open. He connected me with Natasha Martin, who is also on the paper and who is part of UC San Diego’s Return to Learn program, on coming up with a holistic way of operating procedures there. She’s obviously part of a larger team there, but that’s sort of where the partnership came about.

    BL. Nice. What would you say were the most important conclusions that you brought from that past HIV research into now doing COVID modeling?

    RG: Two things. One is uncertainty. There’s a lot of things that we don’t know. And it’s very hard to get that information when you’re looking at infectious diseases—in HIV, in particular, what was very difficult is getting really good data on contacts. In that setting, it’s going to be sexual contacts. And what I have understood is that people do not love revealing that information. When you do surveys where you get that [sexual contact] information, there’s a lot of biases that creep in, and there’s a lot of missing data.

    Moving that to the COVID context, that is now different. Different kinds of uncertainty. Biases may be recall biases, people don’t always know how many people they have interacted with. We don’t have a good mechanism to sort of understand, how many people do interact in a given day? What does that look like?

    And then, maybe some of these that can creep in when you’re looking at this, is that people may not be completely honest in their different risks. How well are they wearing masks? How well are they adhering to some of those distancing protocols? I think there’s some stigma to adhering or not to adhering. Those are biases that people bring in [to a survey].

    BL: Yeah, that is actually something I was going to ask you about, because I know one of the big challenges with COVID and modeling is that the underlying data are so challenging and can be very unreliable, whether that’s, you know, you’re missing asymptomatic cases or it’s matching up the dates from case numbers to test numbers, or whatever the case may be. They’re just a lot of possible pitfalls there. How did you address that in your work with the University of California?

    RG: At least with the modeling, it makes it a little more difficult in the timeframe that we were modeling and looking at opening, both for our work on K-12 and for UCSD. We kicked it off back in April, and May, thinking about opening in the fall. So, the issue there is, what does it look like in the fall? And we can’t really rely on—like, the university was shut down. There’s not data on who’s contacting who, or how many cases are happening. There were a lot of things that were just completely unknown, we’re living in a little bit of a changing landscape.

    I’m sure other people have much more nuance [on this issue], but I’m going to just broadly stroke where this COVID research was different than HIV. For HIV, people might not radically change the number of partnerships that they’re having. When we’re thinking about a study in Botswana, we can say, what did it look like in terms of incidents four years prior? And make sure we’re making our modeling represents that state of how many infections we think are happening.

    Here [with COVID], when we’re thinking about making decisions in September or October. You don’t have that, like, oh, let’s match it to historical data option because there was no historical data to pin it to. So it was pooling across a lot—getting the estimates to run to the model, getting those is, you’re taking a study from some country X, and then you’re taking another different study from country Y, and trying to get everything to work and then hopefully when things open up, you sort-of re-look at the numbers and then iteratively go, what numbers did I get wrong? Now in the setting where things are open, what did we get wrong and what do we need to tweak?

    BL: I noticed that the opening kind-of happened in stages, starting with people who were already on campus in the spring and then expanding. So, how did you adjust the model as you were going through those different progressions?

    RG: Some assumptions were incorrect in the beginning. For example, how many people were going to live off campus, that was correct. But how many people, of those off-campus people, were ever going to come to campus, was not there. A lot of people decided not to return to San Diego. They were off-campus remote, but they never entered campus. Should they have been part of that model? No. So once we had those numbers, we actually adjusted.

    Just this past week, we’ve sort of started redoing some of the simulations to look towards the next terms. Our past miscalculation or misinformation, what we thought about how many people would be on campus, now we adjusted from looking at the data. 

    And some of the things that we thought were going to be higher risk, at least originally, ended up being a little bit lower risk than anticipated. One thing is around classrooms. There have been—at least, from my understanding, there have been very few transmissions that are classroom-related. And we thought that was going to be a more of a higher transmission environment in the model, wasn’t what we saw when we actually had cases. So now we’re adjusting some of those numbers to get it right to their particular situation. It’s a bit iterative as things unroll.

    BL: Where did you find that most transmissions were happening? If it’s not in the classroom, was it community spread coming into the university?

    RG: They [the university] have a really nice dashboard, where it does give some of those numbers, and a lot of the spread is coming from the community coming on to campus, and less actual transmissions that are happening within. I think that’s where the bulk is. I think the rates on campus were lower than the outside.

    BL: Yeah, that kind-of lines up with what I’ve seen from other schools that I’ve researched that, you know, as much as you might think a college is an environment where a lot of spread’s gonna happen, it also allows for more control, as opposed to just a city where people might be coming in and out.

    Although one thing, another thing I wanted to ask you about, is this idea that colleges, when they’re doing testing or other mitigation methods, they need to be engaging with the community. Like UC Davis, there’s been some press about how they offer testing and quarantine housing for everybody. Not just people who are students and staff. I was wondering if this is something accounted for in your model, and sort of the level of community transmission or the level of community testing that might be tied to the university and how that impacts the progression of things on campus.

    RG: The model does incorporate these infections coming in for this community rate, and that was actually based off of a different model modeling group, which includes Natasha, that is forecasting for the county [around UC San Diego]. Once again, you have to think about all the biases on who gets tested. False positives, all of those kinds of caveats. They built a model around that, which fed into the agent-based modeling that we use. We do this kind-of forecasting on how many infections do we think are going to be coming in from people who live off-campus, or staff, or family—what’s their risk?

    That’s where that kind of information was. In terms of quarantining my understanding is, I don’t think they were quarantining people who weren’t associated [with the school] in the quarantine housing.

    BL: Right. Another thing I wanted to ask about, I noticed one of the results was that the frequency of testing doesn’t make a huge difference in mitigation compared to other strategies as long as you do have some frequency. But I was wondering how the test type plays in. Say, if you’re using PCR tests as opposed to antigen tests or another rapid test. How can that impact the success of the surveillance mechanism?

    RG: Yeah, we looked a little bit in degrading the sensitivity from a PCR test to antigen. The conclusion was that it’s better to more frequently test, even with a worse-performing test than it is to just do monthly on the PCR.

    We put it on the dashboard. This is the modeling dashboard… It has a couple of different purposes. So first, there was obviously when the campus was opening, a lot of particular anxiety on what may happen come September, October, and some of that [incentive behind the dashboard] was to be transparent. Like, here’s the decisions being made, and here is some of the modeling work… Everything that we know or have is available to everyone.

    And the second piece was to have a communication that safety on campus is the responsibility of everyone. That’s where the social distancing and adherence to masking comes in, why you’re allowed to change that [on the dashboard], is supposed to hopefully indicate that, you know, this really matters. Here’s where the students and faculty and staff roles are on keeping campus open. That was the two points, at least on my end, in putting together a dashboard and that kind of communication.

    BL (looking at the modeling dashboard): It’s useful that you can look at the impacts of different strategies and say, okay, if we all wear masks versus if only some of us wear masks, how does that change campus safety?

    Another question: we know that university dorms, in particular, are communal living facilities—a lot of people living together. And so I was wondering what applications this work might have for other communal living facilities, like prisons, detention centers, nursing homes. Although I know nursing homes are less of a concern now that a lot of folks are vaccinated there. But there are other places that might not have the resources to do this kind of modeling, but may still share some similarities.

    RG: Yeah, I think that’s a really interesting question. I sit here in Colorado. The state here runs some nursing homes. So we originally looked at some of those [modeling] questions, thinking about, can we model [disease spread in nursing homes]?

    I think there’s some complexities there, thinking about human behavior, which may be a little bit easier in a dorm. The dorm has a sort-of structure of having people in this suite, and then within the dorm—who resides there, who visits there, has some structure. It’s a little bit harder in terms of nursing homes, or probably it’s the same with detention centers, in that you might have faculty or staff moving across a lot of that facility, and how that movement is a constantly-evolving process. It wasn’t like a stationary state, having a structure, if that makes sense?

    BL: Yeah. Did you have success in modeling that [nursing homes]?

    RG: Not really so much with [a long-term model], it was more, we had a couple of meetings early on, providing guidance. My wife works for the state with their COVID response, so that was an informal kind-of work. They were trying to set up things and think about it, so I met with them to share some lessons learned that we have.

    BL: That makes sense. What were the main lessons? And I think that is a question, returning to your university work, as well—for my readers who have not read through your paper, what would you say the main takeaways are?

    RG: I think I would probably take away two things that are a little bit competing. One is, based on both some of the university work and the K-12 work, that we have the ability to open. We have a lot of the tools there, and some things can open up safely given that these protocols that we have in place, particularly around masking and stuff like that, can be very effective. Even in settings that I would have originally thought were very high risk. Areas that could have a very rapid spread, for example college campuses.

    Some campuses, clearly, in the news, [did have rapid spread]. But it’s possible to open safely. And I think some of the positive numbers around UC San Diego showed that. Their case counts were very manageable for us. It was possible to open up safely, and same with the K-12. That requires having a first grader wear a mask all day, and I wasn’t sure it would work! But it seems like some of that takeaway is that these mitigation strategies can work. They can work in these very areas that we would have not thought they would have been successful.

    So that’s one takeaway, that they can work. And the competing side is that there’s a lot of uncertainty. Even if you do everything right, there is a good amount of uncertainty that can happen. There’s a lot of luck of the draw, in terms of, if you’re a K-12 school, are you going to have just a couple people coming in that could cause an outbreak? That doesn’t mean that you did anything wrong. [There’s not any strategy] that’s 100% guaranteed that, if you run the course, you won’t get any outbreaks.

    BL: I did notice that the paper talks about superspreading events a little bit, and how that’s something that’s really difficult to account for.

    RG: Human behavior is the worst. It’s tough to account for, like, are there going to be off-campus parties? How do you think about that? Or is that, will the community and their communication structure going to hamper that and effectively convince people that these safety measures are there for a reason? That’s a tricky thing.

    BL: Did you see any aspect of disciplinary measures whether that is, like, students who had a party and then they had to have some consequence for that, or more of a positive affirmation thing? One thing that I saw a couple of schools I’ve looked at is, instituting a student ambassador program, where you have kids who are public health mini-experts for their communities, and they tell everyone, make sure you’re wearing your masks! and all that stuff. I was wondering if you saw anything like that and how that might have an impact.

    RG: The two things that I know about… I know there were alerts that went out, like, oh, you’re supposed to be tested every week. I don’t know about any disciplinary actions, that’s definitely out of my purview. But talking to grad students as well, I knew that if they didn’t get tested in time, they would get an alert.

    And the other thing that I will say in terms of the planning process—I got to be a fly on the wall in UC San Diego’s planning process on opening up. And what I thought was very nice, and I didn’t see this in other settings, is that they actually had a student representative there, hearing all the information, hearing the presentations. I had no idea who all of these people are on all these meetings, but I know there was a student who voiced a lot of concerns, and who everyone seemed to very much listen to and engage with. It was a good way to make sure the students aren’t getting pushed under—a representative was at the table.

    BL: Yeah, absolutely. From the student perspective, it’s easier to agree to something when you know that some kind of representative of your interest has been there, as opposed to the administrators just saying, we decided this, here’s what you need to do now.

    My last question is, if you’ve seen any significant changes for this current semester or their next one. And how vaccines play into that, if at all.

    RG: That’s the actual next set of questions that we’re looking into. If weekly testing continues, does the testing need change as people get vaccinated? The other thing that they have implemented is wastewater testing and alerts. They’re sampling all the dorms. And how does that impact individual testing, as well? Does that—can you rely on [wastewater] and do less individual testing? That’s some of the current work that we’re looking into.

    BL: That was all my questions. Is there anything else that you’d want to share about the work?

    RG: I will say, on [UC San Diego’s] end… I think you can use models for two things. You can use them to make decisions—or not make them, but help guide potential decisions. Or you can use them to backdate the decisions that you wanted to make. You can always tweak it. And I would say, in the work I’ve done, it’s been the former on the part of the school.

    The other thing is, thinking about the role of modeling in general as we move forward, because I think there’s definitely been an explosion there.

    BL: Oh, yeah.

    RG: I think it brought to light the importance of thinking about… A lot of our statistical models, for example, are very much individual-based. Like, your outcome doesn’t impact others. And I can see these ideas, coming from COVID—this idea that what happens to you impacts me, it’s going to be a powerful concept going forward.

  • What makes a successful semester during COVID-19?

    What makes a successful semester during COVID-19?

    Despite outbreak risks, a lot of colleges and universities brought their students back to campus during the fall 2020 semester. Everyone from epidemiologists to the students themselves asked: What worked, and what didn’t? How do we even measure success, when every campus is unique and every option is complicated?

    A lot of journalists have tried to answer these questions in the past few months. I took a crack at them in a feature for Science News, published this past Tuesday. My editor and I picked five universities, ranging from large state schools to small close-knit institutions. I graphed their cases and tests, attempting to determine both the drivers of campus outbreaks and how school leadership got them under control. And I spoke to administrators and students at each school who explained their campus’ approach to COVID-19 mitigation.

    Obviously, I want you to read the full story. Any institution trying to handle COVID-19 can learn valuable lessons from these universities, especially from those that got their students involved in the COVID-19 protection efforts—like Rice University, which set up a student-run court to judge those who broke safety rules, or North Carolina Agricultural & Technical University, which let students go live on Instagram while they got tested.

    But in the COVID-19 Data Dispatch this week, I wanted to share some bonus material. One of my favorite interviews that I did for this feature was with Dr. Pardis Sabeti, a computational geneticist at the Broad Institute of Harvard University and MIT. The Broad Institute helped over 100 colleges and universities set up COVID-19 testing and student symptom monitoring, most of them in New England. When I talked to Dr. Sabeti, though, she mostly spoke about Colorado Mesa University—a small school in Grand Junction, Colorado that saw it as a moral imperative to bring all of their students back to campus this fall.

    Dr. Sabeti told me all about why the Broad Institute and Colorado Mesa University (or CMU) were a great match, able to try out novel COVID-19 control efforts that many other schools didn’t consider. She also gave me her perspective on what makes a successful pandemic semester—spoiler, she has a pretty high bar.

    The interview below has been lightly edited and condensed for clarity.

    Betsy Ladyzhets: Tell me about how the Broad Institute started working on infectious disease management, and how that led to your current efforts with COVID-19.

    Pardis Sabeti: I do a lot of work in infectious diseases, mostly in West Africa. In 2014, Harvard University set up an outbreak surveillance committee that helped the school through all of these things around Ebola. And then, it was sort-of in-place, we had this committee of folks across the institution that were working together on outbreaks. 

    Then, in 2016, we got re-empaneled when there was a mumps outbreak at Harvard that ended up spreading across Massachusetts. We learned that yes, universities are laboratories for infectious disease spread, and Massachusetts has 110 of them.

    So, there was a lot going on there. We worked with the Mass. Department of Public Health and the higher ed consortium in Boston and we were really able to move things forward together, to cooperate, share data. We even found a transmission link between an outbreak—there was an outbreak in east Boston that happened in an unvaccinated community that was thought to be a separate outbreak, but then our genome sequencing data showed that it was firmly within the Harvard University cluster. And then additional case investigations showed that there were three members of that community that were Harvard affiliates, that were the likely links.

    When we did the genome sequencing, it showed us this idea that traditional epidemiology is very accurate. Whatever links the public health teams had found, we confirmed with genome sequencing. But they missed most of the transmissions. There were a lot of transmission events that were very obviously tied to each other but that the public health teams didn’t catch. 

    So at that point, we really doubled down on this idea of genome sequencing and genomic epidemiology being really important for understanding outbreaks. But then also, we understood that we needed to be very fast about doing outbreaks [sic]. What the rest of the world figured out during COVID, we figured out because of mumps—that we needed apps to essentially allow people to start sharing information about their symptoms, so people can get quick diagnoses.

    It was this funny thing where four people on my team all became infected while we were investigating the mumps outbreak with what looked like mumps. Each of them went to their own PCP [primary care provider], and their own PCP did a work-up, and you’re like—wait a second. Wouldn’t it be useful to know these four people are all in connection with each other? If one of them had a diagnosis, it would probably inform everyone else’s diagnosis.

    We created what’s now called Scout. It’s an app that allows you to share with your contacts what’s going on if you have an infection, allows people to quickly figure out what their diagnosis might be and to alert people. We weren’t thinking about it necessarily for pandemic reporting. We were just thinking, wouldn’t it be something handy, that next time you get sick, you immediately know what you have and what to do about it. Particularly since viral and bacterial infections need entirely different courses of action from people. Like, could we help everybody get informed? And then we also built Lookout, which is a dashboard that collects all that information and shows public health teams and administrators what’s going on.

    BL: Yeah, the CMU administrators I talked to talked about that [dashboard] a lot.

    PS: Yeah, which is great. We joke that CMU has one of the most sophisticated public health systems. The school can see, at this exquisite level, what the cases are, where they’re located. It’s really allowing you to do those investigations that most people I’ve seen elsewhere are doing on the back of an envelope.

    We [Broad] needed a place to work with that was going to be very collaborative and open. And so we were talking to a lot of different folks in different places, and everywhere there were different challenges of getting in the ground. And Colorado Mesa, to us, was this breath of fresh air. One, it was heartwarming to be working with this school in Colorado that has a large population of first-in-their-families-to-go-to-college students. And it was also empowering to hear the need that they had, the fact that they had to come back and they had to come back fully on campus because the students’ livelihoods and future success depended on it. And it was also heartwarming to see the way that the leadership was so engaged, so strong, so open, to anything.

    And also, like, the wastewater testing is being done by faculty and students in the engineering department. The clinical sample collection is being led by Amy Bronson and the nursing team. That’s a lot of what you want to see happening on college campuses. To me, the way I pitched it is, what we were building was the Facebook app for outbreaks that also needed to start with a close-knit community where you could get a lot of adoption. 

    But also, this idea that colleges are both high risk but also exactly where innovation can happen. It’s where people are ready to explore and try things out.

    I hadn’t seen that [mindset] at a lot of other schools. I saw this administrator, top-down, we’re gonna tell you how to behave and you’re gonna be in this room. A lot of schools got into a frame of like, we’re gonna manage these students, whereas CMU really was like, no, we’re going to partner with these brilliant students and figure this out together.

    In my mind, I was always perplexed, where we kept describing this year as this kind-of less-than year, where we were just going to suffer through education. In my mind, it was a more-than year. People learn the best when the stakes are the highest. There’s no other time we’re gonna teach kids about public health, infectious disease, genomics, and epidemiology than now. So we should shift what we’re trying to do. It shouldn’t be like, let’s get the Chaucer done while an outbreak is killing people in our community. We should’ve shifted our attention and all learned math, and stats, and clinical medicine, and public health, and biology around what’s going on.

    And that’s what CMU is doing. They’re hosting classes that are around outbreak response. The coaching teams and the sports teams are the ones doing contact tracing. It’s interesting, because it’s, in a way, it’s a school that doesn’t have all the resources where the ingenuity is going to happen. They can’t just call an outside consultant to do these things for them, they had to rely on themselves and the students.

    Did they show you the videos that they made?

    BL: I watched that “CMU is Back” one, which is great.

    PS: Yeah. They made many of them. They have a new song—I have to make a video later this afternoon for it.

    The fascinating thing is, right, even the art students got in on it and started doing public health messaging. I say, and it’s true—they already had me at the team. I just thought the team was so delightful and inspiring, but they sealed the deal with the video.

    What communities do you know that would make a video together? Most offices, they hate each other, everyone’s resentful and no one’s gonna make a video. In a lot of the schools that I know, there’s taglines that they hate the administration. There’s a fight between the administration and the students. Where here, it’s like, the administrators and the students got together and made a music video. They told me that they have a very close-knit culture and a trust in each other, that would make things go forward.

    And I’m sorry, I know this is very much my pitch for CMU, but I just love talking about this place. Here’s this thing where—did they show you the simulation that happened over Halloween weekend? Did they show you that data?

    BL: I think so, yeah.

    PS: It was the real-time where you’re seeing everybody clustering?

    BL: Oh yeah, yeah.

    PS: Yeah. What is fascinating about that whole scenario is that you had 358 students, voluntarily without any real advertisement from us, download an app that tracks all of their movements over Bluetooth, over Halloween weekend. And then proceeded to go out and do their thing. So here’s that kind-of interaction, and you’re seeing, minute-by-minute, the kind-of high resolution data that we’re getting on how students are interacting with each other. What clusters they’re forming, what times of day we need to watch out for for interactions. It’s pretty bananas.

    These students have an enormous number of contacts. This is the fear that you have with college students. Someone might look at [these data] and say, it’s terrible. But in other ways, it’s like, these kids trusted you enough to download an app, get themselves tracked, and go on and basically engage in behavior that they could get themselves thrown out. That’s trust in the leadership. That is what we need to be able to stop outbreaks.

    And then, the last piece I’ll say before I go off of my CMU storyline is… I’ve been trying in Massachusetts, for a long time, to get people to understand that you’re gonna spend millions. Each of these colleges are spending millions and millions of dollars on diagnostic testing on a daily basis or a weekly basis. That’s an incredible amount of tests that are being used with no hypothesis. Meanwhile, the surrounding communities are talking about seven days ‘til getting a test result, and standing in line for four hours for a test.

    That’s dangerous. I kept trying to convince a lot of the colleges that testing yourself in the middle of a shortage of tests looks selfish and is ineffective. Ultimately, the way that COVID spreads, one person can come into a room and infect 50 people. And so, the metaphor I use is, it’s like being in a drought with a fire alarm shortage, and putting all the fire alarms in your own house. You’ll be exquisitely good at detecting a fire when it hits your house, but at that point, it’s burning to the ground. What you should do is, you should get [the fire alarms], and you should put them in all your neighbors’ homes. For a wide stretch.

    Ultimately, what colleges should do is to support their communities’ testing, by reaching out and saying, okay, every faculty and staff and student, tell us who your contacts are, and have them tell us who their contacts are, and we will prioritize testing for those individuals. We’ll get them tested. That’s how the colleges should have interacted.

    And that really fell on deaf ears in general, there’s a variety of reasons for that. But Colorado Mesa doubled down. We [Broad] tried all these different models, like use 100% of your tests on yourself, use 100% of your tests on other people, or use 25%, 50%, 75%, those different groupings. And we found that the most effective way of stopping an outbreak is if you use 75% of your tests outside of the school. You keep 25% for yourself, but 75% should be used outside the school. That’s how you stop outbreaks on campus.

    We’re writing up that work right now, but even when we showed Colorado Mesa the preliminary data, they were like—that’s now their new model. It’s essentially what they’ve done. They’re putting the majority of their tests [in Grand Junction, the city around the school]. And to me, that’s going to be the really remarkable thing to watch going forward. We’ve created the apps, and the dashboards, and the systems to be able to do this well, but now we really want to reach out to our surrounding community and see where we can go here.

    BL: I know they mentioned to me that they were starting to help the other schools—like, the elementary and middle and high schools in Grand Junction get tested as well.

    PS: Yeah. Our foray into community testing was there. Basically, when the school stopped and they had this break over the holidays, they started pushing this community testing… It’s all about trust, right? They got the trust of their students, and now they’re getting the trust of the community. They’re saying, okay, we’re here to help you, how do we work through this together. That’s the idea behind it.

    So that’s all of my CMU backstory. But it also just generally tells you about the way I think things need to happen. Colleges are both a laboratory for infectious disease spread and also a great laboratory in which to try new technologies out, but it really has to involve community engagement, empowering of all the actors in the system, and trust-building. It does have to involve bringing the students on board on the mission, not just coming top-down and telling them how to do things, and reaching out to the communities and doing testing for your communities.

    It both makes you look more selfless because you’re a college helping your community. That’s always a great way, when you’re going to throw a party in the middle of the night, for them to be happy that you’re there. This[fall 2020] was the opportunity for all colleges to get buy-in from their communities, to show why they’re there and why they’re useful, and that’s another thing where it’s like, why are we not doing that? We have that opportunity.

    BL: That’s definitely something that I saw in part at some of the other schools [in my story], but not to the same degree as what Colorado Mesa was doing. I think you answered a lot of my questions already, because I was going to ask you, like, what makes colleges a good place to try out mitigation methods.

    But one more question is, do you have specific parameters that you would think about when you look at, say, cases and testing numbers, of what you would consider a successful fall semester for a campus?

    PS: The thing is, most schools had very unsuccessful semesters…. For me, success would be… The bar for me for success is really high. To justify coming back when so many people can’t… It would be, not having an outbreak on campus, or not seeding an outbreak in the community. Which could happen—you could not have an outbreak on campus but could have seeded one in the community, if you caught it and you were able to quarantine your people but you already spread it there and the whole thing went on fire. Essentially if your surrounding community has lower case rates.

    I always talk about, when you do something that’s counter to what you should be doing, success is going far and beyond. For me, when I have my students go into someone else’s lab, I’m like, you need to leave that lab better than when you found it. If you’re a guest in someone’s home, if you are treading in a place you shouldn’t tread, your level of success is leaving the college and the community better than when you found it. And having the students learn new skills, be engaged, and feel excited about the future.

    The fact, again, that CMU has their new song— which they just sent me, and it’s a little silly ‘cause it has all these excerpts of me talking—is, “The Future is Now.” And that, to me—even though, by the metrics of what I was just talking about, they weren’t successful. They had an outbreak on campus, it might have spread to the community. But they made a big headway, they learned a lot, the students engaged a ton, and they collectively were making the community around them better. That to me is—I think they had a successful semester in that the students were engaged and they learned, and they attempted to support the community around them. And from that will learn to be even better and stronger.

    BL: Is there anything in particular that you are expecting to be different this spring, learning from CMU and from the other schools you’ve worked with via the Broad Institute?

    PS: This spring is going to be very… it’s going to be hard to know how it will go. You’re gonna get vaccines coming in, that’s gonna make things better, but you have case numbers that are really high, variants that are more infectious, that are gonna make things worse. And a lot of civil unrest and tensions and all of that.

    It’s one of those things where we really have to double down on our civic engagement, I think that’s going to be really important. And on our public health view of what’s going on.