How much should we blame FDA for the COVID-19 rapid antigen test situation?
I cover the clinical lab and diagnostics beat, which, for the last two years, has basically meant COVID-19 testing. As part of that coverage I have, of course, followed the great rapid antigen test debate and, in particular, FDA's role in it all. As regular readers of the internet are doubtlessly aware, many people have strong opinions about FDA's regulation of COVID-19 rapid tests. Inevitably, I've developed my own opinions over the course of reporting on the topic. They've remained a bit formless, though, and having some time on my hands over the holidays, I thought I would try to work through them—both as an exercise in figuring out exactly what I think about the matter and with the idea that maybe it would be of interest to others who've been following this story.
I should say at the beginning that I start from a point of relative sympathy to FDA. The outlet I work for (We’re subscription-based, but if you have an academic or non-profit email address you can register to read for free. Check it out!) covers a lot of pre-commercial and early-stage diagnostics using newer genomic and proteomic technologies. I've been writing about this space for a little over a decade, and while there are a lot of people in it doing very good, very interesting work, there are also a fair number of companies trying to develop tests based on half-baked clinical studies and half-baked business plans. When they fail (which most do), they often point to FDA and its overly onerous regulatory process as a scapegoat. That's not to suggest that FDA's diagnostics regulatory process is perfect, but just to say that I don't necessarily take a testing company's complaints about FDA at face value.
Anyway, rapid tests. In my mind, the COVID-19 rapid test discourse has passed through four different stages. The first started in the spring of 2020 when it became apparent that PCR testing couldn't (or, at any rate, wasn't going to) scale quickly enough to meet the demand for testing and besides would always suffer from a delay in turnaround time. In March 2020, BARDA started providing companies like OraSure with funding to develop rapid antigen tests. In April 2020, you started to see big diagnostics players like Roche, BD, and Quidel announce plans to develop rapid antigen tests. FDA started talking about them, too. At the beginning of May, Timothy Stenzel, who heads FDA's Office of In Vitro Diagnostics and Radiological Health, said that the agency was working with some rapid antigen test developers in hopes of making their tests available "in the not too distant future."
The second stage, which to my mind started to gain momentum in late summer 2020 and emerged full blown that fall was the "why isn't FDA letting any rapid antigen tests through?" stage. This phase was also when you first began to see widespread advocacy on the part of people like Michael Mina for approvals of lower performing (relative to PCR) rapid antigen tests, the notion being that individuals are at their most infectious when viral loads are high, meaning that less sensitive rapid antigen tests could identify people most likely to transmit the virus, especially if testing is done frequently. Ideally that meant cheap tests available over-the-counter. Was FDA at this point actually keeping these kinds of tests from getting to market? Yes? Kind of? Let's see.
At the beginning of August 2020, the agency put out its template for at-home COVID-19 testing, which called for prescription home COVID-19 tests intended for use in symptomatic people to have 80 percent sensitivity compared to PCR and over-the-counter home COVID-19 tests for use in either symptomatic or asymptomatic people to have 90 percent sensitivity compared to PCR. Those are both pretty high sensitivities – especially the 90 percent sensitivity required of the OTC test, so I think it's fair to say that, in theory, FDA was keeping people from getting ahold of potentially useful rapid tests—OTC rapid tests in particular.
How much FDA mattered in practice at this point is less obvious. Around this time, I called a bunch of companies developing rapid COVID-19 tests with an eye toward the OTC market—OraSure, E25Bio, Mammoth Biosciences, UbiDx—and all of them told me FDA hadn't been a problem. Most were either still running clinical trials to collect data for an Emergency Use Authorization submission or were in the process of putting a submission together. They weren't, at least as they described it to me, just sitting on a bunch of OTC tests that were ready to go but for the hapless bureaucrats at FDA. Now, the fact that they all said FDA wasn't a problem doesn't mean FDA wasn't actually a problem. Diagnostics firms might blame FDA after their plans flop, but they don't typically badmouth the agency to reporters when they are trying to get a test approved. (And, while E25Bio told me FDA wasn’t an issue, their co-founder and CEO went on YouTube around the same time and blamed the agency for holding up its tests, so, you know…)
That said, OTC rapid tests weren't really in use outside the US at that point, either. The UK, for instance, didn't roll out its rapid testing program until January 2021. OTC rapid tests weren't available in Germany until March 2021. FDA authorized the first OTC rapid antigen test (from Ellume) for use in the US in December 2020 and authorized OTC tests from Abbott and Quidel at the end of March 2021. So while you can certainly point to the agency's high performance requirements as obstacles potentially preventing companies from bringing OTC tests to market, the US wasn't really behind the rest of the world in this regard through the fall and winter of 2020-2021.
This stage (we're still in what I'm calling stage two) also saw the notion of serial testing using cheap OTC tests gain traction. Again, the idea was that while PCR tests were more sensitive than OTC rapid antigen tests, regular, frequent testing with OTC tests could compensate for their lower sensitivity. If we could just get enough cheap rapid tests on the market, people could use them two or three times a week to monitor themselves for COVID, allowing us to quickly identify positive individuals and break the chains of transmission.
At first this idea was supported mostly by modeling experiments, but since then a few studies have provided real-world data showing it could work. A study out of the University of Illinois looked at 43 people and found that testing every three days with Quidel's Sofia SARS Antigen FIA test detected COVID-19 cases with sensitivity of 98 percent, the equivalent of weekly testing using PCR. The study also used viral culture assays to assess when tested individuals started and stopped being infectious and found that when people ceased to be infectious they also stopped registering as positive on rapid antigen tests. Their PCR tests, on the other hand, often remained positive for days afterwards—bolstering another key argument of rapid antigen proponents, that PCR's high sensitivity actually kept people in quarantine for longer than they needed to be. (On the flip side, though, PCR also did better picking up cases early in infection.)
One caveat to this study—the Quidel test used was a machine-read assay (it used Quidel's Sofia 2 reader) that had received FDA EUA, so it didn't really address the question of whether cheap lateral flow tests that didn't meet FDA's EUA requirements would be effective at catching cases when used serially.
Another study, using E25Bio's test, a simple lateral flow assay that hadn't (and hasn't) received FDA EUA, did address this question. The study followed 257 individuals at three labs in Cambridge and Boston over the course of six months. Participants were tested twice weekly using the E25Bio test and by PCR. Over the course of the study, 15 individuals had COVID-19, with all of these cases picked up by twice-weekly rapid antigen testing. Looking at the performance of the E25Bio test at single time points, the assay was 79 percent sensitive within zero to 12 days of symptoms emerging and 96 percent sensitive within zero to three days of symptom onset.
While in both cases the tests missed around 20 percent of early-stage infections, by using the tests serially people were ultimately able to catch them. Based on these two studies, and the E25Bio study in particular, regular screening with cheap rapid antigen tests seemed to work, just as its advocates said it would.
So why was FDA being so strict in its specifications for OTC rapid antigen tests, requiring, according to its templates, 90 percent sensitivity compared to PCR? If you asked the agency this question in, say, the winter of 2021, officials there would tell you that, actually, they were totally flexible on sensitivity for OTC tests intended for serial use. And it's true that the agency had been putting out this message since at least the spring of 2020. On a September 2020 call for labs and test developers, FDA's Stenzel put it explicitly, telling the audience that "where our recommended levels of sensitivity may not be achieved with a single test result in a home situation, maybe with the paper strip test, strategies utilizing serial testing, for less sensitive tests, could be deployed." He cited the example of a test with 70 percent sensitivity that might be sold as a two-pack for serial testing. When I spoke to FDA at the beginning of March 2021, an official told me that they had been making a concerted effort to communicate this openness to test developers and testing advocates but that they hadn't received any submissions for products pursuing a low sensitivity, serial testing model. This seemed weird. Given the amount of attention the idea of serial COVID-19 testing was getting (at least among people who followed that kind of thing), why hadn't anyone tried taking this kind of test through FDA?
I called some of the test developers I'd been speaking to throughout the pandemic—E25Bio, OraSure, Ellume, Innova—and asked them why they hadn't tried to get a lower sensitivity, serial test through FDA. They all, with the exception of Innova, essentially told me that while they were aware that FDA said it would consider serial tests that didn't meet its stated sensitivity requirements, they didn't actually trust the agency on this point. (Innova told me that they actually had tried to get a serial test through FDA but had been turned down by the agency. FDA never replied when I followed up to ask about the discrepancy between its claim that it hadn't received any serial test submissions and Innova's claim that it had unsuccessfully submitted such a test.)
Stephen Tang, president and CEO of OraSure: "We have had a long relationship working with FDA. We take them at face value. The templates are the templates. And that is what we are targeting and abiding by. That is the way we are playing the submission that we are about to make."
Prashant Chouta, CEO of E25Bio: "Obviously, any application will be considered. It's not the consideration that matters, it is the approval that matters. There's a big difference between, yes, I will consider, and yes, I will approve.
"Obviously, to get to a higher level [of performance] it takes time, it takes validation, it takes research. If you are asking, would you have had a product out there if the sensitivity [requirement] was lower, probably, yes. [But] "we would not waste our time with a submission that does not meet the minimum requirements that the FDA mentioned in its template."
Sean Parsons, CEO of Ellume: "We certainly took to trying to make the most accurate test we possibly could."
To paraphrase Strother Martin, what we had here was a failure to communicate. FDA was saying, hey, we're nice guys, we can be flexible, trust us, and test developers were basically saying, yeah, no. On March 16, 2021, the agency, apparently now aware of the disconnect, put out a new template detailing its requirement for OTC tests meant for serial use. Abbott and Quidel launched their OTC tests at the end of the month, and a few additional OTC tests would make their way to market over the next several months.
It was slow going, though. By October 2021, only eight rapid COVID-19 tests had received FDA EUA for OTC sales. This sluggish flow of tests to market brought on what I think of as stage three of the great COVID-19 rapid test discourse—the "why does Europe have so many more rapid tests than we do, and why are they so much cheaper?" stage.
While at the start of 2021 rapid testing hadn't really taken off yet in the US or Europe, by the late summer and early fall, the discrepancy between the two geographies was obvious and widely noted—especially given the need for testing due to the ongoing Delta wave. The UK and Germany (which were the most commonly cited comparator countries) provided rapid antigen testing free of charge, with the UK letting people order tests for free online or to be picked up from local testing centers and the German government providing asymptomatic individuals with one free rapid test per week to be conducted at a range of healthcare facilities including doctor's offices, public test centers, and pharmacies.
Additionally, in countries like Germany, home tests from dozens of manufacturers were available at a wide range of pharmacies, groceries, and other stores, typically in packs containing multiple tests and at a price of around €1 per test. No trip to Europe was complete until you had tweeted a picture from a Lidl of an overflowing bin of discount rapid tests. Why didn't we have our own overflowing bins of discount rapid tests?
The FDA was definitely one reason, though the situation wasn't quite as straightforward as it was often described. One notable fact: When Germany started allowing rapid antigen tests to market, it was flooded with poorly performing tests, which led the government to put in place a requirement that to be reimbursed or used in an official capacity — clearing a person to enter shops or restaurants, for instance — a test had to pass a performance evaluation done by the country's Paul Ehrlich Institute. To pass, a test needed to show at least 80 percent sensitivity compared to PCR—the same sensitivity required by FDA's serial testing template.
If both countries had the same sensitivity requirements, why were tests so much more widely accessible in Germany? One likely reason is the various regulatory considerations beyond a test's analytical performance—so called human factors, for instance.
"If you want to put a test on the market under an [FDA] EUA, you need to provide a human factors study showing that someone at home, an elderly person, whoever, can use it safely. In Europe, you can get by with a single page saying, 'Here are the instructions for use. We don't see any problem, bye-bye,'" Philippe Etter, a medical regulatory consultant, told me when I interviewed him a few months ago. (User error does seem to be an important consideration when it comes to rapid test performance. Data from the UK showed Innova’s tests had 77 percent sensitivity when done by trained staff but just 58 percent sensitivity when performed by self-trained members of the general public.)
Etter, who works with regulatory agencies around the world, also suggested that the FDA's reputation had likely kept many vendors and distributors from even trying to bring their rapid tests to the US in the first place.
(A brief side note: One suggestion often made by critics of FDA's regulation of rapid tests is that the US should approve for use any tests being used in Europe. This may or may not be a good idea, but it elides the fact that the US and Europe have very different regulatory standards for diagnostics generally. In the EU, most diagnostics are regulated through a self-certification process and don't have to pass muster with a regulatory body. That's in stark contrast to the US (and countries like Japan and China) where nearly all widely distributed diagnostics have to go through FDA. I think most Americans consider US and EU regulatory regimes as being roughly equivalent with, if anything, the EU typically being stricter, but that really isn't the case here. For what it's worth, the EU is in the process of putting in place FDA-style diagnostics regulation.)
"You could clearly see [during the pandemic] European distributors going hunting in China for products and catching everything they could," Etter told me. "They would make a list of documents to be provided by the Chinese manufacturer. They would forward [those documents] to the state. And away they would go. The regulatory burden and the risk for a distributor [in Europe] is not very high."
On the other hand, he said, "in the US, if you try to cheat the FDA [by marketing a subpar test], and the FDA finds out, you're dead. You will be always and forever under scrutiny with anything you try to bring into the country. It would be a very dangerous decision." (Etter generally supported FDA's approach to rapid tests over Europe's, telling me that "the authorities have opened the throttle in Europe … and if you were to calculate the actual effectiveness [of many tests], it would probably be quite mediocre.")
I asked an FDA official about Etter's notion that the agency's reputation had kept some companies from even trying for an EUA. They said that the agency had not received submissions from many of the companies offering tests in Germany, which suggests Etter might be on to something. In November, FDA and NIH began working together through a program called the Independent Test Assessment Program to recruit and evaluate for EUA tests currently being produced at large scale in countries outside the US, which similarly indicates that the agency hasn't been receiving submissions from companies that could meaningfully add to the country's rapid test supply.
So, I think it's pretty clear that FDA has limited the number of vendors selling rapid COVID-19 tests in the US, both explicitly through its higher regulatory requirements and indirectly by scaring off potential applicants. The extent to which this necessarily translated into scarcity and high prices is less obvious. On the one hand, you have your basic tenets of a market economy. More makers of tests competing for sales equals more and cheaper tests. On the other hand, you can imagine a scenario where the federal government purchased oodles of tests from the half dozen or so vendors that were able to meet FDA's requirements and provided them to the public for free or at very low cost. That's essentially how the UK rapid testing program, which has relied on just a few test makers, has worked.
This, then, brings us to the fourth and current stage of our rapid test discourse—the "do these things even work?" stage. Because, as many people have noticed of late, the UK, for all its vaunted rapid test availability, has done a fairly crap job of controlling the pandemic. I'm not including the more recent data from Omicron, which seems to present rapid tests with more problems than have previous strains (as I understand it because Omicron isn't present at high amounts in nasal swabs rather than because of a decrease in the test's ability to detect the viral protein itself) but as of the beginning of October, the New York Times' COVID Tracker had the UK at total cases of 12,113 per 100,000 and 214 deaths per 1000,000 compared to 13,238 cases per 100,000 people and 213 deaths per 100,000 people in the US.
I've been following this question since the first large studies on the use of rapid tests started appearing in the winter and spring of 2021, and the conclusion I've come to is that while they can be very effective when used in a highly structured environment like a school or workplace (a la the U of Illinois and E25Bio studies mentioned above) it's hard to see a clear impact when scaled to the level of a city or county or country.
From November 2020 through April 2021, the UK government ran a trial studying the impact of widespread testing with the Innova SARS-CoV-2 rapid antigen test in Liverpool. Over that time period, 283,338 residents (57 percent of the total city population) took an Innova test, with 47 percent taking more than one test. The study investigators estimated that use of the rapid test increased case detection by 18 percent compared to similar UK areas without a rapid testing regime and prevented between 850 and 6,600 infections. The study saw only an insignificant reduction in hospital admissions, however.
An earlier study, in Slovakia, ran 5,276,832 SD-Biosensor Standard Q rapid antigen assays testing more than 80 percent of the country's population between 10 and 65 years old along with older adults who were still a part of the workforce. The researchers estimated that cases declined by 70 percent over this time period compared to a scenario where no steps were taken to mitigate spread. Sounds great! But there were also a number of other interventions introduced during this period. And the study, as the authors noted, was observational, which made it difficult to separate out the impacts of these different interventions and determine just how useful testing actually was. Another bit from the study's authors worth highlighting: "Transmission in Slovakia has rebounded, despite other interventions, because high-intensity testing was not sustainable."
It's that last part that makes me think, ultimately, that the FDA, with its annoying reluctance to loosen its requirements for rapid tests, has been closer to the mark than its critics give it credit for.
The argument for using lower sensitivity rapid antigen tests has always relied on the idea that you can catch cases and break chains of transmission by testing widely and frequently. But if people aren't testing frequently, in a regular cadence like the participants in the U of Illinois and E25Bio studies, if they are just testing occasionally to spot check here and there, that argument becomes less compelling. I don't have any great data on how people here have been using rapid tests, but whenever I come across someone either IRL or online looking for a test, it's almost always because they have symptoms or they want to test before visiting family or going to an event or something. Single-timepoint, not serial testing, in other words.
Now, you might say that the reason more Americans aren't using rapid antigen tests in a structured, serial way is because FDA's regulations have made the tests too scarce and expensive for such an approach. But I think the example of the UK suggests that even if we did have plenty of tests and low prices, most people wouldn't be testing themselves on a regular basis.
Alex Crozier, a researcher at University College London who has been studying the UK's rapid testing effort, told me that while authorities there had hoped people would use the tests in a somewhat routine fashion, that hasn't actually been the case, and that, he suggested, has undermined the program's effectiveness.
"Challenges arise when individuals, well-meaning members of the public, are given these tests and they have mild symptoms but test negative and then still go out," he said. "They don't understand that these tests are really risk mitigation tools, rather than a sort of definitive yes or no answer."
If you posit a citizenry that is going to adopt regular two- to three-times a week testing in a widespread way, then lower performance tests used serially are a great idea. If, on the other hand, you envision a scenario where rapid tests are mostly used for occasional spot testing at single points in time, then the performance of the tests becomes more important. It seems to me that a lot of rapid antigen test proponents and FDA critics have been assuming something like the first scenario. FDA, whether consciously or just through sheer institutional stubbornness, has acted like the second scenario was the far more likely one, and at this point it looks to me like FDA was right.
(One other side note: early in the pandemic we actually ran the experiment where FDA completely opens up test approvals. In March 2020, the agency announced it would let companies put COVID-19 serology tests on the market without any regulatory process beyond self-certification—basically the same approach the EU has taken to COVID-19 rapid antigen tests. It didn't go great. Dozens of serology tests—many of questionable quality—flooded the market. While the agency had specified that these tests could not be marketed as standalone tests for determining whether or not a person had been infected, many vendors sold them for this purpose anyway. Additionally, governments, ranging from the Trump administration down to various localities started pushing use of the tests in this way. When I spoke to Jeffrey Shuren, director of the Center for Devices and Radiological Health ) at FDA, he told me that this, in particular, was unexpected.
"We didn't expect to see what happened at different levels of government, in the promotion of those tests," he said. "At the federal level, state level, local level, you had people who were latching on and promoting it for opening up the economy, and that started to drive a lot of the demand."
I saw this myself—friends of mine who run small businesses were approached by fly-by-night COVID-19 serology outfits offering to test their workforces to determine who had already had COVID-19 and could therefore stay on the job without worry. The problem with selling mediocre serology tests for identifying if a person had been infected was that, given the still low prevalence of infections at that point in time, any test without very high specificity (like the high 90 percent range) would produce a lot of false positives, leading to people walking around thinking they had gotten COVID-19 and recovered when, in fact, they hadn't. FDA ended this self-certification pathway at the beginning of May and ended up pulling more than 200 tests from the market. I think it's likely that had the agency similarly tried an EU-style approach to rapid antigen tests, the same thing would have happened—you'd get a bunch of shady products flooding the market. Maybe, as FDA's critics have argued, those shady tests would still be effective if used in a serial fashion, but, as I said above, in retrospect, I don't think widespread adoption of serial testing was ever really in the cards.)
So then, taking into consideration all of the above, my "how much should we blame FDA for the rapid antigen testing situation" scorecard works out something like this:
Given the agency's earlier experience with serology testing and the fact that widespread serial testing as envisioned by its advocates hasn’t really taken off in the US or the EU countries often cited as models of what could have been, I don't think FDA was wrong to maintain its relatively strict requirements for test performance, usability, manufacturing etc. I think these requirements certainly did limit test supply. But I also think that, as EU caseloads suggest, a higher supply of low quality tests wouldn't—given how these tests are actually being used—have done much to improve our situation.
Where I might blame FDA is for not doing a better job of identifying companies who were producing rapid tests at scale that could pass agency muster and getting them to make a submission. Arguably, this isn't its job—if a company doesn't want to submit its test for some reason, FDA shouldn't have to chase them down and convince them otherwise (there might, in fact, be rules in place that kept FDA from doing this—I don't know, I'm not a regulatory lawyer). But the success of the Independent Test Assessment Program in bringing some big test makers to the US market suggests there has been a lot of capacity just sitting on the sidelines. HHS rolled out ITAP at the end of October. By the end of the year, Roche and Siemens, two of the biggest diagnostics companies in the world, got EUAs for rapid tests via the program. Ideally, you would have had something like this in place a year ago.
Ultimately, though, I think the key problem was the failure of the federal government to be more aggressive about purchasing and subsidizing rapid tests. You didn't have to open the market to lower performing tests in hopes of driving volumes up and prices down. Instead, (as many, including FDA’s critics, have suggested) you could have guaranteed Abbott, Quidel, BD, etc. a market for as many of their tests as they could crank out. As for whether that would have led to a better outcome, honestly, I'm skeptical even an overabundance of high-quality rapid tests would have meaningfully altered the course of the pandemic. In any case, though, I think FDA's rapid test policies have been imperfect but basically fine and certainly not the disaster many consider them to be.
Fin