Test, test, test.
If only it was that easy.
When New Zealand was at the height of its Covid-19 outbreak – you can remember that, can’t you, don’t tell me you’ve forgotten already – there were a lot of comments and complaints about testing. Were we doing enough tests? Were we testing the right people? What about people who hadn’t been overseas but still had symptoms? What about people who just wanted a test for reassurance? Among the criticisms, one stood out to me. A doctor noted that when she had been working on tuberculosis in Indonesia, they had gone door-to-door testing people. She thought that was what New Zealand should have been doing for Covid-19.
At the time, it struck me as a typically academic solution to the problem. Theoretically, if we could knock on every door and test every person, then we would certainly know how many Covid-19 cases there were in New Zealand. On the other hand, it would be impossible to implement in a time scale that was useful for a fast-moving pandemic.
Tuberculosis and Covid-19 are very different diseases. They both particularly affect the lungs, but that’s really where the similarity ends. Tuberculosis is caused by a bacterium, Mycobacterium tuberculosis, which currently infects about a quarter of the world’s population. In the majority of infected people it is latent, causing no harm and not being transmitted to others. Around 10% of infected people will go on to develop the disease tuberculosis, a slow-moving infection which, untreated, kills about 45% of those who become ill. For those with HIV, untreated tuberculosis is almost always fatal. In 2018, tuberculosis killed 1.5 million people worldwide, more than any other infectious disease (malaria, in comparison, killed 405,000 in the same year).
As long as the infection remains latent, it cannot be transmitted. However, once it becomes the disease tuberculosis, it becomes infectious. Because it starts out with mild symptoms and acts slowly, there are many opportunities for someone with tuberculosis to infect others. Over the course of a year, someone with active tuberculosis can infect 5-15 others. With such high rates of infection – most of these in the developing world – it’s easy to see why going door-to-door in areas with active cases makes sense.
With Covid-19, caused by a virus known as SARS-CoV-2, the picture is very different. There’s no cure and no treatment that can prevent infection. It’s fast – in the worst cases the disease resolves, one way or another, in a couple of months. And in most of the world, it is still not particularly prevalent. In New Zealand, at its worst, less than 0.02% of the population were infected. Even in some of the worst-affected countries, such as the USA, the reported numbers suggest that less than 1% of the population is currently infected.
Searching randomly for a disease of low prevalence is very unlikely to detect many cases, unless there is a massive amount of testing. At the simplest level, if you tested 100 people when only 1% of the population was infected, you’d expect to detect only one case, on average. But the rules of chance mean that you may not pick up that single case at all, or you might be lucky and pick up more than one. But that’s a lot of negative tests to pick up a few cases. That’s why most of New Zealand’s testing has been focused on people with symptoms which could be associated with Covid-19 and close contacts of people known to have the disease. It gives a much greater chance of picking up positive cases.
There’s another problem when testing populations for a disease with a low prevalence, like Covid-19 – test accuracy. No test is perfect, we certainly can’t expect the hastily-developed tests for the SARS-CoV-2 virus to be better than well-established tests.
Initially, I thought it would be useful to compare the tests for tuberculosis, a globally common and well-known disease, with those for Covid-19. Surely, for such an ancient and well-known disease, we would have great tests.
Well, not exactly. When I started investigating some of the more widely used tests, I discovered that most of the common diagnostic tests really aren’t very good at all. The tuberculin skin test – first developed in the late 1800s and, with improvements, still used in New Zealand today – has a shocking accuracy rate when considered on its own. A systematic review of test accuracy, that is, a study of many different studies on the subject, found that the test would return a positive result in only a third of people who were actually infected with tuberculosis, although that depended on how a positive result was defined. To an alarming extent, it would also report that people without tuberculosis actually had it – if 100 people who were not infected with the tuberculosis bacterium were tested, more than a third would return positive results.
However the tuberculin skin test is unlikely to be a good indicator of the accuracy of Covid-19 tests. It is an indirect test, that is, it doesn’t detect the presence of the bacteria which cause tuberculosis, but the immune response to those bacteria. Direct tests using newer technology based on detecting the DNA of Mycobacterium tuberculosis are more accurate. Since these are more like the tests used for Covid-19, they give a better representation.
A systematic review of studies done in China which tested for tuberculosis with different types of DNA test gave much better results than the tuberculin skin test. False negatives, that is the test delivering a negative result in a person who is infected, occurred in around 10% of tests, while false positives, where the test indicates someone is infected when they aren’t, were slightly lower, at around 7%.
This is why disease diagnosis isn’t left to tests alone. In the case of tuberculosis, as well as tests, diagnosis can include medical history, physical examination, chest x-rays and culturing samples to see whether bacteria can be detected.
So where does this leave testing for Covid-19?
Firstly, it tells us that direct tests for the presence of the SARS-CoV-2 virus are likely to be better than antibody tests. Although the low accuracy of the tuberculosis tests related to a wide range of different factors, one of them was the variation in how the immune systems of different people responded. However good the test, an antibody test will always be affected by this individual variation. This doesn’t mean that antibody tests are not useful, but we need to be conscious of their limitations. If someone returns to New Zealand from the USA or Britain, and reports having had a fever and cough while there, a positive antibody test would be confirmation that they had had Covid-19. But if I were to be tested and return a positive result, I’d be suspicious, since I’ve had no contact with any known cases and haven’t had so much as a sniffle in nearly a year.
Secondly, it tells us that we need to have a degree of scepticism about testing in general. Without confirmation from symptoms and patient history, testing may not be particularly useful. What I’ve read suggests that we really don’t know how accurate tests for SARS-CoV-2 are. I did find a useful article from Siouxie Wiles on the subject, and that article gave some reassurance that the SARS-CoV-2 tests are not as inaccurate as the tuberculosis tests. For example, she mentioned four direct tests for the virus which returned no negative results for people who were infected and only 3-4% of positive results for people who were uninfected. These are much better figures than for tuberculosis, but their sample sizes were very small – they used only 50 infected people and 100 uninfected. The systematic review published by Chinese researchers that I mentioned earlier reviewed nearly 8000 publications before selecting 38 for analysis – these 38 studies covered around 25,000 participants.
Mostly, what this tells us is that tests are a useful tool when combined with patient history – for example whether the patient has been in a country or city with community transmission, or in contact with a known case – and symptoms. But they may not be all that useful for surveillance in places where prevalence of the virus is low.
The problem of wider surveillance for Covid-19 has been approached in two sharply contrasting ways, one which involves malodorous reality, the other, mind-bending theory.
Testing sewage for the presence of disease in a community is not a new idea. Many human pathogens, including Covid-19, occur in wastewater, the result of them being excreted in faeces, urine and other bodily substances. Usually, this is an excellent reason to try to avoid contact with water containing human waste, but it can also be turned to our advantage. The concept has even been investigated for bioweapon surveillance. Many of the pathogens which are potential biological weapons can be detected in either urine or faeces, although in most cases they had never been studied in wastewater.
Monitoring wastewater for SARS-CoV-2 has been investigated in a number of countries, including the Netherlands, Sweden, the USA and New Zealand. The approach was tested by the New Zealand Crown Research Institute ESR in April, at the time when we had our greatest number of active cases. They did find the virus in wastewater samples, although the levels were so low that they were right at the limit of what could be detected.
While wastewater monitoring can indicate that there are people in a community who are infected, it doesn’t help identify who is infected. For that, we need to go back to testing people directly. But, right now, New Zealand is testing a lot of people with the common cold or seasonal flu, just to be sure they don’t have Covid-19. It’s costly in both resources and time, and not all countries can afford it.
The idea of pooling samples for testing goes back to the 1940s and a paper by Robert Dorfman with the startling title “The detection of defective members of large populations”. The reference to “defective” was derived from studies of manufacturing processes, but the paper itself was aimed at identifying men with syphilis called up for military service. For a disease with a low prevalence in the population, such as syphilis, the efficiency of testing could be improved by taking blood samples from a number of men and testing them together. If a negative result was returned, all the men were clear, but the disadvantage was that a positive result meant that all the men whose blood was in that sample needed to be retested. For a disease such as tuberculosis, assuming its average prevalence of 25% worldwide, there would be barely any cost saving at all. However syphilis was (and still is) a relatively rare disease, and such an approach was calculated to give up to an 80% cost saving.
One of the first countries to pool samples for Covid-19 testing was Ghana. At first glance the testing rates for Ghana appear fairly average – it ranks 125th for the tests conducted per capita. However among countries with a similar GDP, its testing rate is among the best. That is likely to be, at least in part, because it adopted this efficient approach to analysing tests.
But the most intriguing approach to improving the efficiency of Covid-19 testing appears to have come from another African nation – one of the countries I mentioned previously as doing a good job at managing Covid-19, Rwanda. A group of researchers at the University of Rwanda in Kigali have developed a new algorithm to improve the efficiency of pooled testing. They used a mathematical model based on something called hypercubes to calculate the optimal number of samples to pool in order to detect infected individuals when the disease is at low prevalence.
I admit, I tried to read their paper (which is the second link) and largely failed. It’s full of equations and all I could do was pick out the key phrases. The first link is an interview with one of the authors, and gives a much easier explanation. Professor Leon Mutesa, a medical geneticist, explains that they could pool 20 or even 50 samples and still detect the virus if it was present, and that it has been used in a range of situations, including markets and prisons. The approach saved not only cost but also time, because in the majority of cases, retesting isn’t needed.
That last point – that testing can be completed in a single step – seemed so counter-intuitive that I decided I needed to understand it better. Dorfman’s strategy for syphilis testing required retesting if one of the samples came up positive. How could pooled samples possibly give results that wouldn’t require retesting if a positive result was returned?
The mathematical model which performs this remarkable feat was developed by Professor Mutesa’s colleague from the African Institute for Mathematical Sciences, Professor Wilfred Ndifon. It uses the geometry of multidimensional shapes – the hypercubes I mentioned previously. I can’t really see what a hypercube is in my head, but there’s a nice video explaining a 4 dimensional hypercube, which is enough to help you get the picture. Each test sample is represented by a point on the hypercube, and the pooled samples represent slices through the hypercube. If one of the pooled samples returns a positive result, its location on the hypercube can be worked out by ruling out all the individual samples it couldn’t be, because they were also in a pooled sample which returned a negative result. The closest I can get in my mind to understanding how this works is working out the answers to a sudoku problem.
In all honesty, I wish I could give you a clearer answer, but I can’t. This is the best explanation my brain can manage. Beyond this, all I can suggest is that you join me in quietly admiring the brilliance of the minds which come up with such things, and remembering that, sometimes, exactly what we need is an academic solution to a problem. And perhaps we can also hope that this work is adopted more widely, so that countries around the world can have more efficient testing for Covid-19. Because, although it’s possible to forget about the pandemic when everything seems so normal in New Zealand, it’s not going away anytime soon.
Do you know someone who might enjoy The Turnstone? Please forward it to them.
Did you receive The Turnstone from a friend? You can subscribe and receive it directly every week.
Have you found a factual error or typo? Let me know and I’ll correct it in the archive version.