A lot of people are making statements about COVID-19, comparing one nation/state with another. This article is intended to highlight the conditions that must be satisfied for such comparisons to be valid. Although this article is about COVID-19, one could easily adapt this to another health condition.

When comparing any two entities, one must always compare like with like. This means that one should be comparing ‘apples’ with ‘apples’, not ‘apples’ with ‘oranges’. In order to ensure that one is making a valid comparison, many factors should be identical:

Population characteristics

This includes the age and sex distribution within the population. That is, the number of males and females in any age group should be similar in the populations being compared. If there are more elderly people in one population, the disease is likely to have a different effect than in a population with mostly young individuals.

Where there is a sex difference in susceptibility (males are more susceptible to COVID-19), the relative proportions of males and females assumes significance. In populations with a disproportionately higher number of males, the disease would have a different effect than a population with equal number of both sexes.

It is unlikely that any two populations will be identical in age and sex distribution. Therefore, in order to make comparisons between populations, one must perform ‘standardisation’. Standardisation may be

‘Direct’ (Here the mortality rate of each age group in one population is applied to a ‘standard’ population to obtain estimated deaths for each age group. The procedure is repeated with the other population, and the estimated deaths thus obtained compared directly. However, one needs to have the age-specific death rates and population details of each age group (in both populations).) or

‘Indirect’ (Here the age-specific death rates are not known, but the total deaths are known. Age-specific death rates from a larger (reference) population are applied to corresponding age groups in the smaller population to yield estimated (expected) deaths. The estimated value is compared with the actual deaths observed (ratio is obtained). This is the procedure to obtain Standardised Mortality Ratio (SMR).).

Further, populations may differ in proportion of immigrants/ migrants residing in a location. These two groups are likely to have different travel, exposure and immunity patterns compared to the native population.

In the absence of standardisation, any difference in disease occurrence/ outcome may merely be a reflection of differences in population characteristics, not increased/ decreased vulnerability to disease.

Disease characteristics

It is important to know if the disease agent (SARS-CoV-2 in this case) has more than one strain. The presence of multiple disease strains may make comparisons difficult as some strains may be present in certain locations, but absent in others. There are reports that the strain in India is milder than the strains in Europe and North America. Further, there are reports that the virus has developed more than 10 strains. This does not help make comparisons.

Sometimes, different strains have different virulence- some cause serious disease, while others cause milder illness. In such situations, locations with more virulent strains will report greater deaths than locations with less virulent strains.

Additionally, it is important to know if a person can be simultaneously infected with more than one strain. Generally, those simultaneously infected with more than one strain have more severe disease (as in the case of Dengue).

Diagnostic criteria

Diagnostic criteria offer guidance to doctors on the specific symptoms, signs or test results that indicate the presence of a particular illness. Case definitions are a particular type of diagnostic criteria used to define an illness for disease surveillance or investigations of infectious disease outbreaks. Diagnostic criteria and case definitions work very well when the characteristics of the disease are known with certainty. Where the disease is new/ not understood well, diagnostic criteria are liable to revisions in response to newer/ better understanding of the disease.

The case definition is the basis for determining if a person has disease or not (in an epidemiological survey). Usually, it includes clinical criteria, laboratory findings; temporal and geographical criteria (that is, it specifies time and place for inclusion as a case). In the case of COVID-19, several revisions to case definitions have been warranted on account of variations in clinical presentation (predominantly respiratory features in China vs cardiac features in Italy and the USA); incubation period; infectivity (R0); period of infectivity; duration of infection, etc.

When different countries/ states follow different case definitions, it is possible that a ‘case’ in one region may not be counted as a ‘case’ in another region. This makes it extremely difficult to compare the number of cases- in order to make a valid comparison, the diagnostic criteria/case definitions must match.

Testing/retesting criteria

One of the crucial components in diagnostic criteria/ case definition is the laboratory test result. Given the variety of manifestations reported for COVID-19, a positive test result is the only way to confirm the diagnosis. However, there is considerable variation in testing criteria. While some states/ countries are testing a large proportion of the population, including asymptomatic persons (Iceland, South Korea), others have adopted a policy of selective testing. One cannot compare the case load of a country practicing mass testing with a country following selective testing.

Similarly, different countries have adopted different policies regarding retesting. Retesting is required to determine if a suspected case/ contact who was previously negative has become positive (and vice versa). This information is required to decide on release from quarantine/ isolation. Both the timing and number of repeat tests matter. Some asymptomatic individuals are testing positive on the third or fourth test; or after >28 days’ quarantine. States/countries that perform more tests will naturally have more cases. By corollary, states/ nations having a lot of infected persons and not testing, will have fewer cases (even though many are infected).

Where the total number of tests performed is a multiple of the number of individuals who were tested (that is, more than one test was performed per person), the total number of test positives cannot be taken to indicate the total number of cases.

Number of tests performed

The absolute number of tests performed is not as useful as the number of tests per unit population. In a country like India, the total number of tests may seem impressive, but is not impressive when expressed in terms of population. Again, comparing absolute number of tests (and the proportion of positive results of total tests conducted) between states with vastly different testing ratios does not make sense.

Test employed

With international pressure to develop rapid tests, there are several rapid test kits on the market now. Unfortunately, many of these tests have not been subjected to the usual rigorous testing process required under ordinary circumstances.

While some states/ nations are using rapid tests with uncertain sensitivity and specificity, others are using RT-PCR to diagnose COVID-19. The choice of test has a direct bearing on the validity of the test result.

Thanks to the confusion created by the sudden profusion of tests, there are now several reports of individuals having conflicting test results. Naturally, this limits the ability to make comparisons of ‘cases’.

Stage of pandemic

Different countries/ states are in different stages of the pandemic. Treating all countries as a homogenous group is inappropriate. One cannot make valid comparisons between a country that is in Stage 1 and another country that is in Stage 3 (community transmission).

Cause of death criteria

There is little agreement on cause of death criteria- if a person with known cardiovascular disease is diagnosed to be COVID-19 positive, and dies due to a myocardial infarction [MI] (heart attack), will his/her death be recorded as a COVID-19 death, or an MI death (since there are reports of cardiovascular problems arising due to COVID-19)?

In several regions, testing is limited although many people fit the general profile of COVID-19. If such a person dies due to any cause, it is unlikely to be recorded as a COVID-19 death because there is no evidence that s/he had SARS-CoV-2 infection. This will artificially reduce the number of COVID-19 deaths.

Due to these variations, direct comparison of COVID-19 deaths is not appropriate.

Immunity

Although it was initially assumed that infection with SARS-CoV-2 would result in immunity against it, the World Health Organization (WHO) has recently stated that the possibility of reinfection cannot be ruled out in those who previously recovered from the disease.

If so, this creates a problem- the infection may keep circulating within the population, with people getting re-infected from time to time. This would ensure a large number of cases at any given time. At present, the proportion of re-infected cases is unknown (since lifelong immunity after infection was assumed). It is possible that the factors responsible for re-infection vary from population to population, further confounding matters.

Criteria for ‘cure’/’recovery’

Those who are declared ‘recovered’ are excluded from counts of cases. However, the criteria for declaring ‘recovery’ are heterogeneous. In light of the possibility of re-infection, these criteria are likely to undergo further revisions.

There are reports of persons who made a clinical recovery, but continued to test positive. It is pertinent to note that absence of evidence of infection in one site does not exclude the possibility of the virus being present at another site in the body. There is a report of infection clearing from one site and infecting another site in the same individual. If this is generally true of COVID-19 patients, and only one type of test specimen is collected (throat swab, for instance), one is likely to wrongly determine that a person has recovered (the infection would have simply moved to another site/ organ). Anyone making comparisons would have to consider such misclassification errors.

Useful Links:

Link to WHO scientific brief on issuing “immunity passports”:

https://www.who.int/news-room/commentaries/detail/immunity-passports-in-the-context-of-covid-19

Link to WHO page on COVID-19:

https://www.who.int/emergencies/diseases/novel-coronavirus-2019

Image credit:

National Institute of Allergy and Infectious Diseases (NIAID) / CC BY (https://creativecommons.org/licenses/by/2.0)