✖

How to calculate the effectiveness of vaccinations?

How is it possible that more vaccinated people can die from a given disease than unvaccinated people? It sounds paradoxical, it even sounds a bit like vaccination doesn't work - indeed, it harms - but let's show that this may not be true.

Imagine that we have a city of 100,000 inhabitants and that the evil and insidious disease Provid is running rampant in the city. However, scientists and researchers have managed to create a vaccine that protects against this disease, and they have inoculated some of the inhabitants with this vaccine. Yet, when we look at the data, a thousand vaccinated people and a thousand unvaccinated people died from Provid. The population is confused and feels that vaccination does not work when as many vaccinated people die as unvaccinated people. So does the vaccine work or not work?

Hard to say 🤷 just based on the data we have so far. Because it's not enough for us to know how many vaccinated and unvaccinated people died from Provid. We also need to know how many people actually got vaccinated. Because I could ask you: which is more dangerous? Driving a car or climbing Mount Everest? The death toll is clear: hundreds of people have died trying to climb Everest. Over a million people are killed in a car. Every year. So clearly it's safer to try to climb Everest than to climb in a moving coffin.

A lightbulb must be going off in everyone's head right now - it's actually not safer to climb Everest than to drive a car. The numbers mislead us because we haven't taken into account that while thousands of climbers have climbed Everest, billions of people ride in cars. Similarly, we need to take into account how many people are vaccinated in our imaginary city. Let's try to break this down with a few examples:

  • What if we've only vaccinated 1,000 people in the city and 99,000 people are unvaccinated? That would mean that everyone who was vaccinated died. While only 1,000 of the 99,000 unvaccinated died. So the vaccine would not only not help, but harm!
  • What if it was the other way around and there were 99,000 vaccinated in the city and only 1,000 unvaccinated? So every unvaccinated person would die, while only 1,000 of the 99 would die. The vaccine may not be 100%, but it visibly increases the chance of survival by an order of magnitude.
  • What if half the city was vaccinated and the other half wasn't? Then that would mean that one thousand of the 50,000 vaccinated died and one thousand of the 50,000 unvaccinated died. In both groups, two percent of the population died - meaning that the vaccine had no effect, positive or negative.

What is vaccine effectiveness

Okay, so how do we calculate vaccine effectiveness? We'll say that vaccine effectiveness is the proportion of people saved from death by the vaccine. For an example, let's say a thousand unvaccinated people died from a disease, yet in an alternate reality where we vaccinated those same thousand people, only 300 died and 700 survived. In that case, the vaccine saved 700 people out of 1000, the effectiveness of the vaccine would be 70%. If no one died, the effectiveness would be 100%, if everyone died, the effectiveness would be 0%.

Okay, but we don't have two parallel universes in which we would vaccinate someone once and not vaccinate the same person in the other. So we have to calculate efficacy differently. Let's go back to our town of 100,000 people. Let's add the missing information - let's say we've vaccinated 90,000 people. That's data we can collect in the real world. What's the vaccine efficacy? First, let's calculate the proportion of deaths in each group, i.e. what percentage of vaccinated and unvaccinated died:

  • One in 10,000 unvaccinated died, which is 10%. Let's mark this as NL = 10 % (unvaccinated people).
  • One thousand of 90 thousand vaccinated people died, which is about 1.11%. Let's mark that as OL = 1,11 % (vaccinated people).

We see that 1.11% of the vaccinated died, while 10% of the unvaccinated died. We calculate the resulting effectiveness of U using the formula

$$U=\frac{NL-OL}{NL}\cdot100\%$$

After inoculation, we have

$$U=\frac{10-1{,}11}{10}\cdot100\%=\frac{8{,}89}{10}\cdot100\%=0{,}889\cdot100\%=88{,}9\%.$$

The effectiveness of our vaccine is equal to 88.9%. Thus, the vaccine saved 88.9% of the people who would otherwise have died. We can also do the math: in the unvaccinated group, 10% of the people died. If we had not also vaccinated the 90 000 people, the same 10% of them would have died, which is 9000 people. Because the vaccine is 88.9% effective, 88.9% of those 9000 people ended up surviving because of the vaccine, so 88.9% of 9000 is just 8000, and that leaves us with 11.1% of people who unfortunately still died - that's the thousand dead from the assignment.

We can see that this vaccine gives us a much higher chance of survival, but we can't say that after vaccination we are 100% sure that we won't die from the disease.

The more vaccinated, the more dead vaccinated.

Anyway, this leads to an interesting paradox. Let's imagine that with our 88.9% effective vaccine we vaccinate another 5000 people in the city, i.e. we get to 95,000 vaccinated and only 1,000 unvaccinated. Then it will be true that

  • Of the 5,000 unvaccinated, 10% will die.
  • Of the 95,000 vaccinated, 1.11% will die, roughly 1,055.

‼️‼️‼️ SHOCK ‼️‼️‼️ The vaccinated die far more than the unvaccinated. But this is purely because there are orders of magnitude more vaccinated than unvaccinated. You could also say that while there are 95 times more vaccinated people in the city than unvaccinated, we only have twice as many vaccinated deaths as unvaccinated deaths.

And most importantly, the total number of deaths has gone down. While we now have 500 + 1055 = 1555 deaths, in the previous example we had 1000 + 1000 = 2000 deaths.

Different vaccine efficacies

Let's look at some interesting efficacy values:

  • We would get 100% efficacy if OL were equal to zero (none of the vaccinated died) and NL were equal to 100% (all the dead are unvaccinated). The vaccine efficacy cannot be greater than 100%.
  • We would get 0% effectiveness if NL = OL (if the percentage of vaccinated people dying is the same as the percentage of unvaccinated people). This would mean that the vaccine has no effect.
  • We would get negative efficacy if OL were greater NL, i.e. if more vaccinated died than unvaccinated. In that case, the vaccine would be harmful.

You can imagine a vaccine with an effectiveness of about 83% by saying that if you got sick and were unlucky enough to be the one to get the disease, the vaccine would still give you additional protection - you could roll the dice and if you roll a one to five the vaccine would save you, if you roll a six the disease would still kill you.

Watch out for different groups of people

Note that when calculating the effectiveness of vaccination, it's not a good idea to just blindly follow the formula I've given above. To apply it, it is good to know how the disease behaves and what our vaccination strategy is. Diseases are different, they attack different groups of people, and likewise different groups of people will have different vaccination rates. If a disease only attacks women, we may not care what the vaccination coverage of men is. If a disease attacks the older age groups, we may not care what the vaccination coverage of the younger age groups is, and it will be better to calculate the effectiveness with knowledge of the vaccination coverage of the older age groups.

Similarly, we can measure different types of protection for vaccines. Throughout the paper we have written morbidly straight about death, but with vaccines we can also measure the effectiveness of how the vaccine reduces (or increases...) the chance of getting sick, transmitting it further, or perhaps requiring hospitalization.