In the dark days of 2020, there was a strong belief that every single person in the UK should be supplementing with vitamin D.
There were theoretical reasons to think that vitamin D might be beneficial in COVID.
Instead of going forward with something that we didn’t know the effect of, a proper study was set up. Mass testing reduced death and disabilities by about 25% from severe COVID, reducing COVID by about a quarter.
The idea that little bits of evidence can lead you in the wrong direction if you aren’t doing the right definitive study is absolutely well-established in that field.
Lockdowns are another example. So, it suddenly became the standard of care, not only in the UK but across the world.
How do we avoid causing harm? How might we accelerate doing good?
The example there is the ability to get dexamethasone out quickly because we had an answer.
And that is the example of mass testing. And importantly, how do we also use evaluation to reduce uncertainty for the future?
We need to get an adequate view of what we really know and don’t know, and evidence synthesis is an established methodology.
People need to be able to get the outcome of this and see the outcome so they can both adopt it and challenge it.
It needs to be inclusive because, very often, these questions are not sitting in a single sphere.
What is the outcome that matters and to whom does that matter? When thinking about outcomes, which groups or populations does the outcome matter to?
If you think about the mass testing example, it mattered to the population-level outcome, but it mattered an awful lot to small groups and individuals as well.
There’s also the question of how definitive you need your answer to be – how definitive do you need it to be?
When you look at how results come about, you’ve got to really build into the fact that your intervention itself is changing what you’re looking at.
Bias is a big problem. Your bias as a designer or interpreter and the bias that’s inherent because people start to do things differently, and they may do things differently depending on how you set things up.
Net Zero – we’ve got the most massive societal challenge around Net Zero, with many, many moving parts that are interrelated.
If we don’t evaluate that as we go along, we will end up making big mistakes and they will have knock-on consequences for other areas. Evaluation is going to be important.
One example of an evaluation tool, which is important, is the Office for National Statistics who have now set up a brilliant dashboard to look at all the outcomes for Net Zero.
If we get this right this will change the way we think about policy development, but it has to be done in a way that’s iterative, agile, and appropriately targeted towards the outcomes.
We do have some international comparisons telling us that evaluation does make a big difference in terms of the outcome you’re trying to get.