Data can’t tell us everything

Bridget Rosewell.

Facts and data are important in making decisions but so is judgement, says Bridget Rosewell.

We all want evidence based policy and decisions.  However, deciding what constitutes evidence and what standard of proof is required is another matter.  And of course this is especially challenging when we need to explore what will happen, rather than what has happened.  Will this policy work?  Will this investment pay back?  It’s not clear that evidence is of any use in making these decisions at all.

Notoriously, the Jubilee Line Extension failed to pass the value for money tests at the time, which were overruled by the prime minister at the time.  However, its use in practice, even after overruns of cost, show that it would now pass the test.  In other words, forecasts of passenger use and demand were too low.  At the same time, studies have shown that projects of all kinds suffer cost over-runs which undermine benefit cost ratios.

Unconscious bias, poor statistical analysis and pushing for strong results can all result in reaching conclusions which do not later stand up.  None involve deliberate falsification but rather that to err is human.

The National Infrastructure Commission has been asked to provide an assessment of infrastructure needs across all classes of infrastructure looking out 30 years.  Thinking about such needs means thinking about a future in which very little is held constant – everything can change.  Even on a nearer time frame, thinking about the development of the corridor from Oxford to Milton Keynes and to Cambridge, technologies, housing, employment patterns and institutions can all show potentially dramatic shifts.  In such circumstances, how should we think about the nature of evidence if everything in the future can be different from the patterns of the past?

A good approach in my view is to assess objectives and plausibility, with a strong dose of transparency.

The vision for a project or programme sets out what it is intended to achieve.  Evaluation can only sensibly occur in relation to objectives.  HS2 was first presented as having the objective of saving time – in the first instance to Birmingham.  It was then castigated as being a very expensive way of saving 20 minutes on such a trip.  This accusation still surfaces and if indeed this had been the objective the detractors would be right.  However, a more accurate objective, later clarified, is to improve connectivity to northern cities and the capacity of the rail network more generally.  Thus any evaluation should be set against these objectives and how they might be achieved.  

If the objective of an investment is to sell more widgets, this might be easier to describe than if the objective is to improve access to markets for northern cities.  Objectives, however, remain objectives.  Moreover, setting them out clearly also allows the context and assumptions to be set out clearly too.

What does it take to reach the objective?  Are such changes plausible in either the current state of the world or the possible futures I am assuming?  How different is this future and how do its changes compare with past changes?

If I want to double my sales of widgets, it would be helpful to know if this has been done before, or what market share this might imply for my new factory.  These questions would be standard in a self-respecting market study and investment appraisal.  Somehow they don’t always get asked in a policy study.

If the aim of an investment is to create jobs, what scale of job creation is implied and is this scale unprecedented?  In producing long term forecasts for the London economy for the GLA, we looked back the longest possible period to give the most context possible for future growth.  Such an analysis showed that the fall in London’s employment over the period from the 1970s had been concentrated in manufacturing but these losses had now ended and in the meantime growth in services had continued at a fairly constant rate.  The question became one of asking what it would take to change a trend which had been in place for nearly thirty years.  In the absence of such a shift, London’s employment could be expected to grow and this long term view has indeed been justified over the past fifteen years in spite of cyclical variation.

Of course it remains debatable whether a long term trend will continue.  Once it is understood what a particular view of the future depends on, it becomes possible to have such a debate in an informed way.

Looking at plausibility definitely requires the analysis of scenarios to consider the range of contexts in which an investment succeeds and those in which it could fail.  

The models beloved of technocrats can be very powerful.  But the adage of ‘garbage in, garbage out’ remains.  Strong assumptions and weak data are not a good basis for decision making.  Being able to set out clearly the assumptions so that they can be understood by non-specialists is crucial to a good and informed debate about conclusions.

A narrative about the objectives of a policy or investment, how these are expected to be achieved and the plausibility of these mechanisms is essential for a transparent understanding.  

In making an assessment, we are taking bets on the future.  Futures are uncertain, so setting out a range of possible futures is one step.  Then analysing which ones are supported by the investment in question, which ones would be undermined, and which would invalidate the project can then be described.

Facts matter. Reliable data about the past matter.  These are what we have to inform us about possible futures.  However, they don’t tell us what that future will actually be since this rests on our decisions now.  The consideration of possible futures, their degree of likelihood and on what assumptions they depend, rests on fact and data.  But it is not controlled by them.  That also requires judgement.

Bridget Rosewell is a senior adviser at Volterra Partners and a member of the National Infrastructure Commission. 


Bridget's most important point is about "possible futures, their degree of likelihood and on what assumptions they depend". Using different models based on quite different theories, and testing alternative scenarios of what the future might look like (economically, technically, societally) we can inform the judgment she concludes with and also provide the transparency around the ultimate decision made. For example, under which future beliefs is this particular policy still a good choice? What needs to happen in future to make this infrastructure investment viable? How important are dissimilar but plausible theoretical assumptions on the outcome? What would be the least regret choice or the choice that enables adjustment to future change? We need to use modelling in a quite different manner than we have done in the past - rather than seeking a single version of the truth, exploring and valuing the many alternative truths that models allow us to investigate.