Evidence – Generation, Compilation, Evaluation and Dissemination in the Next Assembly Term

In March PPIW celebrated its 2nd Anniversary. I was pleased to be asked to speak at the event, alongside a panel of fellow experts: Dr Victoria Winckler; Professor Chris Taylor; and Professor Laura McAllister.

The question PPIW posed to us was: “What do you see as the incoming Welsh Government’s evidence needs and how would you advise Ministers to improve the use of evidence in Wales?”

I’d like to try to answer these questions by thinking of evidence as something which needs to be generated, brought together and, crucially, then acted on.


Professor Jonathan Shepherd of Cardiff University, who spoke on crime reduction.

Professor Jonathan Shepherd of Cardiff University, who spoke on crime reduction.

The production (generation) of evidence is a priority

Ministers should ensure that public services, service by service, are constantly infused with reliable new evidence on what works.  An overview is needed so that evidence gaps can be identified. For example, production of evidence on what works in education in Wales is low. The Education Endowment Foundation (EEF) contributes hugely in England thanks to a DfE grant – over a hundred trials carried out in hundreds of schools and involving hundreds of thousands of children. We should use this evidence here of course but trials of education innovations are still needed here – with regard to Welsh language education for example.

Generating evidence relevant to non-devolved services should be welcomed. The evidence produced by the pioneering Police Science Institute in Wales has already helped transform UK policing. Similar opportunities to contribute to the whole UK and globally in this way should be sought and capitalised on. For example for the rehabilitation of offenders.

We now have the still embryonic Public Services Innovation Lab (Y Lab), funded by Cardiff University and Nesta. The work of this new team should be nurtured. It’s potentially UK leading. Evidence production would be dramatically accelerated if the Y Lab training team equips public service analysts across Wales to do policy trials themselves. The EEF, Google and the Behavioural Insights Team have all demonstrated that trials can be done on an industrial scale given the right expertise. Controlled trials can be done rapidly.

The Social Sciences Research Park at Cardiff University also needs to be nurtured. This is designed to bring together evaluators and public services so that real service problems drive the research agenda and evidence from trials rapidly permeates services.

Welsh Government needs to increase demand for evidence. Evidence doesn’t flow on its own. The OMB in the US does this, as does the Treasury, recognising that evidence light policy wastes tax payers’ money, and often does more harm than good. The Welsh Government department responsible for the economy needs to do the same in my view.

We have the PPIW, NICE and seven new What Works centres, NICE style institutions which make recommendations based on pooled international evidence.

So evidence synthesis is probably relatively well taken care of compared with evidence production and use.


The use of evidence

Evidence is only as good as its use! This is a real priority. As the Permanent Secretary memorably said, we need to move on from the era when evidence was something we just sprinkled on policy.

What can ministers do here? Here are four suggestions:

  1. Appoint an evidence champion to every Local Service Board. These should be academics with specific evidence skills (we don’t need more social commentators) and those with links to PPiW and wider evidence networks. A generic job description is needed here.
  2. Insist that public services audit practice against evidence informed guidance to make sure it’s being adhered to. I’ve been surprised in my own Health Board, how very infrequently my service group is asked to do this. Regulators should assure themselves that such audits are taking place and the findings acted upon.
  3. Make sure that evidence is quality assured and published. Evidence free advocacy is not enough! It was clear in my research for the Cabinet Office that, to make a difference, evidence needs to be promoted in usable forms. LinkedIn and twitter are widely relied upon by service managers and commissioners as routes to evidence these days. Long governmental and academic reports, littered with caveats and limitations, won’t be read by busy practitioners and managers. Most important, publication of quality assured evidence gives all parties confidence that what they’re being asked to deliver actually works. It also generates prominent headlines.
  4. Get behind and reward efforts to put evidence into practice. The focus on impact in HEFC research assessment is very welcome. It encourages academics, evidence producers, to tackle real problems and to put their campaigning hats on when they publish new evidence.


From small beginnings, ministers can scale up new evidence informed policy

Some years ago my research group discovered that using information collected in A&E for violence prevention was effective. Policing and CCTV, for example, could be deployed more effectively with this unique anonymised information. In Cardiff, whether measured from police records or hospital admissions, violence fell 42% more than in other similar cities.

The then Community Safety Director for Wales, Gillian Baranski, picked this up and in the mid-2000s ran training workshops for all the relevant professionals in Wales: A&E consultants, police commanders, crime analysts and so on. But implementation across Wales has slipped since. There’s too little reliance on this information now. Violence isn’t being prevented as much as it could be.

In England, where this approach was not invented, the 2010 coalition government decided, a few years later, that this Welsh innovation should be national policy there. It set up a national implementation task force which reports to an inter-ministerial group. An information standard was published so that every NHS Trust with an A&E collects data on violence location, weapon, time and date.

The collection of these data became mandatory under the standard NHS contract. The Department of Health commissioned successive audits to check on implementation. Through a communication strategy, the profile of this approach was raised to the extent that the London mayor, Boris Johnson, was asked in his weekly question time why many hospitals in London weren’t complying. He initiated an Information Sharing Summit. His team successfully applied for a seven figure Home Office grant to help fix the problems.

Violence Reduction Nurses were appointed in all the English regions to drive this agenda. All this came about through Ministers investing in a nationwide, nuts and bolts, implementation strategy.

The good news though, is that the software companies which supply A&Es will include the basic violence prevention data in their products. Furthermore, the Ask and Act policy in the new Welsh law on domestic violence may help ensure that it’s collected and acted upon. The new law, like any new policy, needs to be complied with though. As with all evidence adoption; an organised, sustained campaign is needed.

About the author: Jonathan Shepherd is a Professor of Oral and Maxillofacial Surgery and Director of the Violence Research Group, Cardiff University. His research on clinical decisions and community violence has made many contributions to clinical and public policy and legislation. He initiated and developed the Universities’ Police Science Institute in Wales, the information sharing model for violence prevention which was adopted in the 2008 UK violence reduction strategy and by the Coalition Government in 2010, and a comprehensive care pathway for people harmed by violence.


Comments are closed.