a year of bungling and confusion

Written by Oz Flanagan on . Posted in Features-OLD

When trying to describe the project, beleaguered seems the most illustrative word. Its positive fundamental aims have been overshadowed by questions that stretch into far wider questions of individual privacy, data ownership and trust in the NHS.

In trying to unravel this, we first need to separate’s potential from its risks and poor public information campaign. Firstly, the prospective benefits that holds for improving healthcare. Currently, any health data compiled by the NHS only exists individually. So for example, Hospital Episode Statistics (HES) hold information about patient time spent in hospital.

But there is no other facility for seeing what happened to those patients before they entered hospital or what occurred when they left. That kind of information is held by local GP practices or other health clinics that see the patient outside hospital.

If datasets like this could be linked, statisticians and researchers could discover all kinds of new ways to improve health provision across the whole NHS. Being able to track the patient journey could reveal medical care improvements for a variety of health problems - big and small.

So what went wrong?

In January 2014, NHS England circulated a leaflet entitled ‘Better information means better care’ to households. The leaflet tersely set out the ‘benefits’ of and then explained the ‘choice’ patients had in opting-out of the project.

Unfortunately, NHS England sent out the leaflet as a blanket distribution. This meant anyone who told Royal Mail to not deliver unaddressed mail, did not get a leaflet. This was the primary patient communication issued on before the project would begin in earnest.

The Medical Protection Society commissioned a YouGov poll to see what kind of reach NHS England achieved, 67% of more than 1,400 respondents said they had not received the leaflet and 45% of respondents who had read or heard about the system did not understand it.

Over the next month, patients and journalists began to take an interest in and started asking questions to find out about the ‘approved researchers’ who would access the data and the ‘strict rules that are in place to protect your privacy’ described in the leaflet. NHS England proved to be woefully unprepared for level of scrutiny that followed.

As national newspapers ran troubling stories about the impact of, the situation was made worse when it emerged that Hospital Episode Statistics had been sold to the insurance industry. Although this story was unrelated to, the damage was done by association.

A lack of information on who would access the data and how anonymisation works torpedoed the rollout of the project. As Ben Goldacre wrote at the time, ‘When you're proposing to share our most private medical records, vague promises and an imaginary regulatory framework are not reassuring.’

NHS England then postponed and promised to initiate a ‘listening’ exercise, supposed to last six months.

The rehabilitation of

The postponement of brought a collective sigh of disappointment from many statisticians. NHS England’s cavalier public engagement strategy and the media’s response led to a blockage on new research applications for access to most public health data. Over the next few months, NHS England needed to clarify all the questions raised in those first two months of 2014.

In March, the government added amendments to the Care Bill that sought to deal with some of’s problems. These were:

  • Restating the duty of the Health and Social Care Information Centre (HSCIC) to protect the privacy of patients.
  • The Confidentiality Advisory Group would be on a statutory footing.
  • HSCIC could only grant access to data in pursuit of the ‘provision of health care or adult social care, or the promotion of health.’

That last point has been criticised for being far too broad in its language. When the Bill reached the House of Lords in May, Lord Turnburg proposed an amendment that would limit secondary use of this data to ‘biomedical and health research’. Lord Owen also proposed setting up a statutory oversight panel to offer independent advice on patient data across the NHS. Both proposals were defeated.

In June, The British Medical Association (BMA) held a vote on whether to support an opt-in approach to patient involvement in This would mean patients would have to physically make the effort to opt-in to the data linkage for their data to become available.

The RSS wrote to the BMA before the vote to warn about the damage such an approach would do to the statistical utility of If the project was opt-in only, the data would be far less representative of the general public and miss out groups of the population with substantial healthcare needs. Despite this, the BMA voted to advocate the opt-in approach.

Also in June, Sir Nick Partridge delivered a report that reviewed data released by the NHS Information Centre (the forerunner to HSCIC). The Partridge review showed up major lapses in previous data releases, including the sale of hospital statistics to the insurance industry. One positive recommendation was for a data register to be set up which HSCIC have implemented. This will offer some transparency to track who data releases are given to by HSCIC.

By the time six months had passed since the project was paused, confusion still surrounded the whole issue. Finally in October, news from NHS England revealed their next step. The new plan was to proceed with on a pilot basis, which meant that hundreds of GP practices in four Clinical Commissioning Group (CCG) areas would be involved in a ‘pathfinder stage’ of the roll out. The CCG areas in this pilot stage are Leeds (3 CCGs: West / North / South and East), Blackburn with Darwen CCG, West Hampshire CCG and Somerset CCG.

One aspect that is being tested in these areas is the communication of’s objectives to patients. NHS England had clearly identified the original mass mailshot at the start of the year to be a major obstacle. So in their words, the communications to be tested ’will include an individually addressed letter sent directly to every individual or household from their pathfinder GP surgery, a leaflet and other explanatory materials, as well as emails and texts where the surgery also uses these channels.’

The most recent and positive development has been the appointment of Dame Fiona Caldicott to the position of national data guardian for health and care. Back in 1997, Caldicott chaired a major report on the use of patient data in the NHS and the recommendations from the report are still widely followed. 

NHS England’s October announcement specified that would only be accessed at a secure data facility at HSCIC. However, in a speech on improving the NHS’s technology, health secretary Jeremy Hunt’s said, ‘No data will be extracted from GP practice systems - including during the ‘pathfinder’ pilot phase of the programme - until she [Fiona Caldicott] has advised me that she is satisfied with the programme’s proposals and safeguards.’ So while the pilot areas are forging ahead, apart from the communication method trials, it is unclear what else is taking place.

Confusion continues to reign

NHS England faces an uphill task in reassuring ordinary patients on the whole project. Those who have been actively engaged in the debate, such as the BMA and the patient group Patient Concern have advocated an automatic opt-out from the scheme. The campaigning organisation 38 Degrees has now made stopping’s rollout one of its main campaigns. If organisations like these are supporting a plan that would render almost statistically useless, then there is a great deal more to do to in persuading people in favour of data linkage and the production of more beneficial research and statistics.

Campaigners on the issue like Sam Smith of Med Confidential put this down to big announcements being made with little or no explanatory detail behind them. NHS England and HSCIC could be crafting a secure and fit for purpose system behind the scenes, but from the outside the process seems chaotic. Vital details about how data sharing will operate are ill-defined and confusing for both patients and researchers.

How anonymisation will work and how safe it will be is still uncertain. How much identifiable information will be anonymised? This point is crucial to the whole idea of data linkage, as Katie Harron from University College London demonstrated at a recent event at the RSS. Strip out too many identifiers in the data, and linkage between and other data sets becomes less reliable and statistically problematic.

Leave in too many identifiers and re-identification becomes more of a risk if the data falls into the wrong hands. This is the case Ross Anderson, professor of security engineering at Cambridge and others have been making for some time (most recently at this year’s Cathie Marsh lecture.) It’s this fear that makes secure and properly vetted access to so vital.

So who will have access to the data? There will be a data register of who gains access, but the legal wording of who can access the data is far from specific. Those who have an interest in the ‘provision of health care or adult social care, or the promotion of health’ is open to interpretation and could evolve over time. It is thought that this will be defined at a later date - how and when remains sketchy.

Since NHS England halted the project in February, they have held events across the country to listen to patients concerns about their personal data. Although an admirable exercise, in addition to this they should listen to those who have experience in safeguarding sensitive public data. One example is the Office for National Statistics’ Virtual Microdata Laboratory, which has kept personal data secure and still available for research that serves the public good.

Or they could have approached the Administrative Data Research Network (ADRN), to learn about the extensive procedures they have put in place for researchers to access government admin data. The ADRN will also be dealing with sensitive personal information, but they have been clear that access is only granted in secure facilities and any outputs from the analysis will be subject to statistical disclosure control before it goes any further.

In any event, the project rumbles on. Moreover, the NHS is also driving ahead with plans to digitise all patient records across the NHS. Tim Kelsey, national director for patients and information and the public face of, was interviewed this month about the modernisation of NHS record keeping. In summing up the interview, Kelsey stated, ‘I don’t regard as a failed thing - what I see is something we’ve really learnt from in the spirt of open, honest, transparent attempts to improve the quality of patient care in the digital age.’

The results from a survey the RSS commissioned on data trust this year were clear on the public’s view of data sharing. There was strong opposition to ‘health records being sold to private healthcare companies to make money for government’ (5% in favour, 84% against), but far more support for ‘GP health records being shared with academics and scientists for research to improve treatments’ (53% in favour, 26% against).

Done in the right way, a project like this gives statisticians and researchers access to valuable data resources that could benefit us all. Done in the wrong way, the project has the potential to give birth to a scandal that will make any future health or government data sharing scheme a toxic proposition.

Linking health data and fortifying it behind walls that can only be used for truly benefiting the public good does not need to be this difficult.

The government use of personal data from communications is justified from an argument of fear. If GCHQ don’t do what they do, the terrorists will win, so the argument goes. It’s much easier to persuade people on the virtues of data sharing through an argument of hope. If trust can be gained through the demonstration of clear and proper procedures in the public interest, the product will be of universal benefit to the entire nation.

Join the RSS

Join the RSS

Become part of an organisation which works to advance statistics and support statisticians

Copyright 2019 Royal Statistical Society. All Rights Reserved.
12 Errol Street, London, EC1Y 8LX. UK registered charity in England and Wales. No.306096

Twitter Facebook YouTube RSS feed RSS feed RSS newsletter

We use cookies to understand how you use our site and to improve your experience. By continuing to use our site, you accept our use of cookies and Terms of Use.