If we want social progress, more data must be shared

Written by Dean Machin on . Posted in Features

Administrative data is extremely valuable to improve social mobility and reduce child poverty. Often, it’s the best data: it can be cheaper than ordinary surveys and includes more comprehensive and richer statistics. But while a huge amount of administrative data is collected, most of it is not shared and it is the exception for datasets to be linked. Why? And, what can be done to improve data sharing?

These were the questions I addressed in a report for the Social Mobility and Child Poverty Commission published on March 20. The report – 'Data and public policy: trying to make social progress blindfolded' – focused on the legal and institutional barriers to data sharing and made 19 recommendations to improve matters. It also stated three principles that should underpin government’s approach to data sharing. First, the barriers.

The law

The biggest problem here is complexity. As a Law Commission report stated, there is ‘a widespread lack of knowledge and understanding on the part of staff of public bodies’ about when, to whom, why and what data can be shared. This is far from ideal.


Data sharing is never anyone’s number one priority and it can seem risky. How many civil servants have been sacked for refusing to share data and how many have been disciplined for losing control over data? Beyond this, there can be a determination to protect ‘our’ data. Sometimes government departments refuse to share with other departments ‘even basic data on which policy is made’.

Conflicts of interest

Some non-government bodies control important data and obtain part of their revenue through selling it. UCAS is a good example - it shares less data than it could and fear about the loss of revenue is a key factor.

The social mobility implications of this are profound. Vikki Boliver found that ethnic minority and state school applicants to Russell Group universities have to significantly out perform their respective white and privately educated peers before they are as likely to be offered places. UCAS disputes some of Boliver’s claims but it has not published its research or the data underpinning it.

We need a proper understanding of the problems around access to elite universities. But public policy should only be based on research that can be independently tested and assessed. More data must be shared.

How to improve matters?

Addressing these problems requires a clearer understanding of the benefits of data sharing, safe procedures in which data sharing can occur, some legal changes and, importantly, political leadership.

In addition to three administrative data sharing principles that should guide government policy and future legislative changes, one of the most important recommendations in my report is about the need to re-balance the data sharing debate. The privacy worries around data sharing are clear, the benefits of data sharing are not.

My report recommends that the UK research councils fund research that quantifies the net public benefits of greater data sharing. The revenue obtained by bodies that sell their data, and the costs they would face from facilitating greater data sharing, should be included in these calculations.

A lot of work to improve data sharing is already being undertaken but there remains a long way to go. The government created the Social Mobility Transparency Board (SMTB). This body (which ultimately reports to the deputy prime minister) has had some success but requires greater ministerial leadership.

The SMTB should be given responsibility for leading and co-ordinating the creation of an effective and safe culture of data sharing across government. It should also be responsible for ensuring that non-government bodies understand what is expected of them. There is no reason why the data sharing expectations of bodies like UCAS should be lower than the expectations of government departments.

There is the Administrative Data Research Network, the central purpose of which is to ‘give trusted researchers access to linked, de-identified administrative data in a secure environment.' My report recommends that its funding be made permanent, that it work closely with the SMTB, and that it take a formal role to address the general public’s legitimate worries about data sharing.

There is also a draft data sharing bill on which the Cabinet Office has worked. The bill (which will not be introduced this parliament) was to have a research and statistics section, a fraud, error and debt section, and a tailored public services section.

For social mobility research and policy making purposes the research and statistics section is key – it is also likely to be the least controversial section. Why? Researchers only need to share anonymised (or ‘de-identified’) data. The fraud, error and debt and the tailored public services sections require sharing personal data.

In light of this, my report recommends that any data sharing bill introduced in the next parliament separates the different sections of the bill worked on to date. As a matter of urgency the next government should introduce a stand alone research and statistics data sharing bill.

Finally, the European Union’s draft data protection regulation. As the RSS has noted, it could put an end to many large-scale studies into social conditions and poverty’. My report recommends that our research councils in the UK should work with European research bodies to amend the regulation. So far the noises are good.

Successive government have been determined to improve social mobility and reduce child poverty and they have invested a lot of effort and resources in attempts to do this. But success is not possible without greater data sharing. Hopefully, my report and recommendations will go some way to improve matters.


You can download Dean's report 'Data and public policy: trying to make social progress blindfolded' here.

The views expressed in the Opinion section of StatsLife are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of The Royal Statistical Society.

Data privacy Administrative Data Research Centre (ADRC) Data sharing

Join the RSS

Join the RSS

Become part of an organisation which works to advance statistics and support statisticians

Copyright 2019 Royal Statistical Society. All Rights Reserved.
12 Errol Street, London, EC1Y 8LX. UK registered charity in England and Wales. No.306096

Twitter Facebook YouTube RSS feed RSS feed RSS newsletter

We use cookies to understand how you use our site and to improve your experience. By continuing to use our site, you accept our use of cookies and Terms of Use.