Pupil Advantage Index as an Alternative to Subgroup Analysis in RCTs for Education

Share Download as iCal file
Click the icons above to Share, Tweet or add this event to your calendar (iCal)
RSS local group meetings

Wednesday 13 February 2019, 02:00pm

Location Rolle 211, University of Plymouth

Speaker: ZhiMin Xiao (University of Exeter)

Abstract: Analyses of social interventions need to produce evidence that is relevant to different groups of people in a society. When such a group is not the target group of an intervention, this is called subgroup analysis, even when the group of interest is pre-specified prior to data collection. Amongst statisticians, subgroup analysis is often regarded as a statistical malpractice, as its findings are often underpowered, unreliable, and can be prone to overinterpretation at best, or misleading at worst. Meanwhile, researchers would be criticised for generating irrelevant evidence and accused of wasting research money if they do not conduct relevant subgroup analysis. As a result, “they are damned if they do, and damned if they don’t” (Petticrew et al., 2012). 

In this study, we estimated intervention effects for Free School Meal (FSM) pupils in English schools, which is a pre-specified subgroup in most educational interventions funded by the Education Endowment Foundation (EEF) in England. Specifically, we first ran a treatment-FSM interaction test in each and every outcome to see if the difference-in-effects is statistically significant between FSM and Non-FSM pupils. We then calculated separate effect sizes within the two subgroups. Finally, we examined the p-values from the interaction tests and compared the overall effect sizes for both FSM and Non-FSM pupils with the two separate subgroup estimates. We found that conventional interaction tests can produce self-contradictory results. To help solve the problem, we propose a new approach, Pupil Advantage Index (PAI), as an alternative to subgroup analysis and apply it to real RCT data extracted from the EEF data archive. We demonstrate that PAI does not just indicate where an intervention worked and by how much in existing trials, but it can also be utilised to optimise treatment recommendation for future interventions.

Organiser Name Yinghui Wei

Email Address This email address is being protected from spambots. You need JavaScript enabled to view it.

Organising Group(s) RSS South West Local Group





Join the RSS

Join the RSS

Become part of an organisation which works to advance statistics and support statisticians

Copyright 2019 Royal Statistical Society. All Rights Reserved.
12 Errol Street, London, EC1Y 8LX. UK registered charity in England and Wales. No.306096

Twitter Facebook YouTube RSS feed RSS feed RSS newsletter

We use cookies to understand how you use our site and to improve your experience. By continuing to use our site, you accept our use of cookies and Terms of Use.