Using pupils’ grade obtained in national examinations as an outcome measure in evaluations : some considerations for the design of randomised controlled trials

Smith, Ben, Morris, Stephen P. ORCID: https://orcid.org/0000-0001-6869-8933 and Armitage, Harry (2022) Using pupils’ grade obtained in national examinations as an outcome measure in evaluations : some considerations for the design of randomised controlled trials. Research Papers in Education. ISSN 0267-1522

Preview

Accepted Version Available under License Creative Commons Attribution Non-commercial. Download (437kB) | Preview

Abstract

It is not uncommon for randomised trials in education to have the performance of sample members in national examinations as their primary outcome. In many cases examination results are available as summary measures only. Taking the example of GCSE examination results in England, this paper shows that using summary measures of an underlying score or mark, such as exam grade, complicates the design of trials and can lead to under-powered studies. Simple simulations are used to explore the consequences of powering trials to detect a difference assuming grade or summary measures are the only outcome metric available, where the effects of an intervention is primarily captured in the unknown mark or score distribution. The analysis draws on data that relate the entire distribution of marks in English language and mathematics examinations to grades. Recommendations are made in order to address this problem.