Dealing with randomisation bias in a social experiment: The case of ERA (EALE presentation)

The UK Employment Retention and Advancement (ERA) programme has been evaluated by a largescale
randomised experiment. It has however emerged that due to the experimental set-up over one
quarter of the eligible population was not represented in the experiment: some eligibles actively
refused to be randomly assigned, while some were somehow not even offered the possibility to
participate in random assignment and hence in ERA. The fact that ERA was a study and involved
random assignment has significantly altered how the intake as a whole was handled, as well as the
Page 114
nature of the adviser/individual interaction in a way that would not have been the case had ERA been
normal policy. The pool of participants has been both reduced and altered, which is likely to have led to
some randomisation bias or, alternatively, to some loss in external validity in the experimental
estimate for the effect on the eligible population. The beauty of the ERA set-up and data is that it offers
the rare chance to formally measure the extent of randomisation bias or the loss in external validity.
Specifically, the key objective of the paper is to quantify the impact that the full ERA eligible population
would have been likely to experience had they been offered the chance to participate in ERA, and to
assess how this impact for the full eligible group relates to the experimental impact estimated on the
potentially self-selected and advisor-selected subgroup of study participants. We separately consider how to deal with non-participation when follow-up information on the outcomes of the nonparticipants is available (administrative data) or not available (survey data such as earnings). Nonresponse to the survey and/or to the earnings question among survey respondents can create additional issues when trying to recover the earnings effect of ERA for the full eligible population.