Dealing with randomisation bias in a social experiment: The case of ERA (EALE presentation)

Sianesi, Barbara (2012) Dealing with randomisation bias in a social experiment: The case of ERA (EALE presentation). In: 24th annual Conference of the European Association of Labour Economists, 20 to 22 September 2012, Bonn. (Unpublished)

[img]
Preview
PDF - Presentation
159kB

Abstract

The UK Employment Retention and Advancement (ERA) programme has been evaluated by a largescale randomised experiment. It has however emerged that due to the experimental set-up over one quarter of the eligible population was not represented in the experiment: some eligibles actively refused to be randomly assigned, while some were somehow not even offered the possibility to participate in random assignment and hence in ERA. The fact that ERA was a study and involved random assignment has significantly altered how the intake as a whole was handled, as well as the Page 114 nature of the adviser/individual interaction in a way that would not have been the case had ERA been normal policy. The pool of participants has been both reduced and altered, which is likely to have led to some randomisation bias or, alternatively, to some loss in external validity in the experimental estimate for the effect on the eligible population. The beauty of the ERA set-up and data is that it offers the rare chance to formally measure the extent of randomisation bias or the loss in external validity. Specifically, the key objective of the paper is to quantify the impact that the full ERA eligible population would have been likely to experience had they been offered the chance to participate in ERA, and to assess how this impact for the full eligible group relates to the experimental impact estimated on the potentially self-selected and advisor-selected subgroup of study participants. We separately consider how to deal with non-participation when follow-up information on the outcomes of the nonparticipants is available (administrative data) or not available (survey data such as earnings). Nonresponse to the survey and/or to the earnings question among survey respondents can create additional issues when trying to recover the earnings effect of ERA for the full eligible population.

Item Type:Conference or Workshop Item (Paper)
Uncontrolled Keywords:Programme evaluation Randomization bias Social experiments
Subjects:1. Frameworks for Research and Research Designs > 1.11 Evaluation Research
ID Code:2904
Deposited By: PEPA User
Deposited On:04 Oct 2012 12:56
Last Modified:04 Oct 2012 12:56

Repository Staff Only: item control page