Dealing with randomisation bias in a social experiment: The case of ERA

Sianesi, Barbara (2014) Dealing with randomisation bias in a social experiment: The case of ERA. Technical Report. Institute for Fiscal Studies.

PDF (IFS working paper) - Submitted Version


Abstract: One of the most powerful critiques of the use of randomised experiments in the social sciences is the possibility that individuals might react to the randomisation itself, thereby rendering the causal inference from the experiment irrelevant for policy purposes. In this paper we set out a theoretical framework for the systematic consideration of “randomisation bias”, and provide what is to our knowledge the first empirical evidence on this form of bias in an actual social experiment, the UK Employment Retention and Advancement (ERA) study. Specifically, we empirically test the extent to which random assignment has affected the process of participation in the ERA study. We further propose a non-experimental way of assessing the extent to which the treatment effects stemming from the experimental sample are representative of the impacts that would have been experienced by the population who would have been exposed to the program in routine mode. We consider both the case of administrative outcome measures available for the entire relevant sample and of survey-based outcome measures. For the case of survey outcomes we extend our estimators to also account for selective non-response based on observed characteristics. Both for the case of administrative and survey data we further extend our proposed estimators to deal with the nonlinear case of binary outcomes.

Item Type:Working Paper (Technical Report)
Subjects:1. Frameworks for Research and Research Designs > 1.11 Evaluation Research
5. Quantitative Data Handling and Data Analysis > 5.4 Microdata Methods > 5.4.1 Data linkage
ID Code:3345
Deposited By: PEPA User
Deposited On:21 May 2014 15:06
Last Modified:21 May 2014 15:06

Repository Staff Only: item control page