Fieldwork effort, response rate, and the distribution of survey outcomes: a multi-level meta-analysis

Williams, Joel and Sturgis, Patrick and Brunton-Smith, Ian and Morre, Jamie (2016) Fieldwork effort, response rate, and the distribution of survey outcomes: a multi-level meta-analysis. NCRM Working Paper. NCRM. (Unpublished)

[img] PDF
Available under License Creative Commons Attribution.

1MB

Abstract

We assess how survey outcome distributions change over repeated calls made to addresses in face-to-face household interview surveys. We consider this question for 559 survey variables, drawn from six major face-to-face UK surveys which have different sample designs, cover different topic areas, and achieve response rates which vary between 54% and 76%. Using a multi-level meta-analytic framework, we estimate for each survey variable, the expected difference between the point estimate for a proportion at call n and for the full achieved sample. We find that most variables are surprisingly close to the final achieved sample distribution after only one or two call attempts and before any post-stratification weighting has been applied. The mean expected difference from the final sample proportion across all 559 variables after 1 call is 1.6%, dropping to 0.7% after 3 calls, and to 0.4% after 5 calls. These estimates vary only marginally across the six surveys and the different types of questions examined. Our findings add further weight to the body of evidence which questions the strength of the relationship between response rate and nonresponse bias. In practical terms, our results suggest that making large numbers of calls at sampled addresses and converting ‘soft’ refusals into interviews are not cost-effective means of minimizing survey error.

Item Type:Working Paper (NCRM Working Paper)
Subjects:5. Quantitative Data Handling and Data Analysis > 5.6 Multilevel Modelling
ID Code:3771
Deposited By: NCRM users
Deposited On:22 Jan 2016 15:11
Last Modified:22 Jan 2016 15:11

Repository Staff Only: item control page