How do modes of data collection affect the measurement of demographic key indicators?

Abstract

Rising survey costs, increasing internet usage, and restrictions during the pandemic have accelerated a shift from face-to-face (CAPI) to web-based interviews (CAWI). This paper analyses the consequences of this shift for the comparability of survey-based measurements of demographic key indicators. We use data from a survey mode experiment, conducted in the Gender and Generation Survey in Germany, Croatia, and Portugal. We calculate mode-specific means and ratios of selected indicators and perform regression analyses to estimate mode-measurement effects. In all countries, we find mode-selection effects. In particular, less educated people are underrepresented in the CAWI mode. Mode-measurement effects vary widely across indicators and country contexts. For sensitive questions, there are less social desirability biases in CAWI. For questions with a high respondent burden, there are less survey satisficing biases in CAPI. Loop questions with a high respondent burden are of particular concern. The mode shift challenges comparability between data collections, in longitudinal analyses and in cross-country comparisons. Data users should use weights for descriptive analyses, control for the mode where possible, and consider mode differences when interpreting results. The unique data source allows insights into mode effects among demographic indicators that were not possible before.

Publication
BIB Working paper

Related