Saturday, February 23, 2013

Mixed mode surveys: where will be in 5 years from now?


Some colleagues in the United Kingdom have started a half-year initiative to discuss the possibilities of conducting web surveys among the general population. Their website can be found here 

One aspect of their discussions focused on whether any web survey among the population should be complemented with another, secondary survey mode. This would for example enable those without Internet access to participate.  Obviously, this means mixing survey modes.

Using two different survey modes to collect survey data, risks introducing extra survey error. Methodologists (me inclusive) have worked hard on getting a grip on the existence of differences in measurement effects between different modes. In order to study these properly, one should first make sure that the sub samples that are interviewed in different survey modes, do not differ just because of differences in selection effects between the two samples. I have written some earlier posts on this issue, see some of the labels in the word-cloud on the right.

I have composed a short presentation on ways in which differences in measurement effects in mixed-mode surveys can be studied. The full presentation is here. Comments are very welcome.

In going over the literature, two things stood out, that I never realised:

1. There are few well-conducted studies on measurement effects in mixed-mode surveys. Those that exist show that there often are difference in means, and sometimes in variances of survey statistics. Yet no one (and I'd love to be corrected here), has looked at the effect on covariances. That is, do relations between the key variables in a study change, just because of the mode of data collection?  There may be an analogy to nonresponse studies, where we often find bias on means and variances, but much smaller biases for covariances. In this picture, this reflects the relation between x1 and y1 in two different survey modes. Is that relation different because of mode effects? Probably not, but we need more research on this.




2. What to do about mode effects? We are perhaps not ready for answering this question, looking at how little we know exactly about how measurement differences between modes affect survey statistics. But we should start thinking in general about this question. Can we correct for differences between modes. Should we want to do that? It would create a huge extra burden on survey researchers to study mode differences in all mixed-mode surveys, and designing correction methods for them. Could it be that in five years time, we have concluded that it is probably best to try to keep mode effects as small as possible and not worry about the rest?

6 comments:

  1. Definitely worth spending some time on the estimation issues up front, or some or even most of the data collection could be wasted.
    For example, if there is an observed difference between modes after weighting for selection effects and you want zero bias (relative to the face-to-face mode) then for national estimates you just give zero weight to the data from the other modes.

    ReplyDelete
  2. dear charlie. Thats an interesting thought, but it would mean that we sacrifice a large part of pur data collection. That may be necessary if we reallty can't resolve measurement differences at all, but it would probably make our funders not too happy.

    ReplyDelete
  3. Agree, so it's necessary to think about these things (and test/pilot) before substantial expenditure is made

    ReplyDelete
  4. Hi Peter,
    I can correct you because I wrote a paper about mode effects on covariances about two years ago and also presented this paper at the 2011 ASA Spring Methodology Conference at Tilburg University. Nonetheless, I did not submit this paper to a journal yet because I used the ESS r4 mixed-mode data for illustration but these data are terrible. I'm currently waiting for the new ESS r6 mixed-mode experiments. Nevertheless, in line of this first paper, I also wrote a paper together with Melanie Revilla about mode effects on quality estimates obtained from MTMM experiments. This starts from mode effects on covariance matrices which are required for MTMM models. This paper is currently under review but you can mail me for a draft version.

    ReplyDelete
    Replies
    1. Hi Jorre,
      Thanks, I didn't know these papers yet. Id be especially interested in the one where you look at the effects on the covariance matrix, because that captures basicly all effects of mode measurement differences in one measure. Great paper you wrote in the recent issue of SRM by the way!

      Delete