Saturday, March 26, 2011

causality

A short post in between, so I can share two thoughts:
1. it is now possible to post reactions to my blog posts. Because I'm new to blogging, the settings were not very inviting previously. Now, you can react very easily. Please do if you feel like it. I believe in progress by debate.
2. I am not a specialist in the concept of causality myself, but I love Judea Pearl's blog on the topic of causal relationships, counterfactuals and the role of covariates in the social sciences. He recently posted some great links, new ideas and video's. Go and read it if you have a spare hour.

Wednesday, March 23, 2011

mixed-mode designs: cognitive equivalence

Instead of separating out mode effects from nonresponse and noncoverage effects through statistical modeling, it is perhaps better to design our mixed-mode surveys in such a way so that mode effects do not occur. The key principle in preventing the mode effects from occurring, is to make sure that questionnaires are cognitively equivalent to respondents. This means that no matter in which survey mode the respondents participate, they would give the same answer. In my opinion, there are two ways to achieve this.

1. choose a mix of modes that lead to a cognitively equivalent survey process. The survey process is very different in a questionnaire administered in a telephone vs. an Internet mode. Some mode combinations are can however be combined without great differences between they survey process across the modes:

- combine face-to-face with telephone modes: the mode of communication is in both modes aural with an interviewer asking and recording answers. The only difference is that the interviewer is physically present in the face-to-face survey, and not in the telephone survey.
- combine mail and Internet modes. Differences between these modes are minimal. Whereas in the United States it is difficult to sample addresses (but not impossible), in Europe, this combination can easily be implemented. Don  Dillman talks about some experiments with this method on the 2009 AAPOR conference (thanks to www.pollster.com).



2. The second way is to use nonequivalent survey modes (for example the telephone and internet), but design the individual survey questions in such a way that they are still equivalent across modes. This implies that all questions should be simple, short and clear, and that there should be as few answer categories as possible (i.e. yes/no and similar). This means that it would be difficult to ask for attitudes or opinions in such a mixed mode design.

Tuesday, March 15, 2011

matching to correct for self-selection bias in mixed-mode surveys

Mixed mode surveys have shown to attract different types of respondents. This may imply that they are succesful. Internet surveys attract the young and telephone surveys the old, so any combination of the two can lead to better population estimates for the variable you're interested in. In other words, mixed-mode surveys can potentially ameliorate the problem that neither telephone, nor Internet surveys are able to cover the entire population.

The bad news is that mode-effects (see posts below) coincide with selection effect in mixed-mode surveys. For that reason, it is hard to determine how succesful mixed-mode surveys are, and more importantly, really hard to combine results when there are large differences in the dependent variable across the survey modes.

I think that matching is one of the few methods to adequately deal with this issue: the idea is straightforward. In any survey among the general population, there will be 1. people who are able and willing to only answer in a specific survey mode (i.e. the Internet or telephone), 2. respondents who would respond in both and 3. respondents who would not participate at all. This means that the composition of the telephone and Internet-samples in a mixed-mode survey will contain people unique to that mode, and people who can also be found in the other mode (see below - the match part).

With matching, respondents who are similar on a set of covariates are matched from both survey modes, so that pairs of very similar respondents are formed from every survey mode. After matching, any differences that persists between the matched respondents from both samples cannot be due to selection effects on the covariates. Therefore, any differences that remain between the matched respondents after matching should exist only because of a mode effect: whether a question is asked by the interviewer or self-administered, whether it is audial or visual, and whether answers are spoken or written down.
Matching can be easily done using the package MatchIt in R (amongst others). More information about matching in mixed-mode surveys can be found in a manuscript I wrote with some colleagues.