Monday, December 8, 2014

Satisficing in mobile web surveys. Device-effect or selection effect?


Last week, I wrote about the fact that respondents in panel surveys are now using tablets and smartphones to complete web surveys. We found that in the LISS panel, respondents who use tablets and smartphones are much more likely to switch devices over time and not participate in some months.
The question we actually wanted to answer was a different one: do respondents who complete surveys on their smartphone or mobile give worse answers?

To do this, we used 6 months of data from the LISS panel, and in each month, coded the User Agent String. We then coded types of satisficing behavior that occur in surveys: the percentage of item missings, whether respondents complete (non-mandatory) open questions, how long their answers were, whether respondents straightline, whether they go for the first answers in a check-all-that-apply questions, and how many answers they click in a check-all-that apply question. We also looked at interview duration, and how much respondents liked the survey.

We found that respondents on a smartphone seem to do much worse. They take longer to complete the survey, are more negative about the survey, have more item missings, and have a much higher tendency to pick the first answer. On the other questions, differences were small, sometimes in favor of the smartphone user.

Click to enlarge: indicators of satisficing per device in LISS survey
Is this effect due to the fact that the smartphone and tablet are not made to complete surveys, and is satisficing higher because of a device-effect? Or is it a person effect, and are worse respondents more inclined to do a survey on a tablet or smartphone?

In order to answer this final question, we looked at device-transitions that respondents take within the LISS panel. In the 6 months of the LISS, respondents can make 5 transitions from using 1 device in the one month, to another (or the same) device in the next. For 7 out of 9 transitions (we have too few observations to analyze the tablet -> phone and phone -> tablet transitions), we can then look at the difference in measurement error that is associated with a change in device.

Click to enlarge. Changes in data quality (positive is better) associated with change in device.


The red bars indicate that there is no significant change in measurement error associated with a device change. Our conclusion is that device changes do not lead to more measurement error, with 2 exceptions:
1. A transition from tablet -> PC or phone -> PC in two consecutive months, leads to a better evaluation of the questionnaire. This implies that the user experience of completing web surveys on a mobile device should be improved.
2. We find that people check more answers in a check-all-that-apply question when they move from a tablet -> PC, or phone -> PC

So, in short. Satisficing seems to be more problematic when surveys are completed on tablets and phones. But this can almost fully be explained by a selection effect. Those respondents who are worse completing surveys, choose to complete surveys more on tablets and smartphones.

The full paper can be found here

Tuesday, December 2, 2014

Which devices do respondents use over the course of a panel study?


Vera Toepoel and I have been writing a few articles over the last two years about how survey respondents are taking up tablet computers and smartphones. We were interested in studying whether people in a probability-based web panel (the LISS panel) use different devices over time, and whether siwtches in devices for completing surveys are associated with more or less measurement error.

In order to answer this question, we have coded the User Agent Strings of the devices used by more than 6.000 respondents over a six month period. (see the publication tab for a syntax on how to do this using R).

We find, as others have done, that in every wave about 10% of respondents either use a tablet or smartphone. What is new in our analyses is that we focus on the question whether respondents persistently use the same device.

The table below shows that PC users largely stick to their PC in all waves. For example, we see that 77.4% of PC-respondents in April, again use a PC in May. Only 1.5% of April’s PC respondents switch to either a tablet or smartphone to complete a questionnaire in May.

Table. Devices used between April and September 2013 by LISS panel respondents.
N = 6,226. Click to enlarge
The proportion of respondents switching a PC for either a tablet or smartphone is similarly low in the other months, and is never more than 5%. This stability in device use for PCs is, however, not found for tablets and smartphones. Once people are using a smartphone in particular, they are not very likely to use a smartphone in the next waves of LISS. Only 29 per cent of smartphone users in July 2013, again uses a smartphone in August for example. The consistency of tablet usage increases over the course of the panel; 24% of respondent is a consistent tablet user in April-May, but this increases to 64% in July-August.

Finally, it is worth to note that the use of either a smartphone or a tablet is more likely to lead to non-participation in the next wave of the survey. This may however be a sample selection effect. More loyal panel members may favor the PC to complete the questionnaires.

More in a next post on the differences between respondents answer behavior over time, when they switch devices. Do respondents become worse when they answer a survey on a smartphone or tablet?

You can download the full paper here

Wednesday, August 13, 2014

Can you push web survey respondents to complete questionnaires on their mobile phones?

I am back from some great holidays, and am revisiting some of the research I did over the last 2 years. Back then, I would have not expected that I would become so interested in doing survey research on mobile phones. I do think that a little change of research topic does one good.

I have written two papers with Vera Toepoel on how to do surveys on mobile phones. The first question we had was whether people were actually likely to do a survey on a mobile phone. Last year, Marketresponse, a probability-based web panel in the Netherlands, had changed their survey software so that questionnaires would be dynamically adapted to mobile phone screen settings, and navigation methods. They then informed their respondents about it, and encouraged them to try a short survey on shopping behavior on their smartphone (if respondents had one).

We found that of those respondents who owned a smartphone, 59% chose to use it when encouraged and were positively surprised by this finding. Even with quite little encouragement, survey respondents are willing to try completing the survey on their mobile phone. Also, we found little reason to be worried about side-effects of encouraging mobile survey response.

- We found little differences in terms of demographics between those who did the survey on a mobile phone, or a desktop (including tablets).
- We found no differences in terms of response behavior.
- We found no difference in how mobile and desktop respondents evaluated the questionnaire.
- We found no difference in the time it took them to complete the survey (see the figure below). In fact, the timings were so similar, we could scarcely believe the differences were so small.


The full paper can be found here. There are a few potential caveats in our study: we use a sample of experienced survey respondents and did not use experimental assignments, so self-selection into device could be selective beyond the variables we studied. So far however, it really seems that web surveys on a mobile phone are not very different for respondents than traditional web surveys.

Monday, May 26, 2014

AAPOR 2014

Big data and new technologies to do survey research. These were in my view the two themes of the 2014 AAPOR conference. The conference organisation tried to push the theme ‘Measurement and the role of pubic opinion in a democracy’, but I don't think the theme was really reflected in the talks at the conference. Or perhaps I have missed those talks, the conference was huge as always (> 1000 participants).

The profession of survey research is surely changing. Mick Couper last year argued that the ‘sky wasn’t falling’ on survey research, but it is evolving. Big data may potentially replace parts of survey research, especially if we don't adapt to new technologies (mobile), and learn to use some of the data that are now found everywhere. Big data and survey research in fact have the same basic goal. To extract meaningful information out of datasets (big data) or people (survey research), and use that to inform policy making.

Big data can certainly be useful for policy-making. Out of the 10 or so presentations that I have seen at AAPOR, most were however just talking about potential possibilities over using big data to inform policy makers.
What was in my opinion missing at AAPOR were good case studies that showed how big data can replace survey research and provide valid inferences. I have seen many good earlier examples when it comes to predictions at the level of an individual using big data. When Amazon tries to recommend me books that relate to a book I have previously bought, I find these useful and accurate predictions of what I really like. In politics, voter registration records data can help politicians target likely voters for their party, as the 2012 Obama campaign showed.

But when it comes to aggregating big data to the level of the population, big data is often in trouble (the Obama election campaign is an outlier here, as they collect data on the whole population). Survey research has relied on the principle of random sampling from the population to draw inferences, but for big data, coverage and nonresponse errors are often unknown and unestimatible for the convenience samples that big data ususally are. Paul Biemer made this point in an excellent talk.

Most of the other big data presentations at AAPOR to me were either in the category ‘bar talk’ - anecdotes without a scientific empirical strategy - or just talked about the potential of big data. And don’t get me wrong: I do think that big data are very useful, especially if they cover a late proportion of the population (e.g. voter records), or if the goals is prediction at the level of an individual.

The other conference theme seemed to be mobile surveys. With Vera Toepoel, I gave a presentation on this topic, which may be the topic of a next blogpost. Here, I think survey researchers are much better equipped to deal with the challenge mobile devices pose. I saw many excellent presentations on questionnaire design for mobile surveys, and selection bias.

Finally, this is just my conference take-away. Some other bloggers (here here) seem to have a slightly different view on the conference. Probably this is due to the fact I have only seen 1 out of the 8 presentations given at any time. So be sure to check their posts out if you want to know more about the conference.

Tuesday, April 29, 2014

Are item-missings related to later attrition?

A follow up on last month's post. Respondents do seem to be less compliant in the waves before they drop out from a panel survey. This may however not neccesarily lead to worse data. So, what else do we see before attrition takes place? Let have a look at missing data:

First, we look at missing data in a sensitive question on income amounts. Earlier studies (here, here, here) have already found that item nonresponse on sensitive questions predicts later attrition. I find that item nonresponse does increase before attrition, but only because of the fact that respondents are more likely to refuse to give an answer. And that increase is largely due to respondents who will later refuse to participate in the study as a whole. So, item refusals are a good predictor of later study refusals. The proportion of "Don't know" respondents does not increase over time.

Missing income data in BHPS in 5 waves before attrition (click to enlarge)

Does this finding for a sensitive question extend to all survey questions? No. Over all questions combined, I find that refusals  increase before attrition takes place, but  from a very low base (see the Y-axis scale in the figure below). Moreover, there is no difference between the groups, meaning that those who drop out of the survey do not have more item-missings than those respondents who are "always interviewed". It may seem odd that item missings increase for respondents who always happily participate. I suspect however that this may be related to the fact that both interviewers and respondents may have known in the last wave(s) that the BHPS was coming to an end after 18 years of interviewing.


Missing data for all survey questions in BHPS in waves before attrition (click to enlarge)
What to do with this information? It seems that later study refusals can be identified using a combination of item nonresponses and survey compliance indicators. Once these respondents are identified, the next step would be to target them with survey design features that try to prevent attrition. These survey design features should target some of the concerns and motivations such respondents have that cause them to drop out from the survey.

Friday, March 28, 2014

Do respondents become sloppy before attrition?

I am working on a paper that aims to link measurement errors to attrition error in a panel survey. For this, I am using the British Household Panel Survey. In an earlier post I already argued that attrition can occur for many reasons, which I summarized in 5 categories.

1. Noncontact
2. Refusal
3. Inability (due to old age, infirmity) as judged by the interviewer, also called 'other non-interview'.
4. Ineligibibility (due to death, or move into institution or abroad).
5. people who were always interviewed

In the paper, I study whether attrition due to any of the reasons above can be linked to increased measurement errors in the last waves before attrition. For example, earlier studies have found that item nonresponse to sensitive questions (income) predicts unit nonresponse in the next waves.

For every respondent in the BHPS, I coded different indicators measurement error in every of the last five waves before attrition takes place. My working hypothesis is that measurement errors should increase in the last few waves before attrition takes place, due to decreasing respondent willingness and/or capability to participate.

In the figure below, you find one set of indicators I used. Compliance to the survey does not count as an indicator of measurement error, but I found it interesting to look into nonetheless. I find that respondents are far less keen to do "extra" tasks in the waves before attrition. As measures, of compliance to these extra tasks, I looked at:

1. the respondent cooperation as judged by the interviewer.
2 the proportion of respondents who completes the tracking schedule at the end of the interview, and
3. the proportion of respondents returning a self-completion questionnaire, left after the interview.

In order to be able to interpret the results in a good way, I contrasted the 4 attrition groups with the 5th group of respondents who do not drop out, and are always interviewed.

Compliance with survey task by respondents in last 5 waves before attrition (click to enlarge)
Unsuprisingly, I find that compliance decreases before attrition. Even at 5 waves before attrition, I find differences between the groups, with the "always interviewed" being most compliant, and the later to "refuse" group least compliant. The differences between the groups increase, the closer they get to attrition. Of the groups that attrite, the "noncontacts" and later "ineligibles" do only a little worse than the "always interviewed". The "refusers" and "inables" have sharply decreasing cooperation ratings, and rates of completing the tracking schedule and returning the self-completion questionnaire. The differences between the groups are not large enough to predict exactly who is going to refuse or become unable to participate, but they can help to identify respondents being at risk.

The next question would be what to do with this knowledge.  If a respondent really is unable to participate, there is not so much we as survey practitioners can do about this. Likely refusers may also be hard to target effectively. The rate of noncontacts is to a large degree under the control of survey practitioners, and for that reason, many nonresponse researchers are trying to limit noncontacts. Although refusers may be harder to target than noncontacts, it may be easier to identify potential refusers, and take pre-emptive action, rather than use refusal conversion techniques after a  respondent has refused.

Friday, February 7, 2014

Personality predicts the likelihood and process of attrition in a panel survey

Studies into the correlates of nonresponse often have to rely on socio-demographic variables to study whether respondents and nonrespondents in surveys differ. Often there is no other information available on sampling frames that researchers can use.

That is unfortunate, for two reasons. First, the  variables we are currently using to predict nonrespons, usually explain a very limited amount of variance of survey nonresponse. Therefore, these variables are also not effective correctors for nonresponse. So, socio-demographic variables are not that interesting to have as sampling frame data. See my earlier posts here and here.

It is also unfortunate, because theoretically, we can come up with other respondent characteristics that should explain nonresponse much better. For example, whether respondents believe surveys to be important, whether they enjoy thinking (need for cognition) and whether they have a conscientious personality.

I published a new paper today in Sociological Methods and Research, that links these variables in particular to different patterns of attrition in a longitudinal survey. See the full paper here

Among the 2007 sample members of the Dutch LISS Panel, I tested for different drop out patterns, and classified respondents according to their attrition process. Some respondents are continuous respondents (stayers - on top), while others start enthusiastically, but drop out at various stages of the survey (lines going down), or never really enthusiastically participate (lurkers - erratic lines in the middle).
Figure 1: attrition patterns among original LISS sample members (click to enlarge)

As a next step, I linked the attrition classes to a set of predictors. Among them socio-demographic variables, but also psychological, and survey related ones.

The 3 strongest predictors of the differences between the attrition patterns (latent classes) are:
- Personality. Conscientious and less extravert people drop out less often
- Survey enjoyment. If people enjoy completing surveys, they are not likely to drop out.
- Having received a PC. The LISS panel gave respondents without Internet access a computer and broadband Internet to enable them to participate. Almost none of the people who received such a computer dropped out from the study.

In an earlier post about an upcoming book chapter, I already noted that the correlates of attrition and initial nonresponse are very different in the LISS, so personality may not explain initial nonresponse in other panel surveys or nonresponse in cross-sectional surveys.

It would be very hard to test my hypothesis that personality is a strong predictor of all types of nonresponse. Unless you would do a survey among employees, and have simultaneous access all personality data from tests that every job applicant at that company took. If you sit on top such data and want me to analyse them, do e-mail me.