Monday, December 8, 2014

Satisficing in mobile web surveys. Device-effect or selection effect?


Last week, I wrote about the fact that respondents in panel surveys are now using tablets and smartphones to complete web surveys. We found that in the LISS panel, respondents who use tablets and smartphones are much more likely to switch devices over time and not participate in some months.
The question we actually wanted to answer was a different one: do respondents who complete surveys on their smartphone or mobile give worse answers?

To do this, we used 6 months of data from the LISS panel, and in each month, coded the User Agent String. We then coded types of satisficing behavior that occur in surveys: the percentage of item missings, whether respondents complete (non-mandatory) open questions, how long their answers were, whether respondents straightline, whether they go for the first answers in a check-all-that-apply questions, and how many answers they click in a check-all-that apply question. We also looked at interview duration, and how much respondents liked the survey.

We found that respondents on a smartphone seem to do much worse. They take longer to complete the survey, are more negative about the survey, have more item missings, and have a much higher tendency to pick the first answer. On the other questions, differences were small, sometimes in favor of the smartphone user.

Click to enlarge: indicators of satisficing per device in LISS survey
Is this effect due to the fact that the smartphone and tablet are not made to complete surveys, and is satisficing higher because of a device-effect? Or is it a person effect, and are worse respondents more inclined to do a survey on a tablet or smartphone?

In order to answer this final question, we looked at device-transitions that respondents take within the LISS panel. In the 6 months of the LISS, respondents can make 5 transitions from using 1 device in the one month, to another (or the same) device in the next. For 7 out of 9 transitions (we have too few observations to analyze the tablet -> phone and phone -> tablet transitions), we can then look at the difference in measurement error that is associated with a change in device.

Click to enlarge. Changes in data quality (positive is better) associated with change in device.


The red bars indicate that there is no significant change in measurement error associated with a device change. Our conclusion is that device changes do not lead to more measurement error, with 2 exceptions:
1. A transition from tablet -> PC or phone -> PC in two consecutive months, leads to a better evaluation of the questionnaire. This implies that the user experience of completing web surveys on a mobile device should be improved.
2. We find that people check more answers in a check-all-that-apply question when they move from a tablet -> PC, or phone -> PC

So, in short. Satisficing seems to be more problematic when surveys are completed on tablets and phones. But this can almost fully be explained by a selection effect. Those respondents who are worse completing surveys, choose to complete surveys more on tablets and smartphones.

The full paper can be found here

10 comments:

  1. Wonderful illustrated information. I thank you about that. No doubt it will be very useful for my future projects. Would like to see some other posts on the same subject! iPhone Repair Dubai

    ReplyDelete
  2. I am hoping the same best effort from you in the future as well. In fact your creative writing skills has inspired me. app para espiar celulares

    ReplyDelete
  3. We have sell some products of different custom boxes.it is very useful and very low price please visits this site thanks and please share this post with your friends. small business marketing strategy

    ReplyDelete
  4. Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained! Text Marketing Auto Reply

    ReplyDelete
  5. Site Traffic: As talked about above, it is hard to see how individuals are drawing in with your image through disconnected promoting. ExcelR Digital Marketing Course in Sydney

    ReplyDelete
  6. There are some interesting points in time in this article but I don’t know if I see all of them center to heart. There is some validity but I will take hold opinion until I look into it further. Good article, thanks and we want more! Added to Feed Burner as well.
    Follow me

    ReplyDelete
  7. These online website composition courses cost somewhere in the range of thirty and one hundred dollars for every month so anybody can perceive how much better they are as far as money related ramifications.

    Web Design Curriculum

    ReplyDelete
  8. a debt of gratitude is in order for this usefull article, sitting tight for this article like this once more. Pro Well Tech

    ReplyDelete
  9. This is my first time visit to your blog and I am very interested in the articles that you serve. Provide enough knowledge for me. Thank you for sharing useful and don't forget, keep sharing useful info: http://geolocatel.com/

    ReplyDelete