Thursday, April 27, 2017

Mobile-only web survey respondents

My breaks between posts are getting longer and longer. Sorry my dear readers. Today, I am writing about research done over a year ago that I did with Vera Toepoel and Alerk Amin.

Our study was about a group of respondents we can no longer ignore: Mobile-only web survey respondents. These are people, who do no longer use a laptop or desktop PC and use their smartphone for most or any of their Internet browsing, but instead use a smartphone. If we as survey methodologists want these people to answer our surveys, we have to design our surveys for smartphones as well.

Who are these mobile-only web survey respondents? This population may of course differ across countries. We used data from the American Life Panel, run by RAND, to investigate what this group looked like in the United States, using data from 2014 (so the situation today may be a bit different). The full paper can be found here

We find that of all people participating in 7 surveys conducted in the panel, 7% is mobile-only in practice. This is not a huge proportion, but it may matter a lot if these 7% of respondents are very different from other types of respondents. We find that they are.

In order to study how different these respondents are, we define 5 groups based on the device the use for responding to surveys:
1. Respondents who always use a PC for completing surveys. This the largest group (68%) and therefore serves as the reference group)
2. Respondents who always use a tablet (5%)
3. Respondents who always use a smartphone ( 7% - our group of interest)
4. Respondents who mix tablets and Pcs (7%)
5. Respondents who mix phones and Pcs (10%)
A further 1% uses all devices, but we ignore these respondents here.

Click Figure to enlarge. The effects shown are always in comparison to the reference group, which is the ‘always PC’ group.

The 5 groups serve as our dependent variable in a multinomial logit regression. The average marginal effects shown in the Figure above respresent the change in the likelihood of being in the 'always PC' group as compared to one of the other groups. The negative age coefficient of -.03 for the always phone group means that with every decade respondents get younger (a negative effect), they have a .03 higher probability to be be in the always phone group as referred to the always Pc group. These effects seem small, but they are not. An imaginary respondent aged 60 has a predicted probability of 92 percent of being in the always PC group as opposed to the always phone group, but this probability is about 80 percent for someone aged 20, controlling for the effects of other covariates.

Our take-away? Apart from age, 'Always phone’ respondents are also less likely to have a higher education (Bachelor degree or higher), are more likely to be married, and more likely to be of Hispanic or African American ethnicity. These characteristics coincide with some of the most important characteristics of hard-to-recruit respondents. While designing your surveys for smartphones will not get these hard-to-recruit respondents into your panel, you can easily lose them by not designing your surveys for smartphones.

Wednesday, February 10, 2016

The traditional web survey is dead

Why we should throw out most of what we know on how to visually design web surveys

In 2000, web surveys looked like postal surveys stuck onto a screen. Survey researchers needed time to get used to the idea that web surveys should perhaps look differently from mail surveys. When I got into survey methodology in 2006, everyone was for example still figuring out whether to use drop down menus (no), how many questions to put on one screen (a few at most), let alone whether to use slider bars (it's not going to reduce breakoffs).
We now know that good web surveys don't look like mail surveys. Proponents of 'gamification' in surveys argue that survey researchers are still underusing the potential of web surveys. Perhaps they are right. So, while web surveys are ubiquitous, and have been around for almost 20 years, we still don't exactly know to to visually design them.

The next challenge is already upon us. How are we going to deal with the fact that so many surveys are now being completed on different devices than computers? Touch screens are getting more popular, and screens for smartphones can be as small as 4 inches. People are nowadays using all kinds of devices to complete surveys. The rules for how a questionnaire should look like on a 21 inch desktop monitor are in my view not the same however as for a 5 inch Iphone.

Just take the following picture as an example. It was taken from the Dutch LISS panel in 2013, before they came up with a new and much better design for mobile surveys. Yes this is still what many people still see when they answer a survey question on a smartphone. Is this the way to do it? Probably not. Why should there be a 'next' (verder) button at the bottom of the page? Can't we redesign the answer scale so that we use more space on the screen?

Small touchscreens ask for a different design. Take the picture below. Tiles are used instead of small buttons, so that the entire screen is used. There is no next button; once you press a tile the next questions shows up. To me this looks better, but is it really? Only proper (experimental) research can tell.

I also still see lots of problems with adapting our visual design to smartphones. Where to put the survey question in the example below? On a separate page? What if respondents tick a wrong answer? And, what if some respondents get this question on a PC? Should the questions look the same to minimize measurement differences, or be optimized for each device separately? Web surveys are dead. Mixed-device surveys are here to stay.

I think there are lots of opportunities for exciting research on how to design surveys for the new era of mixed-device surveys. For anyone interested, I recently edited an open-access special issue on mixed-device surveys together with Vera Toepoel, where different authors for example study:

- why using slider bars is a bad idea for (mobile) surveys
- the effects of device switching within longitudinal surveys (study 1 and study 2) on data quality
- different possible visual designs for implementing mixed-device surveys, and their effects on data quality

These studies are just the beginning. Much more is necessary, and it will take trial, error and probably another 15 years before we have figured out how to design surveys for mixed-device surveys properly.

Tuesday, December 1, 2015

Retrospective reporting

Back after a long pause. Panel surveys traditionally interview respondents at regular intervals, for example monthly or yearly. This interval is mostly chosen for practical reasons: interviewing people more frequently would lead to a large respondent burden, and a burden on data processing and dissemination. For these practical reasons, panel surveys often space their interviews one year apart. Many of the changes (e.g. changes in household composition) we as researchers are interested in occur slowly, and annual interviews suffice to capture these changes.
Sometimes we want to get reports at a more detailed level, however. For example, we would like to know how often a respondents visits a doctor (general practitioner) in one year, or when a respondent went on holidays. In order to get at such an estimate, survey researchers can do one of several things:

1. we can ask a respondent to remember all doctor visits in the past year, and rely on retrospective recall. We know that doesn't work very well, because respondents cannot remember all visits. Instead, respondents will rely on rounding, guessing and estimation to come to an estimate.
2. we can ask respondents for visits in say the past month, and rely on extrapolation to get to an annual estimate. This strategy works well if doctor visits are stable throughout the year (which they are not).
3. We can try to break down the year into months, and instead of asking for doctor visits in the last year, ask for doctor visits in each month, reducing the reference period. We can stimulate the retrieval of the correct information further by using a timeline or by the use of landmarks.
4. We can interview respondents more often. So, we conduct 12 monthly interviews instead of one annual interview, thereby reducing both the reference and recall period.

With Tina Glasner and Anja Boeve, I recently published a paper that compared methods 1, 3 and 4 within the same study to estimate the total number of doctor (family physician) visits. This study is unique in the sense that we used all three methods within the same respondents, so for each respondents we can see how reporting is different when we rely on annual recall, monthly recall, and on whether we use an annual or monthly reference period. Our study also included an experiment to see whether timelines and landmarks improved recall.

You can find the full paper here.

We find that respondents give different answers about their doctor visits depending on how we ask them. The estimates for annual visits are;
- annual estimate (1 question): 1.96 visits
- monthly estimate with 12-month recall: 2.62 visits
- monthly estimate with 1-month recall: 3.90 visits

The average number of doctor visits in  population registers is 4.66, so the monthly estimate with 1 month-recall periods comes closest to our population estimate.

As a final step, we were interested in understanding which respondents give different answers depending on the question format. For this, we studied the within-person absolute difference between the monthly estimates with a 12-month and 1-month recall period. The table below shows that the more frequent doctor visits are, the larger the differences between the 1 month and 12-month recall periods. This implies that respondents with more visits tend to underreport them more often when the recall period is long. The same holds for people in moderate and good health. People in bad health often go to the doctor regularly, and remember these visits. More infrequent visits are more easily forgotten. Finally, we find that the results of our experiment are non-significant. Offering respondents personal landmarks, and putting these next to the timeline to improve recall, does not lead to smaller differences.

In practice, these findings may be useful when one is interested in estimating frequencies of behavior over an annual period. Breaking up the 'annual-estimate' question into twelve monthly questions helps to improve data quality. Asking about such frequencies at 12 separate occasions further helps, but this is unlikely to be feasible due to the large increased costs of more frequent data collection. In self-administered web-surveys this might however be feasible. Splitting up questionnaires into multiple shorter ones may not only reduce burden, but can increase data quality for specific survey questions as well.

Wednesday, May 20, 2015

Adaptive designs: 4 ways to improve panel surveys

This is a follow-up on why I think panel surveys need to adapt their data collection strategies to target individual respondents. Let me first note that apart from limiting nonresponse error, there are other reasons why we would want to do this. We can limit survey costs by using expensive survey resources only for people who need them.
A focus on nonresponse alone can be too limited. For example: imagine we want to measure our respondents' health. We can maybe do this cheaply by using web interviews, and then try to limit nonresponse error by using interviewers to convert initial nonrespondents. But what about measurement? If we use web surveys we largely have to rely on self-reports on the respondents' health. But if we use interviewers for everyone and do a Face-to-Face survey among all our respondents, we can use the interviewers to obtain objective health measures for respondents. These objective measurements could be much better than the self-reports. So face-to-face interviews may not be 'worth' the cost if we look at nonresponse alone, but if we also include the effects on measurement, it may be a viable interview option, if we reduce the sampling size.
I think a focus just on any one type of survey error can have adverse effects, and it is Total Survey Error as well as costs we need to keep in mind. Having said this, I really believe nonresponse errors in panel surveys are a huge problem. What could we do (and have others done)?

1.  Targeted mode. Some respondents are easy to reach in all modes, and some are difficult in all modes. There is also a 'middle' group, who may participate in some modes, but not others. I, for example dislike web surveys (because I get so many), but appreciate mail surveys (because I never get them). In a panel survey, we can ask respondents about mode preferences. Some studies
(here, here) have found that stated mode preferences are not very predictive of response in that mode in the next wave, as people indicate to prefer the mode they are interviewed in. This means we probably need a better model than just 'mode preference' to make this work.

Probably wants a different survey mode next time.
2.  Targeted incentives. We know some respondents are 'in it' for the money, or at least are sensitive to offers of incentives. In panel surveys, we can learn quickly about this by experimenting with amounts both between and/or within persons. For example, does it help to offer higher incentives to hard-to-reach respondents? Does that help just once, or is there a persistent effect? It may be unethical to offer higher incentives to just hard-to-reach respondents, as we then put a premium on bad respondent behavior. We could however use different metrics for deciding whom to offer the incentive. Nonresponse bias is a much better indicator for example. There is not too much we know about how to do this, although there is a nice example here.

3. Targeted advance letters. We know quite a lot  about the effects different types of advance letters have on subsequent response. Young people can for example be targeted with a letter with a flashy lay-out and short bites of information of the study, while older people may prefer a more 'classic' letter with more extensive information about the study setup and results.
The effects of targeted letters on response in panel surveys are often small, and only present for specific subgroups. See here, and here for two examples. Still, this type of targeting costs little, perhaps we can find bigger effects if we know what groups to target with what message. As with other targeting methods, we need a combination of data mining and experimentation to develop knowledge about this.

4. Targeted tracking. I am not aware of any survey doing targeted tracking. Tracking is done during fieldwork. Respondents who are not located by an interviewer (or advance letter which bounce), are sent back to the study coordinating team, after which tracking methods are used to locate the respondent at an alternative address. From the literature we know that it is mainly people who move house who need tracking. If we can successfully predict the likelihood to move, we could potentially save time (and money) in fieldwork, by putting cases into preventive tracking. We could also potentially use a targeted order of tracking procedures, as James Wagner has done.


Wednesday, January 21, 2015

why panel surveys need to go 'adaptive'

Last week, I gave a talk at Statistics Netherlands (slides here) about panel attrition. Initial and nonresponse and dropout from panel surveys have always been a problem. A famous study by Groves and Peytcheva (here) showed that in cross-sectional studies, nonresponse rates and nonresponse bias are only weakly correlated. In panel surveys however, all the signs are there that dropout in a panel study is often related to change. Those respondents undergoing most change, are also most likely to drop out. This is probably partly because of respondents (e.g. a move of house could be a good reason to change other things as well, like survey participation), but it is also because of how surveys deal with such moves. Movers are much harder to contact (if we don't have accurate contact details anymore). Movers are often assigned to a different interviewer. This will all lead to an underestimate of the number of people who move house in panel studies. Moving house is associated with lots of other life events (change in household composition, change in work, income etc.). In short dropout is a serious problem in longitudinal studies.

The figure below shows the cumulative response rates for some large-scale panel studies. The selection of panel studies is a bit selective. I have tried to focus on large panel studies (so excluding cohort studies), which are still existing today, with a focus on Western Europe. 

Cumulative nonresponse rates in large panel surveys (click to enlarge)

The oldest study in the figure (PSID) has the highest initial response rate, followed by studies which were started in the 1980s (GSOEP), 1990s (BHPS), and early 2000s (HILDA). The more recent studies all have higher initial nonresponse rates. But not only that. They also have higher dropout rates (the lines go down much faster). This is problematic.

I think these differences are not due to the fact that we, as survey methodologists, are doing a worse job now as compared to 20 years ago. If anything, we have been trying use more resources, professionalize tracking, offer higher incentives, and be more persistent. In my view, the increasing dropout rates are due to changes in society (the survey climate). A further increase of our efforts (e.g. higher incentives) could perhaps help somewhat to reduce future dropout. I think this is however not the way to go, especially as budgets for data collection face pressures everywhere.

The way to reduce panel dropout is to collect data in a smarter way. First, we need to understand why people drop out. This is something we know quite well (but more can be done). For example, we know that likely movers are at risk. So, what we need are tailored strategies that focus on specific groups of people (e.g. likely movers). For example, we could send extra mailings in between waves only to them. We could use preventive tracking methods. We could put these into the field earlier.

I am not the first to suggest such strategies. We have been tailoring our surveys for ages to specific groups, but have mostly done so at an ad-hoc basis,  never systematically. Responsive or adaptive designs try to use tailoring systematically, for those groups that most benefit from tailoring. Because we know so much about our respondents after wave 1, panel studies offer lots of opportunities to implement responsive designs.

Monday, December 8, 2014

Satisficing in mobile web surveys. Device-effect or selection effect?

Last week, I wrote about the fact that respondents in panel surveys are now using tablets and smartphones to complete web surveys. We found that in the LISS panel, respondents who use tablets and smartphones are much more likely to switch devices over time and not participate in some months.
The question we actually wanted to answer was a different one: do respondents who complete surveys on their smartphone or mobile give worse answers?

To do this, we used 6 months of data from the LISS panel, and in each month, coded the User Agent String. We then coded types of satisficing behavior that occur in surveys: the percentage of item missings, whether respondents complete (non-mandatory) open questions, how long their answers were, whether respondents straightline, whether they go for the first answers in a check-all-that-apply questions, and how many answers they click in a check-all-that apply question. We also looked at interview duration, and how much respondents liked the survey.

We found that respondents on a smartphone seem to do much worse. They take longer to complete the survey, are more negative about the survey, have more item missings, and have a much higher tendency to pick the first answer. On the other questions, differences were small, sometimes in favor of the smartphone user.

Click to enlarge: indicators of satisficing per device in LISS survey
Is this effect due to the fact that the smartphone and tablet are not made to complete surveys, and is satisficing higher because of a device-effect? Or is it a person effect, and are worse respondents more inclined to do a survey on a tablet or smartphone?

In order to answer this final question, we looked at device-transitions that respondents take within the LISS panel. In the 6 months of the LISS, respondents can make 5 transitions from using 1 device in the one month, to another (or the same) device in the next. For 7 out of 9 transitions (we have too few observations to analyze the tablet -> phone and phone -> tablet transitions), we can then look at the difference in measurement error that is associated with a change in device.

Click to enlarge. Changes in data quality (positive is better) associated with change in device.

The red bars indicate that there is no significant change in measurement error associated with a device change. Our conclusion is that device changes do not lead to more measurement error, with 2 exceptions:
1. A transition from tablet -> PC or phone -> PC in two consecutive months, leads to a better evaluation of the questionnaire. This implies that the user experience of completing web surveys on a mobile device should be improved.
2. We find that people check more answers in a check-all-that-apply question when they move from a tablet -> PC, or phone -> PC

So, in short. Satisficing seems to be more problematic when surveys are completed on tablets and phones. But this can almost fully be explained by a selection effect. Those respondents who are worse completing surveys, choose to complete surveys more on tablets and smartphones.

The full paper can be found here

Tuesday, December 2, 2014

Which devices do respondents use over the course of a panel study?

Vera Toepoel and I have been writing a few articles over the last two years about how survey respondents are taking up tablet computers and smartphones. We were interested in studying whether people in a probability-based web panel (the LISS panel) use different devices over time, and whether siwtches in devices for completing surveys are associated with more or less measurement error.

In order to answer this question, we have coded the User Agent Strings of the devices used by more than 6.000 respondents over a six month period. (see the publication tab for a syntax on how to do this using R).

We find, as others have done, that in every wave about 10% of respondents either use a tablet or smartphone. What is new in our analyses is that we focus on the question whether respondents persistently use the same device.

The table below shows that PC users largely stick to their PC in all waves. For example, we see that 77.4% of PC-respondents in April, again use a PC in May. Only 1.5% of April’s PC respondents switch to either a tablet or smartphone to complete a questionnaire in May.

Table. Devices used between April and September 2013 by LISS panel respondents.
N = 6,226. Click to enlarge
The proportion of respondents switching a PC for either a tablet or smartphone is similarly low in the other months, and is never more than 5%. This stability in device use for PCs is, however, not found for tablets and smartphones. Once people are using a smartphone in particular, they are not very likely to use a smartphone in the next waves of LISS. Only 29 per cent of smartphone users in July 2013, again uses a smartphone in August for example. The consistency of tablet usage increases over the course of the panel; 24% of respondent is a consistent tablet user in April-May, but this increases to 64% in July-August.

Finally, it is worth to note that the use of either a smartphone or a tablet is more likely to lead to non-participation in the next wave of the survey. This may however be a sample selection effect. More loyal panel members may favor the PC to complete the questionnaires.

More in a next post on the differences between respondents answer behavior over time, when they switch devices. Do respondents become worse when they answer a survey on a smartphone or tablet?

You can download the full paper here

Monday, November 3, 2014

Delphi-study on adjustment techniques in surveys

Some people in my university department (including me) have started a study to how people deal with non-probability surveys. The study is led by Hannah Emerson, an MsC student who is writing her thesis on the topic. What kind of adjustment techniques do people use? What kind of error in non-probability samples do researchers try to adjust for? How do they make decisions on how to do that? These are rather open-ended questions, and so we have opted to do a Delphi-study. In 3 rounds we hope to achieve consensus among survey methodologists on when adjustments make sense and how they should be done and evaluated.

The first round of the Delphi study just started. See the announcement below. Your participation in the study is much appreciated.


In the literature of survey methodology, different weighting techniques are presented for surveys that are based on non-probability samples. It is often unclear, however, how successful the techniques are in reducing bias and as far as what types of bias each technique aims to correct. 
 It is also unclear how different agencies (academia/science, government and market research) apply the techniques and how these differences extend throughout different countries.
With the following survey we intend to gather this information and summarize it via use of the Delphi Method. The Delphi method aims to gather a consensus amongst ‘expert panelists’ in a series of 3 questionnaires.  
In the first wave, our aim is to collect data on the types of weighting adjustment techniques used in nonprobability samples.  In the second wave we will present you with the answers of your fellow panelists and ask if you agree with the ideas of adjustments as well as show you where your response lines up with theirs.  In the third wave we will see if we can get consensus among the panelists. 
Our hope is that we can come together to make sense of the ‘big black box’ that surrounds the weighting techniques used in web surveys. Please consider participating in order to make this happen. If you yourself are not the one who adjusts surveys for bias, we ask you to please forward this email to the person who does. Thank you for your time and consideration for participating in this much needed exploration.
Click on the following link to access the survey, which will take approximately 10 minutes to complete:

with kind regards and many thanks,
Vera Toepoel, Peter Lugtig, Barry Schouten and Hannah Emerson

Wednesday, August 13, 2014

Can you push web survey respondents to complete questionnaires on their mobile phones?

I am back from some great holidays, and am revisiting some of the research I did over the last 2 years. Back then, I would have not expected that I would become so interested in doing survey research on mobile phones. I do think that a little change of research topic does one good.

I have written two papers with Vera Toepoel on how to do surveys on mobile phones. The first question we had was whether people were actually likely to do a survey on a mobile phone. Last year, Marketresponse, a probability-based web panel in the Netherlands, had changed their survey software so that questionnaires would be dynamically adapted to mobile phone screen settings, and navigation methods. They then informed their respondents about it, and encouraged them to try a short survey on shopping behavior on their smartphone (if respondents had one).

We found that of those respondents who owned a smartphone, 59% chose to use it when encouraged and were positively surprised by this finding. Even with quite little encouragement, survey respondents are willing to try completing the survey on their mobile phone. Also, we found little reason to be worried about side-effects of encouraging mobile survey response.

- We found little differences in terms of demographics between those who did the survey on a mobile phone, or a desktop (including tablets).
- We found no differences in terms of response behavior.
- We found no difference in how mobile and desktop respondents evaluated the questionnaire.
- We found no difference in the time it took them to complete the survey (see the figure below). In fact, the timings were so similar, we could scarcely believe the differences were so small.

The full paper can be found here. There are a few potential caveats in our study: we use a sample of experienced survey respondents and did not use experimental assignments, so self-selection into device could be selective beyond the variables we studied. So far however, it really seems that web surveys on a mobile phone are not very different for respondents than traditional web surveys.

Friday, June 27, 2014

Call for papers: Thematic issue on “The collection of survey data using mixed devices” - deadline for abstracts 31 August 2014

With Vera Toepoel, I have recently written two articles on doing surveys on mobile phones and tablets. I will write about these articles another day, but you can find one article and some presentations under the publications tab. I like doing research about how to do survey research on mobile devices, and am with Vera Toepoel, editing a special issue for Methods, Data and Analysis. See the call for abstracts below.


The peer reviewed, free and open access journal "Methods, Data, Analyses (MDA)", published by GESIS ( is soliciting contributions to a special issue on the "collection of survey data using mixed-devices". Vera Toepoel and Peter Lugtig from Utrecht University will serve as guest editors of this special issue, to be published in the fall/winter issue of 2015.

Surveys can nowadays be completed on many devices. Respondents can usually choose which device they want to use if they complete a web survey for example. The devices range from a traditional desktop computer, to a tablet, smartphone or hybrids of these. In more traditional surveys, interviewers may use different devices within one interview to record survey data; for example a CAPI survey instrument, along with specific devices to measure various aspects of health. These mixes of devices between and within interviews poses challenges to survey methodologists, mainly in questionnaire design and measurement error, but also in sampling and nonresponse conversion.

We are seeking papers that highlight potential opportunities and problems in mixed-device data collection, compare different approaches to deal with these problems, and/or propose solutions. For example:

- how can mixed-devices help to draw more efficient samples, and reduce noncoverage and nonresponse bias?
- can survey respondents be effectively nudged towards using a particular device within a web-survey?
- can mixed-device survey open up new possibilities for measurement?
- how to design Web survey questionnaires effectively for use across mixed-devices?

We welcome contributions that incorporate experimental as well as non-experimental data, and are open to both small- and large-scale studies.

Submission procedure:

If you are interested in submitting an article please submit a maximum 2-page abstract, documenting your main research questions, design of the study, and proposed analyses. We are open to abstracts for which data collection has started or is starting soon, but is not completed yet. The "route to publication" listed below allows time for data to be collected.

- deadline for abstracts submissions:           31 August 2014.
- feedback from editors about acceptance:   15 september 2014.
- deadline for paper submissions:                30 April 2015.
- peer review feedback:                             30 June 2015.
- second submission:                                 31 August 2015.
- Special issue to press:                             30 september 2015.

postscript (02-01-2016): the special issue was published in december 2015, and can be found here: