Tuesday, December 1, 2015

Retrospective reporting

Back after a long pause. Panel surveys traditionally interview respondents at regular intervals, for example monthly or yearly. This interval is mostly chosen for practical reasons: interviewing people more frequently would lead to a large respondent burden, and a burden on data processing and dissemination. For these practical reasons, panel surveys often space their interviews one year apart. Many of the changes (e.g. changes in household composition) we as researchers are interested in occur slowly, and annual interviews suffice to capture these changes.
Sometimes we want to get reports at a more detailed level, however. For example, we would like to know how often a respondents visits a doctor (general practitioner) in one year, or when a respondent went on holidays. In order to get at such an estimate, survey researchers can do one of several things:

1. we can ask a respondent to remember all doctor visits in the past year, and rely on retrospective recall. We know that doesn't work very well, because respondents cannot remember all visits. Instead, respondents will rely on rounding, guessing and estimation to come to an estimate.
2. we can ask respondents for visits in say the past month, and rely on extrapolation to get to an annual estimate. This strategy works well if doctor visits are stable throughout the year (which they are not).
3. We can try to break down the year into months, and instead of asking for doctor visits in the last year, ask for doctor visits in each month, reducing the reference period. We can stimulate the retrieval of the correct information further by using a timeline or by the use of landmarks.
4. We can interview respondents more often. So, we conduct 12 monthly interviews instead of one annual interview, thereby reducing both the reference and recall period.

With Tina Glasner and Anja Boeve, I recently published a paper that compared methods 1, 3 and 4 within the same study to estimate the total number of doctor (family physician) visits. This study is unique in the sense that we used all three methods within the same respondents, so for each respondents we can see how reporting is different when we rely on annual recall, monthly recall, and on whether we use an annual or monthly reference period. Our study also included an experiment to see whether timelines and landmarks improved recall.

You can find the full paper here.

We find that respondents give different answers about their doctor visits depending on how we ask them. The estimates for annual visits are;
- annual estimate (1 question): 1.96 visits
- monthly estimate with 12-month recall: 2.62 visits
- monthly estimate with 1-month recall: 3.90 visits

The average number of doctor visits in  population registers is 4.66, so the monthly estimate with 1 month-recall periods comes closest to our population estimate.

As a final step, we were interested in understanding which respondents give different answers depending on the question format. For this, we studied the within-person absolute difference between the monthly estimates with a 12-month and 1-month recall period. The table below shows that the more frequent doctor visits are, the larger the differences between the 1 month and 12-month recall periods. This implies that respondents with more visits tend to underreport them more often when the recall period is long. The same holds for people in moderate and good health. People in bad health often go to the doctor regularly, and remember these visits. More infrequent visits are more easily forgotten. Finally, we find that the results of our experiment are non-significant. Offering respondents personal landmarks, and putting these next to the timeline to improve recall, does not lead to smaller differences.

In practice, these findings may be useful when one is interested in estimating frequencies of behavior over an annual period. Breaking up the 'annual-estimate' question into twelve monthly questions helps to improve data quality. Asking about such frequencies at 12 separate occasions further helps, but this is unlikely to be feasible due to the large increased costs of more frequent data collection. In self-administered web-surveys this might however be feasible. Splitting up questionnaires into multiple shorter ones may not only reduce burden, but can increase data quality for specific survey questions as well.

Wednesday, May 20, 2015

Adaptive designs: 4 ways to improve panel surveys

This is a follow-up on why I think panel surveys need to adapt their data collection strategies to target individual respondents. Let me first note that apart from limiting nonresponse error, there are other reasons why we would want to do this. We can limit survey costs by using expensive survey resources only for people who need them.
A focus on nonresponse alone can be too limited. For example: imagine we want to measure our respondents' health. We can maybe do this cheaply by using web interviews, and then try to limit nonresponse error by using interviewers to convert initial nonrespondents. But what about measurement? If we use web surveys we largely have to rely on self-reports on the respondents' health. But if we use interviewers for everyone and do a Face-to-Face survey among all our respondents, we can use the interviewers to obtain objective health measures for respondents. These objective measurements could be much better than the self-reports. So face-to-face interviews may not be 'worth' the cost if we look at nonresponse alone, but if we also include the effects on measurement, it may be a viable interview option, if we reduce the sampling size.
I think a focus just on any one type of survey error can have adverse effects, and it is Total Survey Error as well as costs we need to keep in mind. Having said this, I really believe nonresponse errors in panel surveys are a huge problem. What could we do (and have others done)?

1.  Targeted mode. Some respondents are easy to reach in all modes, and some are difficult in all modes. There is also a 'middle' group, who may participate in some modes, but not others. I, for example dislike web surveys (because I get so many), but appreciate mail surveys (because I never get them). In a panel survey, we can ask respondents about mode preferences. Some studies
(here, here) have found that stated mode preferences are not very predictive of response in that mode in the next wave, as people indicate to prefer the mode they are interviewed in. This means we probably need a better model than just 'mode preference' to make this work.

Probably wants a different survey mode next time.
2.  Targeted incentives. We know some respondents are 'in it' for the money, or at least are sensitive to offers of incentives. In panel surveys, we can learn quickly about this by experimenting with amounts both between and/or within persons. For example, does it help to offer higher incentives to hard-to-reach respondents? Does that help just once, or is there a persistent effect? It may be unethical to offer higher incentives to just hard-to-reach respondents, as we then put a premium on bad respondent behavior. We could however use different metrics for deciding whom to offer the incentive. Nonresponse bias is a much better indicator for example. There is not too much we know about how to do this, although there is a nice example here.

3. Targeted advance letters. We know quite a lot  about the effects different types of advance letters have on subsequent response. Young people can for example be targeted with a letter with a flashy lay-out and short bites of information of the study, while older people may prefer a more 'classic' letter with more extensive information about the study setup and results.
The effects of targeted letters on response in panel surveys are often small, and only present for specific subgroups. See here, and here for two examples. Still, this type of targeting costs little, perhaps we can find bigger effects if we know what groups to target with what message. As with other targeting methods, we need a combination of data mining and experimentation to develop knowledge about this.

4. Targeted tracking. I am not aware of any survey doing targeted tracking. Tracking is done during fieldwork. Respondents who are not located by an interviewer (or advance letter which bounce), are sent back to the study coordinating team, after which tracking methods are used to locate the respondent at an alternative address. From the literature we know that it is mainly people who move house who need tracking. If we can successfully predict the likelihood to move, we could potentially save time (and money) in fieldwork, by putting cases into preventive tracking. We could also potentially use a targeted order of tracking procedures, as James Wagner has done.


Wednesday, January 21, 2015

why panel surveys need to go 'adaptive'

Last week, I gave a talk at Statistics Netherlands (slides here) about panel attrition. Initial and nonresponse and dropout from panel surveys have always been a problem. A famous study by Groves and Peytcheva (here) showed that in cross-sectional studies, nonresponse rates and nonresponse bias are only weakly correlated. In panel surveys however, all the signs are there that dropout in a panel study is often related to change. Those respondents undergoing most change, are also most likely to drop out. This is probably partly because of respondents (e.g. a move of house could be a good reason to change other things as well, like survey participation), but it is also because of how surveys deal with such moves. Movers are much harder to contact (if we don't have accurate contact details anymore). Movers are often assigned to a different interviewer. This will all lead to an underestimate of the number of people who move house in panel studies. Moving house is associated with lots of other life events (change in household composition, change in work, income etc.). In short dropout is a serious problem in longitudinal studies.

The figure below shows the cumulative response rates for some large-scale panel studies. The selection of panel studies is a bit selective. I have tried to focus on large panel studies (so excluding cohort studies), which are still existing today, with a focus on Western Europe. 

Cumulative nonresponse rates in large panel surveys (click to enlarge)

The oldest study in the figure (PSID) has the highest initial response rate, followed by studies which were started in the 1980s (GSOEP), 1990s (BHPS), and early 2000s (HILDA). The more recent studies all have higher initial nonresponse rates. But not only that. They also have higher dropout rates (the lines go down much faster). This is problematic.

I think these differences are not due to the fact that we, as survey methodologists, are doing a worse job now as compared to 20 years ago. If anything, we have been trying use more resources, professionalize tracking, offer higher incentives, and be more persistent. In my view, the increasing dropout rates are due to changes in society (the survey climate). A further increase of our efforts (e.g. higher incentives) could perhaps help somewhat to reduce future dropout. I think this is however not the way to go, especially as budgets for data collection face pressures everywhere.

The way to reduce panel dropout is to collect data in a smarter way. First, we need to understand why people drop out. This is something we know quite well (but more can be done). For example, we know that likely movers are at risk. So, what we need are tailored strategies that focus on specific groups of people (e.g. likely movers). For example, we could send extra mailings in between waves only to them. We could use preventive tracking methods. We could put these into the field earlier.

I am not the first to suggest such strategies. We have been tailoring our surveys for ages to specific groups, but have mostly done so at an ad-hoc basis,  never systematically. Responsive or adaptive designs try to use tailoring systematically, for those groups that most benefit from tailoring. Because we know so much about our respondents after wave 1, panel studies offer lots of opportunities to implement responsive designs.