Monday, November 3, 2014

Delphi-study on adjustment techniques in surveys

Some people in my university department (including me) have started a study to how people deal with non-probability surveys. The study is led by Hannah Emerson, an MsC student who is writing her thesis on the topic. What kind of adjustment techniques do people use? What kind of error in non-probability samples do researchers try to adjust for? How do they make decisions on how to do that? These are rather open-ended questions, and so we have opted to do a Delphi-study. In 3 rounds we hope to achieve consensus among survey methodologists on when adjustments make sense and how they should be done and evaluated.

The first round of the Delphi study just started. See the announcement below. Your participation in the study is much appreciated.


In the literature of survey methodology, different weighting techniques are presented for surveys that are based on non-probability samples. It is often unclear, however, how successful the techniques are in reducing bias and as far as what types of bias each technique aims to correct. 
 It is also unclear how different agencies (academia/science, government and market research) apply the techniques and how these differences extend throughout different countries.
With the following survey we intend to gather this information and summarize it via use of the Delphi Method. The Delphi method aims to gather a consensus amongst ‘expert panelists’ in a series of 3 questionnaires.  
In the first wave, our aim is to collect data on the types of weighting adjustment techniques used in nonprobability samples.  In the second wave we will present you with the answers of your fellow panelists and ask if you agree with the ideas of adjustments as well as show you where your response lines up with theirs.  In the third wave we will see if we can get consensus among the panelists. 
Our hope is that we can come together to make sense of the ‘big black box’ that surrounds the weighting techniques used in web surveys. Please consider participating in order to make this happen. If you yourself are not the one who adjusts surveys for bias, we ask you to please forward this email to the person who does. Thank you for your time and consideration for participating in this much needed exploration.
Click on the following link to access the survey, which will take approximately 10 minutes to complete:

with kind regards and many thanks,
Vera Toepoel, Peter Lugtig, Barry Schouten and Hannah Emerson