The fundamental problem with surveys stems from the fact that few people who write questionnaires have any serious training in the discipline. When I talk with groups who do survey research and ask, “How many of you have a book on your desk on how to write questionnaires?”, few raise their hands, and many can’t even name a book or an author. When I ask, “What experiments on survey research have shaped how you write questionnaires?”, most can’t name one experiment, and we have hundreds of studies spanning a 75-year history of experimentation on the survey response. Few questionnaires are properly pretested and few know what proper pretesting actually is. I am referring to the kind of pretesting, often referred to as cognitive interviewing, that is described so well by Gordon Willis in his book, Cognitive Interviewing: A Tool for Improving Questionnaire Design.
The fundamental problem is lack of training and expertise in questionnaire design. The industry is clamoring to reduce the length of surveys because so many people are now taking surveys on their smart phones. We are distressed by lower response rates and higher drop out rates. Yet the key answers to these problems have been with us all along.
If we organize research to support decisions and outline the information needed to support those decisions, we end up with a much shorter list of information needs, and thus, shorter questionnaires. If we recognize the difference between qualitative and quantitative information needs, and we implement both qualitative and quantitative methods, then we get much better data and again, shorter questionnaires, because we do not ask the questionnaire to do things it was not designed to do. There is nothing more torturous to a researcher like me than to see a questionnaire that is being asked to provide information that really requires a qualitative approach.
Further, if we do qualitative research first, then we know what to ask and how to ask it. If we exercise guidelines on how to write questionnaires, then we write questions that are clear, answerable, easy and unbiased. We write questions that make sense to respondents.
Finally, if we properly pretest our questionnaires–a task that can be done in a day or two–we find out from our respondents if our questions are working as intended, and we can adjust the words and phrases to make sure the questions make sense and the respondent experience is positive.
Survey length can be reduced by 50 percent just by following the guidelines referenced above. By making questions clear, answerable, easy and unbiased, we improve the respondent experience, and this affects completion rates and the quality of data.
Let’s please not skip the fundamentals of designing research to support decisions and learning how to write better questionnaires. It will require that we treat questionnaire design as a professional discipline. And that will require reading and training. There is no magic pill, like there is for everything else!
