One of the most common tasks we ask of respondents is to ask a question, provide a list of answer choices, and tell respondents to check-all-that-apply. Asking questions using the check-all-that-apply format often leads to incomplete answers. It is better in most cases to use a forced-choice format, where respondents are presented with a list of answer choices and asked to select yes or no for each item.
In one of the classic experiments comparing forced-choice versus check-all that apply formats, students at Washington State University were asked which sports they considered themselves a fan of. Students were randomly assigned to receive either the forced-choice or check-all-that-apply format of the question.
These two formats and the question are listed below:
| Forced-Choice Do you consider yourself to be a fan of each of the following sports? Yes No O O Men’s baseball O O Women’s basketball O O Men’s basketball O O Women’s cross-country O O Men’s cross-country O O Men’s football O O Women’s golf O O Men’s golf O O Women’s rowing O O Women’s soccer O O Women’s swimming O O Women’s tennis O O Women’s track and field O O Men’s track and field O O Women’s volleyball |
Check-All-That-Apply Which of the following varsity sports would you consider yourself to be a fan of? Please check all that apply. O Men’s baseball O Women’s basketball O Men’s basketball O Women’s cross-country O Men’s cross-country O Men’s football O Women’s golf O Men’s golf O Women’s rowing O Women’s soccer O Women’s swimming O Women’s tennis O Women’s track and field O Men’s track and field O Women’s volleyball |
Source: “Comparing Check-All and Forced-Choice Formats in Web Surveys,” by J. D. Smyth, D. A. Dillman, L. M. Christian, 2007, and M. J. Stern, 2006, Public Opinion Quarterly, 70, pp. 66-77.
In the forced-choice format, respondents spent more time answering the question and selected more items than when responding to the check-all-that-apply format. These authors tested several other questions, and in all cases, respondents spent significantly more time answering the question and selected more items when using the forced-choice format than when using the check-all-that-apply format. These findings were consistent for both Web and paper surveys.
The forced-choice and select-all-that apply formats are simply different tasks. As Dillman, Smyth and Christian (2014) explain, “Whereas the check-all format provides respondents with a group of items and asks them to choose those that apply from the group, the forced-choice format requires respondents to make an explicit judgment about each item independently.”
Additional analyses of these experiments suggests that the check-all-that apply format leads to satisficing—a term that refers to respondents satisfying the requirements of the question by selecting a few items with limited time and energy. They are, in a sense, just getting by. In the forced-choice format, respondents are required to give full attention to each item. Hence, they take more time and select more items.
In most cases, it is better to use forced-choice rather than the check-all-that apply format.
Source: D. A. Dillman, J. D. Smyth, and L. M. Christian, 2014. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method. Hoboken, NJ: John Wiley and Sons.