Tools & Tips

Get Better Data #4: Use Forced-Choice Instead of Check-All-That-Apply


One of the most common tasks we ask of respondents is to ask a question, provide a list of answer choices, and tell respondents to check-all-that-apply. Asking questions using the check-all-that-apply format often leads to incomplete answers. It is better in most cases to use a forced-choice format, where respondents are presented with a list of answer choices and asked to select yes or no for each item.

In one of the classic experiments comparing forced-choice versus check-all that apply formats, students at Washington State University were asked which sports they considered themselves a fan of. Students were randomly assigned to receive either the forced-choice or check-all-that-apply format of the question.

These two formats and the question are listed below:

Forced-Choice

Do you consider yourself to be a fan of each of the following sports?

Yes No

O     O     Men’s baseball

O     O     Women’s basketball

O     O     Men’s basketball

O     O     Women’s cross-country

O     O     Men’s cross-country

O     O     Men’s football

O     O    Women’s golf

O     O     Men’s golf

O     O     Women’s rowing

O     O     Women’s soccer

O     O     Women’s swimming

O     O     Women’s tennis

O     O     Women’s track and field

O     O     Men’s track and field

O     O     Women’s volleyball

 

Check-All-That-Apply

Which of the following varsity sports would you consider yourself to be a fan of? Please check all that apply.

O   Men’s baseball

O   Women’s basketball

O  Men’s basketball

O   Women’s cross-country

O   Men’s cross-country

O   Men’s football

O   Women’s golf

O   Men’s golf

O   Women’s rowing

O   Women’s soccer

O   Women’s swimming

O   Women’s tennis

O   Women’s track and field

O   Men’s track and field

O   Women’s volleyball

 

Source: “Comparing Check-All and Forced-Choice Formats in Web Surveys,” by J. D. Smyth, D. A. Dillman, L. M. Christian, 2007, and M. J. Stern, 2006, Public Opinion Quarterly, 70, pp. 66-77.

In the forced-choice format, respondents spent more time answering the question and selected more items than when responding to the check-all-that-apply format. These authors tested several other questions, and in all cases, respondents spent significantly more time answering the question and selected more items when using the forced-choice format than when using the check-all-that-apply format. These findings were consistent for both Web and paper surveys.

The forced-choice and select-all-that apply formats are simply different tasks. As Dillman, Smyth and Christian (2014) explain, “Whereas the check-all format provides respondents with a group of items and asks them to choose those that apply from the group, the forced-choice format requires respondents to make an explicit judgment about each item independently.”

Additional analyses of these experiments suggests that the check-all-that apply format leads to satisficing—a term that refers to respondents satisfying the requirements of the question by selecting a few items with limited time and energy. They are, in a sense, just getting by. In the forced-choice format, respondents are required to give full attention to each item. Hence, they take more time and select more items.

In most cases, it is better to use forced-choice rather than the check-all-that apply format.

Source: D. A. Dillman, J. D. Smyth, and L. M. Christian, 2014. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method. Hoboken, NJ: John Wiley and Sons.


Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>