Tools & Tips

Reducing Bias: The Effect of Category Ranges on Responses


One of the most common tasks in survey research is to ask respondents to select from a list of numeric ranges. For example, we might ask how many hours per day do they watch TV, how much time do they spend on your computer, or how many hours per day they study. Several authors have shown through experimentation that the range of response options we offer respondents has a substantial impact on their answers.

For example, J. D. Smyth, D. A. Dillman, and L. M. Christian, 2007*, conducted an experiment where students were randomly assigned to receive the following question:

How many hours per day do you typically study?”

One group had a Low Range of response options; another had a High Range of response options; while another was given an Answer Box to write in their response. The response options are listed below.

 

   Low Range    High Range         Answer Box
___ Less than .5 hours per day

___ .5–1 hour

___ 1–1.5 hours

___ 1–2 hours___ 2–2.5 hours

___ More than 2.5 hours

___ Less than 2.5 hours

___ 2.5–3 hours

___ 3–3.5 hours

___ 3.5–4 hours

___ 4–4.5 hours

___ More than 4.5 hours

Screen Shot 2015-06-22 at 9.00.58 AM

If all that mattered were the question stem, results from the three groups would be the same. However, as a number of researchers have shown, respondents look to the answer choices to gauge their responses.

Below is a table showing some of the results.

 

Low Range High Range Answer Box
Up to 2 ½ hours 70% 29% 42%
Over 2 ½ hours 30% 71% 58%

So how many hours per day do students typically study? If you want to know what percent study up to 2 ½ hours per day, it is either 70%, 29%, or 42%, depending on the range of response options you offer. If you want to know what percent study over 2 ½ hours per day, it is either 30%, 71%, or 58%, depending on the range of response options you offer. Clearly, the range of answer choices offered to respondents affects responses. This issue has been studied with other activities, such as computer usage and watching TV.

Several authors have suggested ways to address this effect. You should always have the range of response options reflect the distribution of the real answers in the population. If you don’t know, then start with an answer box to eliminate influencing answers by the range of response options you offer respondents. You might start with an answer box the first time you ask this question then use those answers to determine the range of responses for subsequent studies.

This is one of many experimental effects that all of us survey researchers should know. I am working on a paper on the topic of the top ten experimental effects all survey researchers should know. If you have ideas of other experiments that should be on the list, please let me know.

* “Context Effects in Web Surveys: New Issues and Evidence,” by J. D. Smyth, D. A. Dillman, and L. M. Christian, 2007, in A. Joinson, K. McKenna (pp. 427-443), New York, NY: Oxford University Press.


Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>