In previous posts I’ve talked about the tendency for respondents in self-administered surveys like Web and mail to choose answers from the top of the answer list, especially if the list is long. The implication is that respondents are not reading the full list, or at least are putting more cognitive energy into items at the top and less for those further down.
Now one of our colleagues in the Joint Survey Methods Program (Mirta Galesic) has written a paper reporting on the results of an eye-tracking study designed to help us understand how respondents process Web questionnaire pages. The results are pretty startling. In one experiment, Miirta showed half the respondents a single choice item with 12 possible answers and the other half saw the same question but with the answers in reverse order. We already had done the experiment online and found that respondents more often chose from the top half of the list , regardless of the order. In the qualitative phase Mirta used eye-tracking technology to get a clearer view of how respondents actually read the questions. The two images below are gaze charts that show where respondents looked and for how long. The warmer the color, the longer they looked. While some respondents read the entire list, it’s obvious that the top half got the bulk of the attention and hence more answers were selected there.
So what to do? The traditional fix here is to randomize the answers, and while that will make your distribution look better it doesn’t solve the basic problem of respondents not fully considering the full set of answers and then accurately reporting their opinion or behavior. The best solution would seem to be to stay clear of long lists.
One potential piece of good news here: There is some indication that this problem may not be quite so severe when the answers are on a scale. It seems that respondents infer the scale order after looking at the first few answers and then "jump" to an answer. But more research is needed to get a clearer fix on what might be going on.