The issue of ranking questions keeps coming up. In general, survey methodologists seem to favor rating style questions over rankings, especially if the number of items to be ranked or rated is large. Dillman says it best when he writes ". . .avoid asking respondents to rank large numbers of items from top to bottom. Such a task is quite difficult for most respondents to accomplish . . ." Beyond the difficulty for respondents (as if we needed another reason) rankings have the inherent weakness of not expressing the magnitude of differences in respondent preferences. In other words, we don’t know if the R prefers #1 over #2 by a lot or just a little. Hence the preference to ask the R to rate the items individually rather than put them in an order of preference.
Randy Thomas at Harris Interactive (and thy guy from whom I stole the title for this post) did an interesting experiment in which he tested rankings vs. ratings in a Web study. I have this paper in hard copy and can make a copy for anyone who might be interested. I’ll simplify the design by describing it as assigning one set of Rs to rank a set of items and another set to rate them, even though it’s a lot more complicated than that. Some got five items and some got 10. He tracked the time it took to do the task and asked a couple of follow-up questions of Rs about difficulty. The key results:
- There were no real differences in outcomes, that is, ratings and rankings produced essentially the same results.
- The ranking tasks took longer, Rs reported them to be more difficult, and Rs perceived them as being less accurate.
- Rs assigned to the 10 item ranking condition of the experiment found them more difficult and the outcomes were less comparable to ratings than those assigned to the five item condition.
So the take way on this would seem to be to avoid ranking questions and use ratings instead on the grounds that rankings don’t provide us with significantly better data but they make the Rs task harder, never a good thing. In addition, ranking questions get increasingly more difficult as the number of items to be ranked increases.
So if a client comes to you with a questionnaire draft that has ranking questions in it I would suggest the following:
- Try to convince him or her to use rating questions instead.
- If your client insists on rankings and the number of items to be ranked is large (i.e., more than five) then try to convince him or her to ask the R to choose only his or her top three. Trying to rank eight or 10 items without ties is really difficult for an R, and the programming effort to check for ties and help the R correct errors is significant.
- From a programming perspective, the best approach is to have a series of questions in which the R selects his or her first choice and then gets the list back again with the first choice removed and is asked to make a second choice.
It also goes without saying that we should never use ranking questions in an aural mode such as a telephone survey. They are tough enough when you can see the items to be ranked. Doing all of that in your head based on what you heard is more than most can manage.