One of the guiding principles of Web questionnaire design should be to constantly look for ways to make it easier for people to answer our questions. We have just moved into the analysis stage of our latest set of experiments with the good folks at ISR. One of the experiments on this round looked at whether it was helpful to break up long answer lists into categories and label them.
One of the questions we tested was educational achievement. Here is a screenshot of the standard question.
We wondered if we could make it easier for people if we put the list into categories or columns. We tested a couple of formats that grouped the answers under headings. Two examples are shown below.
Our results demonstrated pretty convincingly that the use of headings made it harder not easier for people to answer. It took respondents almost twice as long to answer the question when we used headings, in part because many people thought they had to select an answer in each group. One in five respondents tried to select multiple responses when we used headings compared to just 3 percent in the standard single column presentation.
To my mind this reaffirms a key principle we should ruthlessly employ as we try to ease respondent burden and increase engagement: never assume how people will react to a design feature and always make design decisions based on data.