Andrew Cober has been working with an Energy client for whom we do a phone study. While monitoring the client heard respondents struggling some with questions like this:
Would you describe the price you currently pay for electricity as very reasonable, somewhat reasonable, neither reasonable nor unreasonable, somewhat unreasonable or very unreasonable?
So Andrew and the clients decided to run an experiment using an alternate format (known technically as "branching"). Half the respondents were randomly assigned to the question format above and the other half got a format that goes like this:
A. Would you desscribe the price you currently pay for electricity as reasonable, neither reasonable nor unreasonable, or unreasonable?
B. Is that very reasonable or somewhat reasonable?
This format is popular in political polling and in fact was invented by one of the pioneers in the field, V.O. Key. Probably the landmark study on the topic was done by Krosnick and Berent and showed that data collected with this branching technique had greater statistical reliability than the one question format. By strange coincidence I heard a paper at the last AAPOR Conference by Doug Rivers (a colleague of Krosnick’s at Stanford) that argued the Krosnick and Berent study was flawed because it ignored missing data, something that the branching format produces more of. Rivers showed that when you bring missing data into the equation, the one question format outperforms the branching format.
All of that may have a certain academic interest but it doesn’t help Andrew with his problem which is to figure out what to recommend to his client. He might start by citing the literature which seems to say that while respondents might struggle more with the single question format in the end it produces better data. In his test Andrew was seeing roughly equivalent levels of nonresponse (DK or Refused) across the two treatments. There were, however, some differences in the distributions as shown in the graph. For example, the branched format
produced a few more responses at the extremes. It’s difficult to know whether that is a good or a bad thing. But what would seem to me clearly to be a bad thing is the slightly elevated "no opinion" group. This is the combination of respondents who say "Neither reasonable nor unreasonable" and those who refuse to answer or say they don’t know. The single question format produces 11 percent in these combined categories while the branched version produces 16 percent. Andrew tested this with two different questions and produced roughly similar results for both.
So my advice to Andrew is to stick with the one question format because:
- The most recent research on political attitudes is finding that the one question format has greater statistical reliability.
- Your tests indicate that we get more non-committal responses with the branching format even though it produces more responses at the extremes.
- The branching format takes longer and therefore costs more in terms of interviewing time and respondent good will.
Someone else might look at these data and come up with a different recommendation, and another experiment in another domain might yield different results. After all, the stock answer to every methodological question is, "It depends." That’s what makes this survey methods stuff so intriguing.