One of Don Dillman’s Principles for Writing Survey Questionnaires says: "Avoid specificity that exceeds the respondent’s potential for having an accurate, ready-made answer." It’s a good principle but hard to follow because clients often want to know how many of this or how many of that. So we ask it anyway and instead of getting an accurate number from every respondent we get an estimate, an educated guess. One obvious indicator that we are getting an estimate rather than an accurate number is "heaping," that is, lots of numbers that end in a five or a zero. No one ever seems to answer "13" but we have lots of people answering "10" and "15."
In The Psychology of Survey Response Tourangeau, Rips, and Rasinski describe a technique they call "Decomposition" that can be useful in those instances where a respondent might not have a "ready-made answer." They suggest that in place of the big "How many" question you ask about smaller subcategories that eventually total up to the big one. Example: Rather than ask, "How many sporting events did you attend last year?" you might ask "How many baseball games? How many football games? How many basketball games?" etc.
Too often we do this in the opposite direction, and that can lead to inaccurate data. Example:
Q1. How many patients have you treated in the last six months with condition x?
Q2. Thinking about the patients you’ve treated with condition x in the last six months, how many did you treat with each of the following treatments? (Total must sum to the number of patients reported in Q1)
The theory behind Decomposition would suggest that you get a better count of patients from Q2 than you do from Q1, and indeed questions like Q1 almost always show lots of heaping. To make matters worse, we insist that the respondent juggle his numbers in Q2 to match Q1, even though the total in Q2 may be more accurate.
Decomposition is a technique worth considering in questionnaire design, especially if you believe the respondent can be helped to a more accurate answer by stimulating his or her recall in this way. Some respondents may take themselves through this kind of exercise in their head to get to a number without our help, but sadly not every respondent is that well motivated.