Faster is better

It’s the college basketball
season and that means yours truly is spending way too much time in front of his
TV.  One of the more annoying commercials
that gets repeated over and over is this one by AT&T, driving home the message that faster is better.  At least on your iPhone.  It’s a sort of focus group with elementary
school kids.  All staged, of course. It
reminded me of the current buzz about System 1 and System 2 thinking, probably
best described in Daniel Kahneman’s excellent book, Thinking Fast and Slow, and what that might tell us about what is a
“good” survey question. 

Over about the last decade I had
the good fortune to work with a group of old friends (who happen to be
world-class survey methodolgists ) fielding experiments on web survey
design.  The three of them have
collaborated on a
book to be published in April that pretty much sums up what they learned
over the course of those experiments.   The experiments mostly consisted of varying the presentation of a set of
questions to see what changes those different presentations produced in how
people answered the questions.  One
important variable analyzed in almost every case was response time.  Presentations that allowed respondents to
answer quickly were generally judged to be better than those that took longer.

This sort of fits with what I
think I have always known about good questionnaire design.  When we can design a question for which a
respondent has a well formed and easily retrieved answer (System 1 thinking) we
get good data, at least in the sense that the answer is what the respondent
believes to be true and probably acts on. 
But the more respondents have to think (System 2) the shakier it
gets.  Or, in some cases, don’t bother to
think at all.  Look no further than
customer sat questionnaires and the difference between the top of mind opinion
you get when you ask about overall satisfaction first (System 1 thinking again)
versus when you ask it later after you taken the poor respondent through the
full attribute set (i.e., forced System 2 thinking).  

The folks arguing for
gamficiation of surveys questions seem to think that the longer someone takes
to answer a question the better the answer. 
While that may be true for certain types of questions, in most cases it’s
probably a bad sign.  Faster probably
really is better.