Trying Hard to Engage

I find it strangely heartening that researchers are finally beginning to tune into the reality that many of the newly-recognized problems of online research are rooted in bad questionnaires.  High termination rates, high levels of satisficing, and falling response rates are at least partially blamed on questionnaires that are too long, too complex, too poorly implemented, or just plain boring.  The heart of the problem, it seems to me, is length and complexity, but those are client issues and difficult to affect in the short-term.  Implementation is easier to deal with.  One popular approach is the movement toward different kinds of answering devices, visual analog scales (a.k.a, slider bars) being a primary example.  I’ve ranted against VAS before, and will return to that topic in a later post.

For now I want to describe a paper I heard at GOR08 presented by someone from an interesting German company called Psychonomics.  They reasoned that because face-to-face interviewing gets such good data we should try to emulate in our Web surveys what we can of the face-to-face experience.  They had two ideas: (1) use questionnaires transitions to express a variety of different sentiments to respondents (e.g., gratitude, understanding, familiarity, etc.) and (2) use an avatar to "conduct" the interview.    I had seen the avatar idea tested before and in general it’s always a dud.  Same here.  Too many technical problems and too slow.  People are not fooled into thinking they are being interviewed by a real person.

But I had hopes that the messages might show some helpful impacts.  No such luck.  Respondents ran away from the survey with reinforcing messages at essentially the same rate as a survey without them.

Later I heard a paper about using stories to induce mood changes in online surveys.  Apparently there are well-established procedures in Psychology for doing this.  They are called Mood Induction Procedures or MIPs.  And it actually works!  I won’t go into the details, but it turns out that if you have people read a very sad story or a very happy story it carries over to how they answer survey questions.  Maybe we should think about that.