I’ve spent the last two days at the Association for Survey Computing’s conference they’ve called Challenges of a Changing World. One continuing theme from the earlier workshop on the Interview of the Future is the use of more interactive features in Web surveys.
On the one hand, you might think that it’s about time. Almost from the start of the Web survey movement its evangelists talked about the Web’s ability to dramatically change how surveys are conducted by creating a more interactive and engaging experience for respondents. We finally are seeing things like sorting exercises that you accomplish with a mouse, slider bars and temperature gauges of all shapes and colors, increasingly elaborate virtual store shelves, and, the most obvious of all, video of an interview reading the questions to the respondent.
Then again, while these things really are very cool to see and their creators are completely enchanted with them there is not a whole lot of evidence that they help in any way. The well-designed experiments I’ve seen mostly conclude that these devices don’t really improve data quality in any measurable way. Sometimes the results are a little different, but it’s hard to tell whether those differences are positive or negative. Sometimes respondents give more positive feedback on survey design. And sometimes they actually seem to hurt us in any of a number of ways. For example,
- Many require special browser plug-ins or enabled software such as Java that can cause them to load slowly or not at all, thereby forcing some respondents (usually a small percentage) out of the survey.
- Some classes of respondents just don’t like them and so you see differential breakoff rates that might create an age, gender or attitudinal bias. For example, older respondents may not like them much at all while young males think they are the coolest possible thing.
- They tend to take longer to use than conventional interfaces like radio buttons and extending the length of the interview is never a good thing.
- They add to the programming load making the surveys that use them more expensive and longer to get to field.
My personal (totally untested) hypothesis is that these sorts of things are still not in the mainstream of how people interact with the Web. Whether you’re talking about buying books at Amazon, searching Wikipedia, or even posting to YouTube the Web is still mostly about filling in forms. Why should a survey be different? As long as these gadgets are not standard fare on the Web generally I don’t think we can expect respondents to embrace them in our surveys. This may be a generation thing, as one of the enthusiasts suggested, or may come hand-in-hand with Web 2.0 as suggested by another. We will just have to wait and see.
In the meantime, I hope the testing continues. It’s very interesting stuff. We have another test with slider bars in the field now and it’s going to be interesting to see how it turns out.