I’m here in Hamburg at the 10th General Online Research Konferenz or simply GOR08. The morning featured a keynote from Randy Thomas, a veritable Web research machine from Harris Interactive. His talk was mostly around research he has reported on previously, but he did make four interesting observations about online as compared to telephone that I think represent something of a consensus to which the MR industry has evolved:
- Mostly, online compares favorably with telephone even without weighting schemes like propensity scoring. He didn’t qualify that much but I would add the qualifier: "if you are working with online samples balanced to reflect the basic demographics of the population."
- One clear exception is attitudinal scales where online consistently delivers lower scores than telephone. This most people ascribe to differing use of scales in the visual as opposed to aural mode.
- Another clear exception is topics with a social desirability bias. It might be something as simple as health status or as complicated as illegal acts, but anytime a survey asks a question the answer to which might put us in an unfavorable light we are more likely to be truthful in the relative anonymity of a self-administered survey like Web than when a human interviewer is involved. For example, I have seen a number of comparisons that consistently show higher rates of smoking in Web survey than on the telephone.
- The final exception is topics that might correlate with people’s proclivity to go online in the first place. This might be attitudinal or demographic. For example, we might expect serious differences in Web vs phone if the survey topic were attitudes toward personal computing, or where we are interested in demographic subgroups that we know have low Internet penetration (e.g., elderly or minorities).
These four observations are probably a pretty good framework for thinking about when a Web survey is a good idea and when it is not. That said, there is no real underlying science with Web in the same way we have probability theory underlying telephone so we should always be wary of exceptions. Although Randy did not tie the two together he made the observation that having been trained as a psychologist his instincts were always toward replication, that is, find the truth by repeating the survey over and over but with different types of samples. He was surprised when he got to Harris and the political science types there saw the way to the truth through survey once and then weight. There probably is a lesson in there for how we should be thinking about online surveys.