One of our Energy clients recently found herself in a tough spot. We have been doing their customer sat research for years and putting those results in the context of our benchmarking studies to show how they rank compared to other similar utilities. All of this work has been done by phone. Now along comes J.D. Power with a study that shows their rank to be considerably worse than what we have been telling them and her management is asking tough questions. The J.D. Power study was done by Web using a variety of online panels. So she called us to get our take on this. It caused me to write up a brief email summarizing what I see as the issues. An edited version of that email follows.
First, in any transition of a study from phone to Web one has to be concerned about mode effects, either because of the change from aural to visual or because of the presence or absence of an interviewer. In the case of the former there is some research to suggest that on long scales respondents in the visual mode will tend away from selecting extreme responses. In sat surveys this can lead to lower scores on the Web than on the telephone. We have tested that on a key accounts study for a Utility and not found it to be a problem. The presence/absence of an interviewer can lead to social desirability effects, and while there is substantial evidence of this in areas such health or other sensitive topics I know of little evidence that it operates where satisfaction is concerned. Bottom line is that the differences you are seeing are probably not due primarily to mode. Or at least that’s my view based on what I know at the moment.
Second, sometimes these transitions from Web to phone are more than just mode changes, they also can be sampling changes which is what you are facing. Switching from a probability sample randomly drawn from your client list or from a high coverage frame such as D&B to a volunteer Web panel is very challenging. In a probability sample from a good frame everyone has an equal chance of being selected and therefore you are more or less guaranteed a representative sample. But with a Web panel, significant elements of the population are under represented or not represented at all. And while the Web vendor can draw a sample that may appear to represent your customers on some key characteristics, they can’t ensure that the sample is representative on all of the things that might matter–such as attitudes and behaviors–and so there is bias. Understanding that bias and measuring it so we might adjust for it is extremely difficult. And in fact, the academic survey community is largely in agreement that even in an age of very low response rates, the advantages of probability sampling outweigh the disadvantages of a low response rate. Bottom line here is that I think the differences you are seeing are probably due to the shift in sampling strategy rather than mode.
Finally, there are important issues with the panels themselves that go beyond the fact that they are non-probability, convenience samples. These are people who have signed up to do surveys. Sometimes they are not who they pretend to be and sometimes their extensive experience as survey takers leads them to answer differently from respondents who may do only one or two surveys a year. Using them well to produce valid research requires a strong QA methodology. I would be interested to know who J.D. Power has used as a sample source and what steps they took to clean the data.