This is essentially a query in my email today. Its full text: "MSI has been using Internet panels such as e-Rewards for consumer research. What should MSI’s response be to clients who question the bias associated with using these panels? "
A good place to start is to conceive of an Internet panel as a sampling frame with some serious coverage problems. It can only contain people who have Internet access and right now roughly a third of US households don’t. That would be manageable if Internet access were distributed randomly in the population but, of course, it is not. In general, the elderly and lower SES groups have less access than other demographic groups and so we have one obvious level of bias. But panels also do not include everyone who is on the Internet, only those who have volunteered to do surveys. And so we have a second level of bias. Not only do these panels not represent the full US population, they also don’t represent everyone on the Internet. In sample speak, every member of the population does not have a known and non-zero chance of being selected for a survey, so there is bias. Even the most enthusiastic online survey evangelist will admit this. Some, like Harris Interactive will claim that they understand that bias well enough to correct for it, but that’s a pretty tall order.
Acknowledging the bias in Internet panels does not necessarily mean that we should not use them, but we should be sure to consider the bias in the context of how the client wants to use the data and to make sure there is a good fit. In general it probably is a bad idea to use an Internet panel as the source when your goal is to accurately estimate some characteristic of the population such as satisfaction, product ownership, or brand awareness. Since the sample is not truly representative, you can’t expect to generate an estimate equal to the true value in the population.
But an Internet panel might be a good choice if, for example, the client is interested in how individual characteristics correlate with satisfaction, how different attitudes and circumstances affect product purchase decisions, or what types of brand attributes appeal to different kinds of people. They also can be helpful in studies that trend changes in attitudes and behaviors over time, even though the absolute measures for those attitudes and behaviors might not match up well with studies using true probability samples.
And while sample bias is an important issue, other factors such as cost, turnaround time, the complexity of respondent task, or the need to present images or other graphical material are important considerations. There are some types of studies that simply cannot be done in a survey mode other than Web.
Finally, the size of the geographic area to be studied can also be a factor. Many panels simply are not large enough to provide sufficient sample for smaller geographic areas such as MSAs . Or a panel may have to use every bit of sample that it has for that area, and in the process create a much more biased sample than would be created for a national study. Most panel companies offer the option to control demographic bias by creating samples that are balanced to Census demographics. For smaller geographic areas, this is sometimes impossible because they just don’t have enough sample to start.
The academics would argue that the advantages or probability sampling outweigh even the disadvantages of the very low response rates that have become the norm in telephone surveys. In some absolute sense, this might be true. But our design decisions should be guided by a wider set of considerations foremost of which is its ability to provide the client with valid, actionable results at a price s/he can afford and in a time frame where those results can be useful.