I had an email yesterday from a friend and colleague telling me that MarketTools was sending an email around intimating that I somehow had blessed their True Sample product. The claim is a bit of a stretch but I’ll save that for another day. More importantly, the friend went on to say that she had seen Andy Morrison’s interview in Research which was a little tough on online panels. Given that she and I have worked together on a number of panel quality initiatives she asked me if I would update her on my latest thinking about those and other similar initiatives. Hence this post.
Andy’s interview was all about the scientific perspective which says that there is simply no way using the standard statistical tools we all know and love to accurately estimate population parameters from a sample drawn from an online panel. The sample frame—the panel—is inherently biased. To begin with, not everyone is on the Internet and then not everyone on the Internet chooses to join a panel or, for that matter, the specific panel we might use to do a specific bit of research. The differences between people on the panel and the population at large go well beyond the demographics we typically use to describe “representativeness” and include a whole range of complex attitudinal and behavioral characteristics that thus far have escaped measurement. Probability sampling stipulates that the sample frame be a full list of everyone in the population of interest or if a subset (as in a multi-stage designs) systematically arrived at. Opt-in panels fail this test. There simply is no legitimate way using standard statistical methods to project estimates produced by a sample drawn from a panel back to the population as a whole, unless, of course, the panel is recruited using probability methods and includes the offline population.
All of which is not to say that we can’t generate great insights and do good research with opt-in panels. The challenge is to understand the bias involved and what impact if any it may have on the business problem we are trying to solve. In other words, we need to be sure that panel research is the best method for the problem at hand given the client’s problem, the target respondent, and the time and money available.
The panel quality initiatives now underway are something all together different. I see them as coming out of the TQM tradition rather than a scientific or statistical tradition. Ever since Kim Dedeker uttered her now famous “Two surveys a week apart by the same online supplier yielded different recommendations … I never thought I was trading data quality for cost savings” the industry has been focused on figuring out what went wrong and how to fix it. Viewed in TQM terms, we have too much variation in output (the survey results) and the way to fix that is to reduce the amount of variation in the inputs (everything that happens from panelist recruitment through to completion of a survey).
There is now a lot of research on research aimed at understanding best practices in recruitment, validation, panel management, sampling, survey design and so forth. Even as we learn more not everyone will agree on the best way to do these things, but once an individual panel company chooses its approach, standardizes on it, and operates consistently within those standard approaches survey results from that panel will be more predictable, replicable, and, in the eyes of clients, more valid. Initiatives like ISO 26362 and the ESOMAR 26 Questions take the next step by creating transparency so that an educated consumer can make informed choices about which panels to choose for a specific study.
I think all of this is absolutely terrific and the industry deserves enormous credit for how it has responded to what is admittedly a self-inflicted wound. I think research using panels will become much more reliable and predictable as a result of this industry-wide initiative. But it will not mean that all panels are the same or that the same study executed with two different well-managed panels will yield the same results. Nor will it mean that we can make stronger claims of representativeness or describe our results in terms of “true values” in the statistical sense.
Online has given us both a new set of research methods and an enlarged responsibility for more nuanced interpretation. It is more important than ever that we interpret results in the broader context of other available information on the same and similar topics. Fortunately, we live in an age rich with detailed and easily accessed information and a broader set of research techniques than ever before. Our challenge is to use them wisely and in ways that help our clients to understand their businesses better.