The Barcelona conference was mostly a gathering of panel companies gathered together to talk about their business. Randy Thomas of Harris Interactive and Programme Committee Chair wrote an interesting "editorial" that was delivered as a sort of Foreword to the set of published papers. It provides a good concise statement of the rational for panels that might be subtitled, "Why We Need To Do This And Why It’s OK."
The "Why We Need To Do This" part is familiar to all of us. For a whole lot of reasons response rates in traditional modes have fallen to the single digits and in the process surveys have gotten significantly more expensive, take longer to execute, and are less and less representative of the population. Panels were invented to solve these problems by making the it faster and cheaper to interview lots of people.
The "Why It"s OK" part is tougher, but Randy argues that we can assure generalizablity (defined as "congruence in results between modes and sample sources") if four factors are present:
- "The topic being measured is not closely related to how a person was chosen to be in the sample." In other words, don’t do online surveys about Internet or online issues and expect your results to reflect the population.
- "A sufficient number of respondents participate." This, of course, is the traditional argument that the larger your sample the less error in your results. Probability sample devotees call this "The Literary Digest Fallacy," a reference to one of the colossal failings in the history of survey research, the botched call of the 1936 US presidential election. So, yes, larger sample sizes are better than smaller sample sizes, but a large sample alone does not guarantee that it represents the population of interest.
- "Respondents are sufficiently diverse and in general proportionate to the people to which we seek to generalize our results." It’s hard to argue that point, and there probably is a corollary about post stratficition adjustment (aka weighting).
- "Respondents have sufficient motivation and cognitive capacity to complete a self-administered survey."
This last point is looking increasingly like an Achilles heel for the panel industry and it occupied a good deal of attention at the conference. Research buyers, which is to say clients, have pretty much accepted the "Why We Need To Do This" rationale and points 1 thru 3 above. But doubts have begun to emerge around number 4, the panel companies know it, and the smart ones are doing their best to get out ahead of it. So far their efforts have been less than convincing, and I expect that things will get worse before they get better. But in the end the responsibility for the quality of the research we deliver to our clients rests with us as researchers, not with the panel companies. Interesting times.