Sample Blending

This is a euphemism coined by the good folks at Lightspeed Research for combining respondents from multiple panels into a single sample. This sometimes looks attractive when a single panel can’t deliver enough respondents either because of low incidence or a small geographic study area.  Sometimes "sample blending" is a deliberate design decision and, sadly, sometimes it’s a decision that a panel provider makes without bothering to consult their client.  One hopes the latter cases are few and far between.

Back in 2006 Lightspeed Research did a study to to identify the impacts, if any, of using multiple panels.  They used their own panel plus two others unnamed.  They interviewed 600 respondents per panel and in each case used quotas to get census-balanced demographics from each panel.  The questionnaire was mostly standard CPG brand awareness and use questions.  They found some differences, but nothing dramatic.  Most differences where in the single digits.  They conclude "that behaviorally LSR and the other two sample sources used for this test are similar."  To quote the Lightspeed person who gave me the report, "We did the study and proved that it’s OK."

Well, I’m not so sure.  One problem with the study is that it does not appear they did anything to try to dedup across panels.  We know that multiple panel membership is fairly common, 30% to 60% depending on the panel, and that people who belong to multiple panels also tend to be frequent participants.  I’m not about to try to compute the likelihood of panel overlap in a study with and N of 600 but it just seems that some effort ought to be made to deal with this problem.

More importantly, I think most of us believe that panel recruitment and management practices have a major impact on panel composition and on individual samples drawn from those panels.  In other words, we would expect every panel to have some bias based on recruitment sources and management practices, and understanding that bias is important.  For example, blending panels with differing SES biases might produce more differences in response patterns than using panels with more or less the same bias.

I don’t imagine this problem going away anytime soon.  Clients and researchers alike have developed a fondness for studies that are hard to do in any mode but Web.  Those studies are generally less expensive than other modes, have faster turnaround times, and use questionnaires that work best in a visual, self-administered format.  And staying with large, national or multi-state study areas is less and less practical.    Doing these studies well and producing results that clients can safely use to make their business decisions is going to require a more sophisticated approach than what Lightspeed has done in this study.