Kristin Luck writes a column for RBR and for two issues in a row she's worked the worry beads about what she sees as declining interest in online panel data quality. In the current column Kristin draws heavily on a reaction to her first column by Jackie Lorch. Jackie summarizes two legs of the online data quality stool (respondent validation and better survey design). Kristin points out that there is still a problem, quoting an anonymous client who says, "There hasn't been the progress with online research that many in the industry think. We have internal tests that clearly demonstrate that."
As I have written in this space before, respondent validation and better survey design while necessary are not sufficient to really fix the problem with these sample sources. We also must address the third leg of the stool: the selection bias built into pretty much every online panel and intercept method. Every time the issue of online panel data quality comes up that graph from the first stage of reporting of the ARF Foundations of Quality Initiative showing all of the variation in results across the 17 panels pops into my head and I ask myself whether improved respondent validation and better survey design can make this problem go away. I think not.
We need a lot more focus on the third leg of the stool and there are encouraging signs. More and more researchers are concluding that simple weighting by demographics is not enough. Propensity weighting has been widely discussed although not broadly used, and while evaluations of it in academic journals have not been all that positive additional experimentation could produce a breakthrough. More promising, I think, is experimentation with new approaches that focus on the sample selection stage and the use of high quality probability surveys (.e.g, the American Community Survey, the General Social Survey) as a reference point for selecting representative samples from single or multiple panels.
Kristin may be worried but I am more optimistic than at any time since online panels first came on the scene almost 15 years ago.