Back again at CASRO Panels. The afternoon is dedicated to survey routers. As this has unfolded it’s clear to me that it is the perfect way to end to the conference. It’s the logical endpoint. So let’s go back to the beginning and note that there seem to be four overriding themes to this conference.
The first was implicit in Kim Dedeker’s opening remarks about the focus on reliability rather than validity. It’s hard any more to find anyone whose head has been in the game to argue that panel research as we practice it today (with a few notable exceptions) produces representative samples or results that are projectable to target populations with any specific accuracy. That is at long last a settled issue. As Kim suggested, clients can deal with results that they know have some bias as long as those results are not jumping all over the place from survey to survey. For many of their purposes directional results are just fine.
The second is that the panel data quality crisis (which Kim sort of launched) is no longer the focus. Panel companies and research suppliers have developed a set of solutions to deal with the biggest issues and these are being implemented all over the industry. It may be too soon to pronounce the problem solved, but I think it’s clear that we are out of the woods on this one. There still is good and important research on this issue, some of it presented at this conference, but we seem to have figured this one out.
The third is the clear realization that there is wide variability in panels and it’s unwise to expect to get consistent results from panel to panel. One of the themes of the ARF research is to protect against variability by making sure that the panel you choose to work with as the depth to support the full run of your research. If the panel can’t sustain it and you are forced to change, you could be in for a rocky time.
Finally, the era of the panel as we have known it over roughly the last 15 years is rapidly closing. The old model of sending a bunch of invitations to the panel and directing panelists to a single survey is increasingly untenable. As people have been pointing out for years the panel model is not sustainable. The pool of willing respondents is limited and we need strategies that tap multiple sources to draw in the number of respondents we need to meet the demand. And so we need to create a nearly constant stream of willing respondents from panels, from river, from social networks, from IM, from sms messaging ,etc. .
Which brings us to routers. We need effective ways to allocate those respondents. These things have been around for over a decade and mostly used with river sampling, but they have been black boxes that most of us know very little about. Now getting routers right seems like a critical issue. We heard nice presentations from Western Wats and OTX on various approaches to routing. The main takeaway seemed to be that random assignment of Rs to waiting surveys is the most efficient in terms of sample utilization as well as the best way to moderate the diversity/bias across these multiple sample sources. But it’s not that simple. There are other issues as well. For example, we need routers that minimize respondent burden. Some routers keep sending Rs through multiple surveys until they find one that the R qualifies for. People can get trapped in these and are unlikely to come back a second time. Equally problematic is the difficulty of computing some standard metrics we are used to like response rate or contact rate. Or should we have willing Rs to more than one survey out of the same recruitment? What happens when the input stream varies by source? Are the types of screening questions that Jackie Lorch described in her paper a good idea? The more people talk about it, the more difficult the problem becomes.
So my main takeway for the last two days is that a dramatic change is upon us and it’s not clear just how ready we are for that change. Most of the buzz in the MR world over the last year or so has been about social media, MROCs, Twitter, etc. Panels were passé.
For better or for worse, panels are becoming a lot more interesting.