Yesterday I took part in the inaugural meeting of the CASRO Panel Integrity Task Force. I was impressed on two fronts.
First, there is a clear determination to be in sync with the ARF’s Online Research Quality Council, the much-watched panel quality initiative launched last year. Two members of the CASRO Task Force are also members of the ORQC and there is clear determination to try to reach across the various professional and trade association boundaries and to speak with a reasonably unified voice for the industry as a whole. This is a lot tougher than it might sound.
Second, and probably more importantly, both CASRO and ARF are determined to go beyond the usual platitudes and create fact-based guidelines that may actually have some meat in them. ARF is in the midst of some very interesting research on research that hopes to make some clear statements about the seriousness of some of the central problems of panels (such as multipanel membership). This is a healthy sign that the industry is moving away from hand wringing and toward answering the essential question of how various behaviors on the part of online panelists may be affecting results.
Now here is the really interesting part: a major focus of ARF’s research will be respondent engagement. This would seem to be a clear statement that while there are a set of quality problems that fall on the panel companies, the "inattentive" problem (a.k.a. satisficing) is a problem of survey design. I expect them to take aim at survey length, task complexity, and interface design and argue that researchers are creating their own quality problems through respondent-hostile design.
On the one hand, I say, "Good for them." On the other hand, these are probably even more difficult problems to solve than the fraudulent and hyperactive respondent problems that we have been worrying about over the last few years. Survey length was an intractable issue long before online. The ability to give respondents complex tasks such as discrete choice has been a major driver of online’s growth. While there is no shortage of companies claiming to produce more engaging survey designs using Flash and cute answering devices the research on the impact of these interfaces is not even close to being conclusive.
Good things are happening, but we have a long way to go.