I'm at the CASRO Online Conference in San Francisco. This is one of my favorite conferences because it's all about people trying to make things work. It's generally not about grand pronouncements and dire warnings about the future. It's a conference that features people doing research on research, running experiments, and trying to solve the real world methodological challenges that we face today. It's taking new methods to the next step, smoothing out the rough spots, and making the practice of MR better.
All that said, Gayle Fuguitt, the keynoter, wants to talk about 2020 and beyond. Her key concept seems to be "the voice of authenticity." Technology now provides tools for people to express themselves, either indirectly (big data) or directly (social media). The industry's challenge is to create the roles to make sense of it for clients.
Second big idea: Clients like trusted partners and are disinclined to change. They also have norms they like to see maintained. So the opportunity for MR companies in trusted partner roles is to bring new solutions. She sees consolidation of "traditional MR methods" augmented by new online methods (communities, social media, etc.) driving growth. Above all, don't give up on delivering real value. So it's not a replacement strategy, it's an augmentation strategy. And it's based on leveraging a long-term relationship and continuing to deliver value.
This has been a sane, reasonable view on the evolution of MR. No hyperbole, just good, thoughtful advice based on 30 years in the industry. Nice.
Next up is Pete Cape, a guy I quote regularly on a whole bunch of topics. His topics is inattentive respondents. He's run an experiment with grids in which he put some classic traps — "click the response at the far right." He also had a version where he broke the grids into individual screens. His general point seems to be that people just have cognitive lapses. They lose focus on the task. Their minds wander. Nothing nefarious about it. But then people have different heuristics for answering when they don't have an answer. Some will choose the same answer (modal answer) all the time. Other wills use the last answer (frequency bias).
Overall, what Pete sees is about a 5% failure rate on the classic traps when there are separate questions. But the failure rate is much higher when the trap questions are in the grid. That 5% is normal, according to the cognitive psychology literature. So the problem here is not bad people, it's bad design. The grids are the problem.
As I was listening it occurred to me that his is just another chapter in the ongoing saga of finding the reasons why online has sometimes produced bad results. The usual suspects are online questionnaire design and the respondents, some of whom are gaming the system to get the incentive. Cheaters. But there is a third suspect that gets too little attention: convenience sampling from sources of unknown quality. I had thought that the industry was finally getting around to go after that with approaches like those being used by Toluna, YouGov, GMI and others. No evidence of that at this conference so far.
Now David Bakken is going to talk about sample size and whether it matters. He points out that sample size choices typically are driven by the desired precision of the estimates and the likely data collection costs. But David is a Bayesian so this will be interesting.
David has done a nice job of demonstrating how a Bayesian approach can be used to give some sense of how good an estimate is. He's done an equally good job of demonstrating how difficult it is. The thing about probability sampling is that it's easy to do and there are lots of readily-available tools to help you do it. Not so with Bayesian methods. But the ideas are really intriguing and I would hope to see more people experiment with them. Because what we are doing now, which is mostly pretending that our non-probability samples are just like probability samples, is not just wrong, it's not working.
The final presentation in this session is by Melanie Courtright and Annie Pettit. They're going to talk about scales. Risky stuff. But they've conducted an experiment to test various kinds of scales across samples from 10 countries. As it turns out their goal is to understand how response styles (ERS, ARS, MRS, etc.) play out across different scale designs and cultures. It turns about that we age we drift toward more extreme response styles. Of course, says the 65 year blogger, we are wise and know more than younger people!
They spend a little time testing some hypotheses from Hofstede having to do individualism and masculinity. Nothing there. Slight differences in response styles as scales get longer. When they looked at reliability they found no real differences across 8 batteries on different topics and found no significant differences. They found some of the classic differences in response styles by country.
I've been doing this on the fly, so pardon the escalation in typos.
More later.