Last week two posts on the GreenBook Blog, one by Scott Weinberg and a response by Ron Sellers, bemoaned the quality of online research and especially its sampling. And who can blame them? All of us, including me, have been known to go a little Howard Beale on this issue from time to time. We all know the familiar villains—evil suppliers, dumb buyers, margin-obsessed managers, tight-fisted clients, and so on. I was reminded of a quote from an ancient text (circa 1958) that a friend sent me a few months back:
Samples are like medicine. They can be harmful when they are taken carelessly or without adequate knowledge of their effects. We may use their results with confidence if the applications are made with due restraint. It is foolish to avoid or discard them because someone else has misused them and suffered the predictable consequences of his folly. Every good sample should have a proper label with instructions about its use.
Every trained researcher has an idealized notion of what constitutes a good quality sample. Every experienced researcher understands that the real world imposes constraints, that any one who says you can have it all—fast, cheap, and high quality—is selling snake oil. So we make tradeoffs and that’s ok, as long as we have “adequate knowledge of their effects.”
It helps to be an informed buyer, and ESOMAR has for about ten years now offered some version of their 28 Questions to Help Buyers of Online Samples to help. It’s an excellent resource, often overlooked. More recently, ESOMAR has teamed up with the Global Research Business Network to develop the soon-to-be-released ESOMAR/GRBN Guideline on Online Sample Quality (here I disclose that I was part of the project team). In the meantime, there is an early draft here. Its most prominent feature, IMO, is its insistence on transparency so that sample buyers are at least informed about exactly what they are getting and in as much detail as they can stand.
Granted, that’s not the same as “a proper label with instructions about its use.” That still is a job left to the researcher. If you can’t or won’t do that, well, shame on you. Getting depressed about it is not an option. Or, as another of the ancients has said, “You're either part of the problem or part of the solution.”
Comments
3 responses to “Online Sampling Again”
Hi Reg,
It always frustrated me when one or another portion of the research process is pointed out as having particular problems when in fact every part leaves much room for improvement. Anyways, two comments.
First, I really wonder what companies ‘validate’ their sample. Providing the same name name or address twice does not mean they match my drivers license. It’s really a reliability or memory measure.
Second, why does 3.6 exist is this guideline? If we’re going to add comments about survey design to a sampling guideline, then why not add comments about focus groups and moderators and every other component of research that includes sampling somewhere.
I’m only being partly rhetorical here.
In the end, I’m all for guidelines because some people really don’t know what they should be doing or looking for and guidelines are essential for that.
Thanks for the continued focus on this topic, Reg. I agree the ongoing tension between business realities and methodological rigor has always existed, and still does today. It seems this is a tough battle to wage for the smaller companies who aren’t dealing in large volumes, but compete with those who do. I agree with you, Annie, as well, in regards to the process parsing. Sampling today seems to have become a high-volume, often-automated, transactional enterprise. Is this a reflection of the researcher’s resignation about the notion of “quality”? Or maybe it’s a sense of security around the API connections, etc. that offer some comfort around validation, etc. Probably only subjective answers to that.
Nice breakdown. I enjoyed the Howard Beale reference.