For about the last week or so I have been getting regular calls on my home answering machine from Governor Mike Huckabee whom I gather is once again running for President. While it seems to be the Governor's voice it also is a recording inviting me to do a survey by IVR. Somewhere back in my blog archive there are a couple of posts about what the politicos like to call "robo polls," that is, RDD samples autodialed with a survey in which the questions are recorded and played back while respondents answer using the telephone keypad. I wrote those posts because a colleague had asked me about the validity of the methodology and specifically with regard to representivity. I did my best to discredit it.
My posts hardly landed a blow compared to the job that Nate Silver of fivethirtyeight.com has done in a recent post exposing the methodology of the most well known of the robo pollsters, Scott Rasmussen. There are the usual problems with Rasmussen's methodology–limited calling windows (5:00PM-9:00PM weeknights), no call backs, very low response rates, taking the first person who answers the phone, etc.–but Nate has taken things a step further by demonstrating that at best these calls can only reach about 50 percent of the population. In fact, it's probably much lower. Putting together the most recent data on cell only households with time use data from the American Time Use Survey Nate shows that the likelihood people are home or if they are home the likelihood they will answer the telephone is somewhere around 50/50.
I would like to think that this is the last nail in the coffin of robo polls but I know better. It's just one more example of how little the consumers of survey research, whether in the media, the general public, or even our MR clients, understand about the underlying scientific principles of what we do. For this we probably only have ourselves to blame.