This is a question we are fond of asking, and it has emerged as a major focus as we look to migrate healthcare-related telephone studies to the Web. The full question looks like this:
How would you rate your health right now?
- Excellent
- Very good
- Good
- Fair
- Poor
When we did you full-fledged test of RDD sample vs. an SSI sample back in 2004 here is what we got even after weighting to account for demographic differences:
These data suggest that the telephone respondents may be healthier than Web respondents, or at least they say they are. And while it sort of evens out if you combine "Excellent" and "Very Good" the 12 point gap on "Excellent" is troubling.
These data also bear a striking resemblance to data from a similar experiment conducted at RAND and reported in Schonlau et
al., "A comparison Between a Propensity Weighted Web Survey and an Identical RDD Survey," Social Science Computer Review, 2004, 22, 128-138.
Here are their data and our data combined with the phone results in blue and the Web results in yellow and orange. Pretty interesting.
Moving forward to 2005 we are looking at data from a Web experiment we are doing on a study of health plan choices by employees of a major manufacturing company. Here are those data:
The distributions are somewhat different but the general finding of poorer health reports by Web respondents holds up.
So what’s up? Well, when I presented the PULSE results at a conference in 2004 no less a luminary that Don Dillman said, "Reg, you have a mode effect. People shy away from choosing extreme values in the visual mode." Could be. There is some evidence of that in other research. But in the 2005 health plan study we also asked about skill using the Internet with the exact same scale as our health question and displayed in identical fashion. Here are those results: 
Web respondents here seem to have had little difficulty choosing the extreme response, although the percent choose "Fair" is interesting.
I don’t believe this is about "extreme responses" at all. I think it’s social desirability. People want to present themselves in a positive light to other people (such as telephone interviewers) but that goes away in the context of the relative anonymity of a Web survey. There are lots and lots of studies that have shown people will under report really sensitive behaviors like abortions and drug use in interviewer administered surveys. For these kinds of issue self-administration is the way to go. I’m surprised to find that respondents apparently view their health status as sensitive, but the data are pretty persuasive if not compelling.
