The problem of router bias

Everyone worries about router bias and it's not clear that anyone has figured out how to deal with it. At its core it seems to come down to the priority given to one survey over others and how that impacts the samples delivered to all of those other surveys. Before panels we drew samples that were independent of one another. But online samples create a dependency because sample drawing for a group of surveys is centralized, drawing on a fixed pool of respondents, in the case at hand, those entering the router. But, of course, the needs of one survey may conflict with those of another. I think of routers are chaos systems in action and each time a new survey is introduced you get a classic butterfly effect.

Here is a classic example. Let's say we have a general survey of car buyers as well as a survey of Hummer buyers both active in the router at the same time. Because of the low incidence of Hummer buyers all of them might go to that survey and the general survey of car buyers will have no Hummer buyers in it. (And if you happen to believe, as I do, that Hummer buyers have a unique perspective on the world that perspective, too, may be under represented in other surveys that have nothing to do with cars.) In other words, meeting the needs of low incidence surveys in the router could well bias the samples for the other surveys active at the same time. It gets really complex when you realize that there are hundreds if not thousands of surveys, all being serviced simultaneously by the same router and from the same pool of respondents.

Some argue that random assignment makes this problem go away but it's not clear if that's the case or how widely it's practiced. The fallback seems to be that this problem is inherent to panels, has always existed and has been managed inconsistently even within a single panel company. But everyone also agrees that it's a problem requiring a good deal more thought and empirical research before we understand it well enough to deal with it effectively.