Have You Stopped Beating Your Dog?

I was really eager to find out more from Charles Franklin, Founder of PollsandVotes.com and the Director of the Marquette Law School Poll about online poll standards. As noted in my previous blog post, he paid JOUR4953 a visit this week.   Franklin had a good deal more than that to share and I believe that he brought our group a better understanding of the value of polling.  Much of it had to do with his enthusiasm and self-proclaimed "OCD" on the topic.  To highlight some problems with poll data reporting, Franklin mentioned a Time Magazine Poll on George W. Bush's approval ratings that really he said still "sticks in my craw!" His contention was that when you don't look at the aggregate data and only take the results from two points/dates, you not only miss the actual change, you use misleading data.  From his example, I was concerned with how I had been looking at and possibly misusing data.

Franklin's discussion flowed into why message control matters and why opinion polls matter.  Wording that pollsters use can mitigate or create bias.  Asking someone a question like "have you stopped beating your dog?" gives them no real choice.  Organizations that are paying for a poll may actually be anticipating the results they want and will framework the questions to meet their needs.  As he explained the finer points of polling needs, use and impact, I realized that the work went beyond writing questions. Analysis of polling data is a statistical nightmare, or dream-come-true if you like that kind of thing.  When Franklin was asked to define margin of error, he avoided giving us the formula above and kept it simple.  Consider doing the same poll 20 times, with a margin of error of 3 points, you would get the exact same results 19 times with only one resulting in a variance of 3.  I say, trust the experts.

Luckily, I was able to ask why Franklin said that online polling was still in an "R&D" phase.  His response required him to first explain that sampling is a proven statistical model, "You don't have to drink the whole pot of soup to make sure it'll taste good!"  Methodology also requires randomization and equal distribution, which he equated to stirring the pot of soup and grabbing a spoonful to sample.  Truly an effective way to explain something that is overly statistical. Phone polling participation has gone from 90% down to 20% in the past two decades, but still follows proven methodology. Online sampling will need to find a way to develop a statistical model that can prove accuracy from a non-random sample group. I'm glad he clarified, but I'm fully convinced that online polling will be the method of the future.  I am also certain that soup is in my future, too!