DailyKos: Kos: Zogby "interactive" polls are junk

Zogby "interactive" polls are junk
by kos
Wed Nov 28, 2007 at 06:53:47 AM PST

I know Hillary's opponents are jumping on Zogby Interactive's latest poll showing Hillary doing substantially poorer than her opponents in head-to-head matchups than her opponents.

Let me make this as clear as possible: Zogby interactive polls are JUNK. They are about as solid as the Daily Kos cattle call polls would be if we were trying to claim the community represented all Democrats.

Witness this little bit of disclaimer:

The poll of 9,355 people had a margin of error of plus or minus one percentage point. The interactive poll surveys individuals who have registered to take part in online polls.

How a poll that is essentially a web poll can be considered credible is beyond me. But you don't have to take my word for it. Look at how poorly the Zogby interactive poll performed in 2006 (after a disastrous debut in 2004):

I opted to use Mr. Mitofsky's method in my own number crunching. I looked at five pollsters that were among the most prolific: Rasmussen, SurveyUSA, Zogby (which releases separate telephone and online polls) and Washington, D.C.-based Mason-Dixon. For all but the latter, I used the numbers posted on the organizations' own Web sites. For Mason-Dixon, which keeps some of its poll data behind a subscriber wall, I used Pollster.com to find polls from the two weeks before the election. I checked the results against vote counts as of this Tuesday [...]

On to the results: In the Senate races, the average error on the margin of victory was tightly bunched for all the phone polls. Rasmussen (25 races) and Mason-Dixon (15) each were off by an average of fewer than four points on the margin. Zogby's phone polls (10) and SurveyUSA (18) each missed by slightly more than four points. Just four of the 68 phone polls missed by 10 points or more, with the widest miss at 18 points.

But the performance of Zogby Interactive, the unit that conducts surveys online, demonstrates the dubious value of judging polls only by whether they pick winners correctly. As Zogby noted in a press release, its online polls identified 18 of 19 Senate winners correctly. But its predictions missed by an average of 8.6 percentage points in those polls -- at least twice the average miss of four other polling operations I examined. Zogby predicted a nine-point win for Democrat Herb Kohl in Wisconsin; he won by 37 points. Democrat Maria Cantwell was expected to win by four points in Washington; she won by 17 [...]

The picture was similar in the gubernatorial races (where Zogby polled only online, not by phone). Mason-Dixon's average error was under 3.4 points in 14 races. Rasmussen missed by an average of 3.8 points in 30 races; SurveyUSA was off by 4.4 points, on average, in 18 races. But Zogby's online poll missed by an average of 8.3 points, erring on six races by more than 15 points.

Seriously, Zogby polls suck. Yet according to Google News right now, the "Hillary loses against all Republicans in the general" poll has been cited by over 200 media, while the far more respectable Gallup effort which shows that Hillary in fact beats them all has been far less reported.

The Zogby survey was covered repeatedly on CNN, earned coverage from MSNBC, Fox News, and Reuters and was covered by multiple other smaller outlets.

By contrast, I can't find a single example of any reporter or commentator on the major networks or news outlets referring to the Gallup poll at all, with the lone exception of UPI. While the Zogby poll was mentioned by multiple reporters and pundits, the only mentions the Gallup poll got on TV were from Hillary advisers who had to bring it up themselves on the air in order to inject it into the conversation.

You could argue that the Zogby poll got all the coverage it did precisely because it is out of sync with multiple other polls, and thus is news. But the truth is that the reporters and editors at the major nets know full well that the Zogby poll is bunk -- yet they breathlessly covered it anyway.

Worse, the Zogby poll was covered with few mentions either of its dubious methodology or of the degree to which its findings don't jibe with other surveys. Bottom line: The Zogby poll was considered big news because many in the political press are heavily invested in the Hillary-is-unelectable narrative for all kinds of reasons that have little to do with a desire to, you know, practice journalism.

The media has its agenda, which right now is the "Hillary is fading" narrative. The hard core supporters of the other Democratic primary candidates have their agenda -- to raise bullshit "electability" arguments against Hillary.

And those of us who remain reality-based and dispassionate throughout this all can only shake our heads at the credence being put in a discredited shill of a "pollster".

Update: Pollster.com:

It is reasonable that the people who volunteer to take political polls over the internet are considerably more interested in politics (and likely more strongly partisan) than is a random sample of likely voters. That should be expected to lead to fewer people with "don't know" responses as better informed and more partisan respondents are likely to both know more about the candidates and to have made up their minds sooner than a proper random sample. That helps explain why Zogby's 2006 internet polls looked as they did.

But this does no good in Clinton's case.

Actually, it's a perfect explanation. You see Hillary's results in the dKos straw poll? In all internet polls she far underperforms her "real world" numbers.

http://www.dailykos.com/story/2007/11/28/95347/662