This is cross-posted at my blog Worthwhile Canadian Initiative.

There are a lot of published opinion polls out there: I count 41 since September 8. The trouble is that many of them contradict each other – just what are we supposed to make from all this? Let’s haul out our introductory statistics textbooks and take a closer look.

Firstly, let’s get an idea of the sort of variations that we should expect to see, that is, the inherent uncertainty with drawing random samples. Ekos, Nanos and Harris-Decima are all publishing daily tracking polls, with announced 95% confidence intervals of 1.6%, 3.1% and 2.6%, respectively. If sampling error were the only source of uncertainty, what would a typical deviation between Nanos and Harris-Decima look like? (I’m choosing those two because they have the larger announced MOE; what follows will be an upper bound for the variations we should expect.) The answer is not 3.1 + 2.6 = 5.7%.

If we play around a bit with the arithmetic of confidence intervals, the standard deviations for the Nanos and Harris-Decima estimates are 1.58 and 1.33, respectively. If Nanos and Harris-Decima are operating independently, the standard deviation for the difference between the two estimates is (1.58² + 1.33²)^1/2 = 2.06. From this, the half-width of the 95% confidence interval is 1.96*2.06 = 4.05.

If sampling error were the only thing to worry about, the chances of seeing a gap of more than four percentage points is 1 in 20. It also follows that the probability of observing a discrepancy of 5 or more points is 1 in 65, and the odds of seeing a 6-point gap are 274:1 against.

So much for the theory; what are the variations we’re actually seeing? First up is the support for the Conservative party; click the image for a full-sized version. In all the graphs that follow, the scale of the vertical axes is the same, so that differences in poll results are the same distance apart.

So far, so good: it is almost always the case that the polling firms differ by 3 percentage points or fewer. Indeed, there have been many days where two or three polling firms come up with the same estimate. It’s also interesting to note that of the 26 polls published since September 15, only two are outside the range of 35-38%.

Now the New Democrats; again, click the image for a larger version.

The range of variation is greater than was the case for that of the Conservatives, but at least some of that can be attributed to the fact that NDP support appears to have been growing during this period. Aside from Ekos – which started with high initial support – all the polling firms show a discernible trend upwards.

For the Conservative and NDP numbers, the variations across polling firms is pretty much in line with what we’d expect. But things start to break down when we look at the Liberals; click on the graph for a full-size version:

In contrast to the fairly tight variation we’ve seen so far, the gaps between the reported estimates for Liberal support are implausibly large: discrepancies of 8 points – in theory, an event that should occur 1 time in 10,000 – are not uncommon. Something is seriously wrong here.

The same problem for the Greens; again, click the image for a larger image:

Discrepancies of 5 or 6 percentage points should be rare (especially for the Greens), but we’re seeing them pretty much every day.

It seems pretty clear that Nanos is the outlier here: their numbers for the Liberals are consistently higher, and their estimates for the Greens are consistently lower than what the other firms are reporting. It seems that there is a significant chunk of the population – something around 4% – that is telling Nanos that they’ll vote Liberal, but telling everyone else that they will vote Green. Why? And what does this mean for what will happen on October 14?

It’s not too hard to come up with a plausible answer to the first question:

Mr. Nanos says the key difference is methodology. Unlike other polling firms, his asks open-ended questions on voter intention. Instead of offering a list of choices — “Would you vote a) Conservative, b) Liberal …” — Nanos phone operators ask an open-ended question that requires respondents to come up with their own answers instead of multiple choice.

“If they don’t get the list, you get the cleanest read because they have to articulate their support,” Mr. Nanos said. The open-ended question eliminates the importance of the order in which the parties are listed, although most companies vary the the order to mitigate this factor.

Also, the open-ended method tends to put the Greens lower than other parties because, Mr. Nanos believes, respondents are not reminded of the party when they answer. Some will choose the Greens as a none-of-the-above if they hear the party name on a list before answering.

But it’s much less easy to figure out what this segment of the population will actually do on election day. Will they indeed vote Liberal? Will they be reminded of the alternatives when they get to the voting booth (the party names are listed on the ballot) and vote Green? Or will they simply not vote?