NS Premier-designate Tim Houston grins as he makes fun of pollsters & pundits on election night (YouTube-CPAC)

In his election night victory speech, Nova Scotia’s Progressive Conservative leader (now, Premier-designate) Tim Houston gleefully referred to opinion polls in the spring that showed the Liberals with a massive lead.

“As recently as the end of May, the pollsters and the pundits, they had us back 28 points in the polls. It was going to be a historic Liberal landslide,” Houston said before delivering his punch line accompanied by a trademark, toothy grin.

“As for those so-called experts, they were all writing us off. Well, I wonder what they’re writing right now.”

Houston was obviously enjoying the fact that in the one poll that really counts, the PCs had won a comfortable, 31-seat majority in the Nova Scotia legislature.

An online CBC headline read: “Progressive Conservatives surge to surprise majority win in Nova Scotia election.”

Yes, to the pollsters and pundits and, to the journalists who like to read the tea leaves, otherwise known as opinion polls, the PC victory was truly surprising.

But maybe reading tea leaves isn’t always the best way to forecast election outcomes.

And now that a national campaign is in full swing, let’s see what federal voters could learn from the polling in Nova Scotia.

Nova Scotia’s last poll

The Toronto-based firm Mainstreet Research conducted its final campaign poll in Nova Scotia on August 14-15, just a couple of days before the August 17th election.

First, here’s the good news — at least for the pollsters. In its survey of 502 “decided and leaning” voters, Mainstreet Research accurately predicted the percentages of votes the three main parties would win within the +/- 4.3 percentage-point, standard margin of error for a sample that size.

Mainstreet awarded the Liberals 38% and sure enough on election night, the official results show they scored 36.67% well within the plus or minus four-point margin of error. Hooray, the poll accurately predicted that outcome!

Mainstreet gave the PCs 36% and sure enough on election night, they scored 38.43%, again well within the margin of error. Another accurate prediction.

Mainstreet awarded the NDP 21% and on election night, the N-Dippers scored 20.94%. Yes, that last minute poll sure got things right.

The not so good news

Now, here’s the bad news for ardent tea leaf readers.

In its summary of findings, Mainstreet reported: “In our final poll before the election tomorrow, we find the Liberals with a narrow two-point lead over the Conservatives.”

Oh dear.

That finding might lead unwary tea-leaf readers to conclude that the NS Liberals would win, with a minority maybe?

But wait. That two-point Liberal lead doesn’t mean much when you factor in the margin of error.

OK, so Mainstreet gave the Liberals 38%, but since the small sample size of 502 voters carries a standard 4.3-percentage-point margin of error, the Liberals could have scored as high as 42.3% on the plus side and as low as 33.7% on the minus side.

In other words, polling results always show a range and not just the fixed numbers that the pollsters release and the media report.

Now let’s look at the PC range with a high of 40.3% and a low of 31.7%.

Obviously, the two ranges overlap, but on election night, it was the PCs who came in about two points ahead with 38.43% while the Liberals were back at 36.67%.

That two-point lead was enough in a first-past-the-post electoral system to win the Progressive Conservatives 31 seats while the Liberals came in with only 17.

On analysis, the Mainstreet poll — accurately forecasting percentages of votes won — told us everything — and nothing.

Based on the small sample size of 502 and the correspondingly wide margin of error, it’s not possible to tell from those polling results which party was leading and which one might win the election.[efn_note]As a general rule, be cautious about any sample size under 1,000, especially when parties are within a few points of each other. A sample size of 1,000 carries a standard margin of error (MOE) of +/- 3.0 percentage points. For a sample size of 1,200, the MOE is +/- 2.8. Note: the number of undecideds should be subtracted from the overall sample size. To calculate MOE divide .5 by the square root of the sample size, multiply by 1.96 and then by 100. E.g. Let’s say the sample size is 800 and therefore, the square root is 28.284. Divide .5 by 28.284 = 0.0176778 X 1.96 = 0.034648488 X 100 = +/-3.46. https://goodcalculators.com/margin-of-error-calculator/%5B/efn_note%5D%5Befn_note%5DHere is an example of how to apply the margin of error (MOE) to the latest national results (Aug. 21) from Nik Nanos in polls conducted for CTV and the Globe and Mail. Sample size: 1,200. MOE +/-2.8. Results Liberals: 34.3%. Conservatives: 32.4%. Conclusion: It is not possible to say which party leads in this poll when the MOE is applied. The Liberals are somewhere between 37.1% and 31.5% while the Conservatives are somewhere between 35.2% and 29.6%. Either party could be anywhere within its range.[/efn_note]

Perhaps the Mainstreet summary should have read:

“In our final poll before the election tomorrow, we find we can’t tell whether the Liberals or Conservatives are ahead or behind and by how much since our sample size was so small and our margin of error so wide. We strongly urge all tea leaf readers to wait for tomorrow’s actual outcome to see whether it’s a win, lose or draw.”