Welcome to my first blog for The Politics Counter. I hope you enjoy it. If you have received this by email and do not wish to receive further blogs, simply click the ‘unsubscribe’ link below. If you have come to this via the Politics Counter site, and DO wish to received future blogs by email, please complete the “subscribe” form to the right.
Two rival narratives are competing for our attention in the EU referendum campaign. Online polls say the contest is neck-and-neck, while telephone polls insist that “remain” holds a clear lead. I believe that the telephone polls are nearer the truth. Unless at least two million “remain” supporters are scared into switching sides by some sudden crisis (to do, say, with terrorism, refugees or the Eurozone), the UK will vote by a comfortable margin to stay part of the European Union. This blog sets out my reasons for saying this.
Let’s start with the basic numbers. Seventy-nine referendum polls have been published this year. Fifty-six have been conducted online. Most have shown a very close race. If we combine them to create a giant poll-of-polls, then both sides have 42%. The other 23 polls have been conducted by telephone. All bar one have shown “remain” ahead, usually by a significant margin. Their poll-of-polls has “remain” ahead by 50-39%.
The past ten days have maintained this pattern. Three online polls show “leave” on average just ahead by 43-42%; three telephone polls indicate, again on average, to a clear lead for “remain”, by 52-39%.
ALL POLLS HAVE A PROBLEM
Before we discuss these differences, a broad point needs to be made about all polls. They face a huge challenge getting anything right. By definition, online polls survey people who have chosen to join a polling panel. In theory, telephone polls are able to reach everyone who has a phone. But they are at the mercy of response rates; and in recent years these have collapsed. Twenty years ago, according to one of our most respected telephone pollsters, they achieved a response rate of around 30%. To obtain 2,000 interviews, they asked their computer to generate 7,000 random residential telephone numbers. These days, to obtain 2,000 interviews, they need to start with as many as 28,000 phone numbers, for response rates have fallen to just 7%.
In short, all polls require research companies to extrapolate from the small proportion of the general public they can reach to the far larger number of people who neither join online panels nor respond to telephone polls. The surprising thing about is not that today’s polls sometimes get things wrong, but that they have such a good record of getting so many things right.
The reason why our leading polling companies are so good, so often is that they go to great lengths to make their samples match the country’s population – by age, gender, region, social class, past vote and so on. Usually this process generates accurate results. But sometimes it doesn’t, either because the silent, unpolled, majority, differs from the poll-friendly minority in some way that is not captured even by the smartest demographic sampling – or because of “mode effects”, in which the way a poll is conducted prompts some respondents to conceal their true feelings.
THE TWO RIVAL EXPLANATIONS
The debate about the difference between telephone and online polls in the current referendum campaign is essentially a debate between these two explanations. Let us consider them in turn.
Supporters of telephone polls say that their samples are better in the current referendum. Even though their response rates are so low, they still reach a fair number of two key groups : a) the people with little interest in politics who tend to favour the status quo rather than take risks; and b) well-educated social liberals who lead busy lives and are less likely to join online panels. Both groups, for different reasons, veer towards “remain” rather than “leave”.
Supporters of online polls say that their recruitment, sampling and weighting methods ensure a good mix of the politically interested and uninterested. As for the matter of social liberalism, the reason why online and telephone polls sometimes diverge is that the anonymity of online surveys prompts more honest answers. Some people are unwilling to admit their true less-than-liberal feelings on, say, gender or race equality, to a stranger over the phone. By the same token, some people who plan to vote to leave the EU think it more prudent to tell a telephone pollster that they will vote “remain”, for this is the view espoused by the government, main opposition, most business leaders and, generally, much of the establishment.
It is possible that both sides are at least partly right – that, say, online panels generate less good samples but more honest answers. Indeed, when I worked for YouGov, we gathered a lot of evidence from different countries about “social satisficing” – an ugly (and, inevitably, American) term for the habit of some people sometimes being more candid online than when interviewed by a stranger. If I wanted to uncover people’s true views about sex, illegal drugs or tax-and-spend, I would always trust a well-conducted online poll more than a telephone poll.
However, I am not sure that social satisficing explains the difference in the current referendum. Here are five reasons – none of them individually decisive but, together, in my view, persuasive.
- ONLINE POLLS CONTAIN TOO MANY UKIP SUPPORTERS
Four of the six latest surveys report voting intention. The two online surveys (by YouGov and ICM) both show Ukip on 18%. The two telephone surveys show Ukip on 14% (ICM again) and 10% (Ipsos MORI). My judgement is that the figures in the online polls are too high, while Ipsos MORI’S figure looks too low.
My reason for saying this is that the lower figures are more consistent with Ukip’s performance in this month’s elections. YouGov, uniquely, polled in London, Wales and Scotland. It got all the big stories right – Sadiq Khan winning comfortably in London, the Tories coming second in Scotland, Ukip breaking through in Wales – but in each case it overstated Ukip’s support, and by roughly similar amounts. Overall, for every ten votes that YouGov predicted that Ukip would secure, the party obtained only seven.
YouGov has recognised this point. In its latest poll it has tweaked its methods with the aim of eliminating its overstatement of Ukip’s support. However, even after the change, its 18% for Ukip still looks too high.
The relevance to the referendum is clear. Ukip supporters are almost unanimously for Brexit. If their numbers are overstated, so is support for Brexit. My own judgement is that the latest online polls overstate Ukip’s support by around four percentage points. If we reallocate these to the Conservatives, whose supporters divide roughly evenly on the EU, then this has the effect of increasing the “remain” vote by 2% and reducing the “leave” vote by the same amount. This turns an online-average of a one point “leave” lead of 1% to a “remain” lead of 3%. It doesn’t completely close the gap between online and telephone results, but it reduces the difference. (The opposite adjustment to Mori’s low Ukip figure would reduce its exceptionally large “remain” lead from 18% to 14%, and bring it closer to the figures produced by ICM’s and ORB’s telephone surveys.)
- OTHER EVIDENCE POINTS TO SAMPLING DIFFERENCES
Some weeks ago Populus conducted a comparative study, putting the same questions to online and telephone samples. The results appear in a report by Matt Singh (whose Number Cruncher blogs are essential reading) and Populus’s James Kanagasooriam. Among the questions it put to respondents in England was whether they considered themselves “more British than English”, “more English than British”, or “equally British and English”. This was a question that the British Election Study had posed to a random, face-to-face sample last summer. The key thing is the number who said they were “more English than British”, for this correlates closely with attitudes to the EU: people who say they are “more English” tend also to be more pro-Brexit (and also more pro-Ukip). These are the numbers found by each method: face-to-face: 24%; telephone 24%; online 32%.
I see no reason for these differences to flow from “social satisficing”. I doubt that anyone will respond differently to a stranger than to an online survey. It reinforces the point about differences in Ukip support: sampling differences may not tell the whole story, but they tell a significant part of it.
(Before proceeding, and before anyone else points this out, I should add that Populus’s director is Andrew Cooper, a Conservative peer and pollster for Britain Stronger in Europe. Those who think this taints his, and Populus’s, analysis are of course free to do so. But I know Andrew; and I also know from my experience at YouGov that a polling company’s reputation depends on its honesty, objectivity and quest for accuracy. All pollsters should be judged on their merits, not on imagined, and invariably false, assertions of wilful bias.)
- SOCIAL LIBERALS AND “REMAIN” VOTERS ARE HARDER TO REACH
Populus has also analysed data from last summer’s face-to-face survey by the British Election Study. This was an old-fashioned, time-consuming and expensive survey. It started with a list of random names and addresses. Its interviewers were asked to speak to them, and ONLY to them. If the named respondents were not available first time round, interviewers were instructed to return at least a dozen times over a period of weeks.
Why is this relevant to the EU referendum? Because, according to Populus’s analysis, there was a systematic difference between those who were reached on the first visit and those who were reached later. The longer it took to get hold of a respondent, the more likely was the respondent to be a socially liberal Conservative, “more British than English” – and more favourable to British membership of the EU.
Now, it does not necessarily follow that people who are harder to reach in a face-to-face survey are less likely to join an online panel and answer their surveys; but it does provide indirect corroboration of the view that the differences between online and telephone polls are more to do with sampling than “social satisficing”.
- WE HAVE BEEN HERE BEFORE
National referendums are rare, but we had one five years ago, when the UK voted by 68-32% to retain the first-past-the-post voting system for electing MPs rather than switch to the Alternative Vote. All the final polls pointed correctly to a big victory for the status quo, so their performance did not attract as much attention as they did in last year’s general election.
However, once again there was a clear difference between the final online and telephone polls. The two online polls (Angus Reid and YouGov) predicted a 61-62% vote for keeping the existing voting system; the two telephone polls (ComRes and ICM) had the figure much closer: 66-68%.
On its own, this record is of only limited relevance today. The EU is a vastly different issue, and polling methods have evolved over the past five years. That said, it does look as if the online surveys in 2011 ended up systematically understating the number of (less politically engaged?) voters who preferred the status quo.
- IT’S HAPPENING IN AMERICA, TOO
Once again, the evidence is indirect, but suggestive. This week the New York Times has done a detailed analysis of the polls in this year’s US primaries. It’s main conclusion is simply stated. On average, the telephone polls correctly measured Donald Trump’s popularity, while the online polls generally overstated his support. Could it be that in both Britain and the US, online surveys are overstating the desire for change? The New York Times surmises that one reason is that online surveys generate more “don’t knows” than telephone surveys. It may be that a number of these don’t knows are people who would take sides in a telephone poll – and do vote, tending to side with the status quo.
If this is right, it sounds uncannily like part of the debate here in Britain, where online polls in the current referendum elicit more don’t knows and – perhaps – understate the fear of change by politically less-interested voters.
There is an alternative explanation which online pollsters prefer: that turnout in the referendum won’t be that high, and that the higher don’t know figures produced by online pollsters are more realistic. According to this view, the telephone polls are including too many less-interested status-quo respondents who will end up not voting at all.
Were this the only evidence about telephone-versus-online polls, I’d be tempted to support that view. But as the US telephone polls have proved generally more accurate at measuring Trump’s support, and the New York Times analysis is so consistent with the evidence of what is happening here in Britain, I’m inclined to side with the analysis of the telephone pollsters.
Two conclusions in fact. The first is that online polls should not be pilloried for their figures. Given the hazards of extrapolating from the responses of those who join their panels to the rest of the population, they have a remarkable record for accuracy. Sometimes they outperform telephone polls – not just on “social satisficing” issues, but occasionally with superior samples. For example, in 2008, YouGov alone predicted Boris Johnson’s victory; all the telephone polls consistently and wrongly put Ken Livingstone ahead.
On the great majority of issues, online and telephone polls produce comparable results. And because online methods are cheaper, this means for the same money, more people can be asked more questions, and so yield a deeper understanding of the public mood.
However, the EU referendum is one of that minority of occasions when there is a significant difference; so far, telephone polls seem to generate more accurate results. We shall see whether the online polls acknowledge this and make further changes to their methods.
Conclusion number two: I reckon that the current state of opinion among those who express a view is something like: remain 55-57%, leave 43-45%. The gap probably narrows when one takes account of the greater determination of “leave” supporters to turn out to vote, to around: remain 53-55%, leave 45-47%.
However, past referendums in Britain and abroad have generally produced a late shift to the status quo, as people who make up their minds late tend to prefer the familiarity of the status quo to the risks of change. Given the character of the debate about the economic impact of Brexit, a similar shift looks likely this time.
This could lead to a significant majority in favour of the UK staying in the EU. From David Cameron’s point of view, this is just as well. A narrow 51-49% victory would be unlikely to settle the matter for long, or bring the two sides within the Conservative Party together. He probably needs a percentage majority in double digits – 10% or more – to have any chance of marginalising his anti-EU critics inside his party. The good news for the Prime Minister from this analysis is that he is currently on course to achieve it. The bad news is that it could still go wrong for him, especially if some kind of crisis erupts in the next five weeks.