Skip to main content

Why did the polls go wrong? By Jon Mellon and Chris Prosser

The British Election Study Team

The post-election wave of the British Election Study (BES) Internet panel allows us to take a closer look at possible causes of the polling miss during the recent General Election. We previously identified five possible explanations: 1) “don’t knows” shifting, 2) a late swing among voters, 3) Shy Tories, 4) problems achieving a representative sample and 5) differential turnout. This post outlines the evidence we have gathered so far (see our working paper here for a more detailed account).

Late swing and “don’t knows”

The post-election data immediately casts doubt on two of the theories. In our campaign wave, 7% of people said that they “don’t know” who they would vote for. In the post-election survey (when we can see how undecided respondents ended up voting), we find a very small edge for the Conservatives among previously undecided voters. However, “don’t knows” only contribute around 0.05 percentage points towards the polling gap, so it is unlikely to have been a major factor. Similarly, we find that there is no difference between the proportion of respondents supporting the Conservatives in the campaign wave and the post-election wave, making it unlikely that there was a late swing.


Figure 1 Support for each party among BES respondents in the campaign and post-election waves

Shy Tories

We also have evidence against the Shy Tories theory. We can examine the Shy Tory theory by considering where there is likely to be social pressure on Conservative voters. For example, it seems unlikely that Tories would need to be shy in the heavily Conservative Shires but it is more plausible that they would be by shy in traditional Labour heartlands like Sunderland. Figure 2 shows that we actually observe the opposite pattern. The deviation between the proportion of BES respondents saying they voted Conservative and the actual proportion of voters who did is highest in strong Conservative areas where we would expect the least social pressure against voting Conservative.


Figure 2 Conservative 2015 vote share in the BES post-election survey and actual results according to 2010 Conservative and Labour shares

We also find no evidence for another aspect of the Shy Tories theory. Several pollsters have suggested that placing the vote intention question later in a survey makes respondents more willing to admit that they plan to vote Conservative. In the first three waves of the BES we randomized the placement of the vote intention question to be either at the start of the survey or at the end after all other questions. We find that the question placement makes no difference to the proportion of respondents intending to vote Conservative. Taking these findings together, we are doubtful whether Shy Tories were a major contributor to the polling miss.

Representative samples

We also have more evidence about the representativeness of polling samples used before the election. One possible source of non-representativeness could be the groupings used for weighting by polling firms. For instance, in the graph below, we look at whether the age groupings used for weighting (both by YouGov and many other polling firms) hide variation within those groups (thanks to Helmut Platt for suggesting this possibility). The red line shows the lead in share of the vote that the Conservatives have over Labour by age. The vertical lines represent the breakpoints between the standard age bands used for weighting. The bars show the difference between the percentage of BES respondents of a particular age and the percentage of the 2011 population of the same age (e.g. positive bars mean that an age is over-represented in the BES).The most important deviation is the oldest age group, where younger (less Conservative leaning) respondents are overrepresented whilst older (more Conservative leaning) respondents are underrepresented. The net effect of this difference is to dampen the Conservative lead. This problem is even greater for the oldest respondents in the sample –those over age 80 make up 5.1% of the population, but only 0.5% of the BES. This evidence suggests there is some pro-Labour bias due to the age groupings used, but this might yet be cancelled out by other parts of the weighting scheme. We will need to examine all the weighting variables before we can draw conclusions about the contribution of non-representative samples to the polling miss.

age and lead overlayed

Figure 3 Deviation in proportion of respondents of each age compared to census (left axis). Vertical lines designate the age boundaries of the weighting age groups. The red line is a LOWESS regression of Conservative-Labour lead against age among BES respondents in wave 6 (right axis).

Differential turnout

There is also new evidence for the differential turnout theory. 91.6% of our respondents claim to have voted compared with 66.4% in Great Britain as a whole. While this partially reflects the fact that polling respondents tend to be more politically interested than the general population, we also have considerable evidence that respondents overstate their turnout: 20% of respondents in areas without local elections claim to have voted in them in 2015; 3-6% of respondents in the campaign wave claim to have voted by post before the postal ballots were actually issued and 46% of respondents who we could not verify as registered to vote in June 2014 claim to have voted in the 2014 European Elections. In all of these cases, the fibbers lean significantly more Labour than other respondents.

We look at the impact of overstated turnout more precisely by building a predictive model of turnout based on the validated vote in the 2010 BES face-to-face survey. The model accounts for a respondent’s stated likelihood of voting prior to the election, turnout in previous elections, their age, marital status, household income, unemployment and trade union membership, as well as several constituency factors, including the overall turnout in their constituency in the previous General Election. After accounting for these factors, we estimate that our respondents’ turnout is likely to have actually been around 73.4%. Importantly, we can look at how vote intention differs among respondents who have different predicted probabilities of voting. Figure 4 shows that the Labour lead among unlikely voters grew hugely between 2010 and 2015, suggesting that differential turnout is an important factor in explaining the polling miss: considerably fewer of those saying they were going to vote Labour are likely to have actually turned out to vote. Re-weighting our respondents according to their predicted probabilities of voting explains about 25% of the gap in the Conservative lead between the pre-campaign wave of our survey and the actual election results.

2010 vs 2015 turnout lpoly pre shaded

Figure 4 Predicted probability of pre-election vote intention by predicted turnout probability in 2010 and 2015. The white bars show the distribution of predicted turnout probability in each year. The shaded areas illustrate the size of the Labour-Conservative gap amongst those less likely to vote for each year.

The evidence in the BES suggests that the reason for the increased impact of differential turnout is not due to a change in the relative enthusiasm between Labour and Conservative supporters since 2010. 84% of Labour supporters in 2015 said that it was “very likely” that they would vote, compared to 86% of Conservative supporters, while in 2010 the figures were 87% and 90% respectively. Rather the data suggest that the increase in the turnout gap between Labour and the Conservatives can be explained by shifts in party support amongst those who are actually less likely to turnout to vote, even if they say they will. This evidence strongly suggests that differential turnout was a major factor in the polling miss.

If differential turnout is the primary cause of the polling problems, this is relatively good news for pollsters. It should be possible for pollsters to fix many of their by using turnout weighting that accounts for the wider set of factors we have identified.


Our analysis of the post-election BES data makes us much more sceptical about late swing, “don’t knows” and Shy Tories. By contrast, we are leaning strongly towards differential turnout as part of the explanation and think that it’s likely that sampling and weighting also played at least some role.

  • cosyblackbird

    ‘it seems unlikely that Tories would need to be shy in the heavily Conservative Shires but it is more plausible that they would be by shy in traditional Labour heartlands like Sunderland’
    You could equally assume that those in more afluent areas are more likely to feel ‘liberal guilt’ about voting Tory. There is a middle class educated discourse, embodied by Twitter, which is big on Tory-shaming. And given that people are telling pollsters, or computers, not neighbours, who they mean to vote for, the likely judgement of neighbours may be irrelevant. Wealthier Tory voters are more likely to feel internalised guilt, as they are portrayed as voting for their own interest at the expense of the less fortunate.
    Also, didn’t the exit pole still understate Tories? What other than shy Tories can account for that?

    • Jon Mellon

      Thanks for your comment. Liberal guilt is an interesting theory. We’ve had a look and think that it doesn’t hold true in the data though.

      We plotted the average authoritarianism score by constituency (higher numbers are more authoritarian and lower numbers are more liberal) against the results and respondent answers. We see that the largest number of missing Tories are in the most authoritarian areas (where we would presumably expect to see the least liberal guilt).

      Another way of looking at this is by focusing on London. Intuitively, I would expect that London would be the center of Liberal guilt: rich people, high education levels, metropolitan values etc. However, the Labour lead in London is actually underestimated (the Tory share is about right, but we underestimate Labour).

      • Noah Carl

        Jon, I’d be very interested to see a comparison between the exit poll percentages and the actual results, but I can’t seem to find the exit poll percentages anywhere. As cosyblackbird notes, surely any relevant differences between the exit poll percentages and the actual results would have to be attributable to shyness. But maybe there weren’t any such differences?

      • Gareth Davies

        John, I’m surprised by the general lack of attention that is being given to the swing of Lib Dem voters to the Tories.

        Are these Tory voters exactly the ones who would be most reluctant to fess up to their intentions – both during the campaign and subsequently?

        My suspicion is that they are a core explanatory group whose shy dynamics will likely be missed out of the eventual published explanation.

  • Billy Bob

    The evidence against the Shy Tory explanation seems non-existent here. You make a supposition about when people might be shy about voting Tory which, as cosyblackbird points out, is contestable at least. Your evidence then appears to confirm the flimsy nature of your supposition. The Shy Tory explanation remains a strong runner.

    • Chris Prosser

      In 1992 two things were taken as evidence that there were shy tories and that they caused the polling problem: 1) Those who said they didn’t know who they were going to vote for were disproportionately
      Conservative leaning in recontact surveys and 2) Those who previously said they voted Conservative were less likely to answer later surveys.

      We don’t find either of those things in the BES survey – we discuss “don’t know’s” in this blog post and find that although they are slightly Conservative the difference they make to the polling numbers is negligible. We also look at retention across waves in the full paper (which we link to above) and find the opposite of what you would expect from the shy tory theory – previous Conservative voters are actually more likely to take later
      waves of our survey than previous Labour voters. We also find no evidence from the question ordering experiment we discuss in this blog.

      We wanted to test the shy tories theory in as many ways as we can so we tried to look for indirect evidence about where you would expect to find shy tories. On its own our interpretation of the graph might not be
      particularly convincing but it is only one piece of evidence and we have four – none of which suggest that shy tories played a big role in the polling miss.

      Without looking over the shoulder of survey respondents as they cast their votes the shy tory theory is very difficult to assess so we have to look for evidence indirectly. This is a work in progress and we will
      continue to look for other ways to test for shy tories and if anyone has any ideas about how to do so then we welcome suggestions.

  • Dave Franks

    What about the fact that people who take part in research are generally different to those that don’t? You see this in other areas of research. For example in grocery market research you’ll find a panel that is balanced for age, sex, social class, region and various other standard demographics will massively over estimate the number of Aldi shoppers and massively underestimate Waitrose.

    I noticed your sample comes from YouGov. I’d expect their panel to have massive inherent biases due to the fact anyone can just go to their website and sign up. It’s a completely self selecting sample. I think that across all demographics there’s a difference between those who have the time and inclination to sign up for and participate in online surveys are very different to those who do not. I don’t believe you can account for this with standard demographic weighting.

    It will be very interesting to compare the results from online panel polls against your face to face study which I assume will use more traditional random sampling techniques.