Yesterday, the firm I work for, J.L. Partners, released a poll showing a 12-point lead for Labour over the Conservatives. This gap between the parties is much narrower than other polling companies are predicting – with several other polls showing a 20 point plus lead for Labour. Our poll still shows that Labour is heading for victory, but it has already been seized on by the Conservatives as a sign that they are still in contention in this election campaign.
What explains the difference in these polls? And could the election in fact be closer than many think?
To understand how we have arrived at this result, you also have to understand the more nuanced features of this election. I have heard from countless activists on the doorstep from every party that voter apathy and indecision is a significant issue in this election. For every staunch Labour supporter, or dedicated Conservative, there is a stubborn group of voters who simply haven’t decided how they’re going to vote. Statistically and mathematically, this group is a challenge for pollsters to deal with. To work out how they are likely to vote, pollsters often need to use some form of prediction.
Similarly, turnout is a huge problem in this election. In 2019 turnout was 67.3 per cent – the second highest figure since 1997. But that still leaves us with just under a third of the population which doesn’t tend to vote. So we have to adapt our models accordingly, in the knowledge that not everyone who claims they support a given political party will make it to the polling booth on election day or will send off a postal vote.
To start with modelling the ‘don’t knows’, our company has implemented a new methodology based on machine learning – essentially using an algorithm to predict the way people will behave. We use a variety of predictors such as previous voting habits and demographics, and we combine this with current assessments and perceptions of the leaders of the political parties. At the simplest level, we map undecided voters onto political parties, by seeing how they respond to certain policies. So if a voter says they like Sunak, thinks the Conservatives are better on cutting crime and voted Conservative at the last election, then we predict they will likely vote Conservative again – even if they claim they are unsure about who they are going to vote for this time.
Using this approach, these undecided voters end up being split largely between the Conservatives, Labour and Reform UK. Based on our latest analysis, when you factor in the undecideds, this gives an extra point in the polls to the Conservatives, removes two percentage points from Labour and adds two points to Reform.
Although we are still predicting a solid Labour victory, winning over undecided voters will be absolutely crucial for some of the key battleground seats the party is hoping to win for the first time ever – seats like Hexham in the north-east, and Basingstoke in the south-east. Undecided voters also pose a risk to the Conservatives, who should be very wary of the impact of Reform. Over the last few months, an estimated 25 per cent of undecided voters moved to Reform. Last year, we reported that 11 per cent of Conservative voters, around 1.1 million people, intended to vote for Richard Tice's party. Failing to convince these undecideds could be the difference between a bad loss and a total annihilation of the Conservative party.
When modelling turnout, we have a similar machine learning model. We ask our polling respondents how much attention they pay to politics, and combine this with demographic information and constituency level data. We then predict how likely various groups are to turn out on election day. For example: 79 per cent of over-65s will vote, whilst only 35 per cent of 18- to 24-year-olds are likely to cast their ballot.
This is then factored into our voting intention calculations. The demographics that support the Conservatives are simply more likely to turn out than Labour voters – and so this reduces Labour’s lead in the polls. It also explains why the Get Out the Vote operation on polling day will be so crucial and could make all the difference for Labour.
It is worth stressing that it is not just our methodology that explains why the Labour lead is tightening. It seems the act of calling the election has shored up Conservative support with some key demographics, especially the over-65s. In April, the Tories had only an eight-point lead with this group. Now they are 20 points ahead. Given this, it is easy to understand why the Conservatives are focusing on policies like national service and the ‘pension triple lock plus’ aimed squarely at the ‘grey vote’. We have also found that fewer Tories are saying they would now consider voting for Reform than before the campaign got underway.
In polling, there are so many unknowns – piecing together predictions is often like creating a jigsaw with only half the pieces available. As well as tracking changes, we try to work out two of the most challenging unknowns – the undecideds, and who will turnout – in a robust and mathematically consistent way.
Our view is that if you do not control for these numbers in your polls, you are missing an important part of the picture. That is why our Labour lead looks so different to the other pollsters.
Comments
Comments will appear under your real name unless you enter a display name in your account area. Further information can be found in our terms of use.