Labour leadership thought public polls were too optimistic
Labour's election pollster says public polls "showed a much more favourable position for Labour than we were finding in our internal data" both before the campaign and during it.
James Morris, who worked for Labour from when Ed Miliband was elected leader in 2010 until the election last week, told Newsnight that while "the lead in the public polls suggested Labour had got past the issues that sunk the party in 2010 - its record on the economy and immigration - we knew we had much more work to do and were still dogged by a loss of trust."
That is why, he said, the party ran a campaign based on a more "pessimistic scenario" than was the political consensus.
Mr Morris, a partner at Greenberg Quinlan Rosner Research, said that "from January 2011 to spring 2013, Labour's average vote share in the public polls rarely dropped out of the low 40s. We consistently had it around seven points lower."
He continued: "While the public polls had Labour ahead until the spring of this year, in our polls cross-over [when the Tories overtook them] came right after conference season in 2014. A four-point Labour lead in early September turned into a tie in October, followed by small Tory leads prompting the party to put reassurance on fiscal policy and immigration at the heart of the campaign launch."
These polls, which were unpublished and prepared for Labour, suggested that this plan had worked through the opening weeks of the campaign.
Labour had, they suggested, pulled ahead in the English marginals following Mr Miliband's performances in the debates and after the announcement about non-doms. The final poll of the campaign in late April, however, "told a different story".
Mr Morris said: "As focus groups showed the SNP attacks landing, we had Labour behind in the marginal seats." This was, he said, despite the fact that "a public poll in a similar set of seats at the same time showed a three-point Labour lead".
He continued: "The campaign strongly toughened our stance on the SNP before the final Question Time [TV appearance for Mr Miliband], but it was not enough. The Tories successfully used the fear of Scottish influence as a way of catalyzing pre-existing doubts about Labour in a way that had not been possible earlier in the campaign. Labour's unexpected post-referendum collapse in Scotland transformed the election across the whole of Great Britain."
Mr Morris believes that his experience suggested an answer to the question about why public pollsters got it so wrong.
It was, he said, "more in the questionnaire design than who gets interviewed. Whenever we tried adjusting our polls to match what other pollsters do, or tried telephone rather than online methodologies, the numbers stayed more or less the same.
"The main difference between our polls and the newspaper polls is that we don't ask the voting intention first... we first ask respondents to think about the country, the economy, their top issues, the parties and the leaders. We think it gets them closer to their ballot box mindset."
This design "delivers a much lower 'don't know' number - generally half the level found in the public polls... Of course, that requires many more questions and so is more expensive to implement especially for a phone pollster where every minute costs money.
"If we had run a final poll close to election day, would we have got the Tory margin right? It's hard to know. But if this explanation is broadly true, it means the drift to online polling remains valid."
He was hopeful that the inquest by the British Polling Council into pollsters' methods would have good results.
He said: "The British polling industry is full of people with great integrity and deep understanding of survey methodology. It is the most open industry in the world when it comes to data. It has rightly recognised that it got this election wrong, and that error had a real impact on the reporting of the campaign."
Mr Morris added that he believed the generous figures in public polls for Labour gave it "momentum in the face of a hostile media. Ultimately, however, it proved costly, allowing the campaign to become a referendum on the popularity of a putative Labour-SNP government.
"If Labour had been seen to be a couple of points behind, the scrutiny on the parties would have been more balanced, asking as much about the future of the NHS and living standards as they did about the role of Nicola Sturgeon."