Mr Miliband called the prime minister's refusal to take part unless the Green Party was involved a "pathetic excuse".
He said he would debate with "anyone invited by the broadcasters".
But Mr Cameron said the Labour leader was "chickening" out of facing the Greens and all "national parties" must be represented.
The exchanges came after Mr Miliband, Lib Dem leader Nick Clegg and UKIP leader Nigel Farage urged broadcasters to press ahead with televised debates before the 7 May poll even if Mr Cameron refuses to take part.
The three leaders have written to Mr Cameron saying it would be "unacceptable" for the prime minister to refuse to appear, saying an empty podium should be provided in his absence.
In response, the BBC, Sky News, ITV and Channel 4 said they "remain committed" to staging debates before the poll on 7 May.
Raising the issue in Parliament, Mr Miliband said Mr Cameron had argued back in 2010 that it would be "feeble" for any party leader to walk away from the debates but was now threatening to do just that.
A high stakes poker game in which the ultimate prize is power - that is what the row about TV leaders election debate has now become.
The players at the table are not just the nation's party leaders but also the bosses of the country's biggest broadcasters.
Watching and waiting are highly paid lawyers who know that this may end up being resolve by a judge and not by the court of public opinion.
The question is - will anyone blink before it gets to the court-room?
Read more from Nick
"It is not for him, it is not for me, it is not for any party leader to decide who is in the debate," he said. "It is up to the broadcasters, that is the country we live in.
"I think he does protest too much. He has run out of excuses, he is running scared of these debates and in the words of his heroine Margaret Thatcher 'he is frit'".
Mr Cameron said Mr Miliband wanted "to debate having a debate" because he did not want to talk about the economy and other issues.
He said he was keen to defend his record during the campaign, suggesting he would support the idea of two debates, one a head-to-head encounter between him and Mr Miliband and another in which all "national parties" were represented.
"The more time he and I can spend on television and in the television studio, the happier I will be," he said.
But Deputy Prime Minister Nick Clegg, who was sitting next to the prime minister, was heard shouting "excuses, excuses" as Mr Cameron answered one of Mr Miliband's questions.
In identical letters to Mr Cameron, Mr Miliband, Mr Clegg and Mr Farage called for a repeat of the 2010 debates - the first ever leaders' TV debates in a British election - and insisted they "not the property of the politicians".
They wrote: "I believe it would be a major setback to our democratic processes if these debates were not repeated in 2015 because of one politician's unwillingness to participate."
The added: "Therefore, if you are unwilling to reconsider, the three party leaders who have committed to participate will ask the broadcasters to press ahead with the debates and provide an empty podium should you have a last-minute change of heart."
Under plans put forward by the BBC, Sky News, ITV and Channel 4 in October, the pre-election live TV debates would include the Conservatives, Labour, Liberal Democrats and UKIP.
Mr Cameron would take on Labour leader Mr Miliband head-to-head in one debate, another would feature Mr Cameron, Mr Miliband and Lib Dem leader Mr Clegg, and a third would also include UKIP's Mr Farage.
The suggested schedule is for debates on 2 April, 16 April and 30 April, ahead of the general election on 7 May.
Mr Farage said the BBC would be "within its rights", under the terms of its editorial guidelines, to stage a debate without Mr Cameron and providing an empty podium would be "entirely appropriate".
"If David Cameron chooses not to turn up that is his hard luck," he told the BBC News Channel. "My guess is that if the broadcasters hold firm, Mr Cameron will buckle and say yes."
Lord Ashdown, who is leading the Lib Dems' general election campaign, urged broadcasters to "stick to their guns" or else he feared the impetus for the debates would "ebb away".
He told Sky News: "You just can't let the prime minister or anybody else basically hold the country and our democracy to ransom by saying no."
But Green Party leader Natalie Bennett said she backed Mr Cameron's stance and said it was in the interest of the three other leaders to help persuade broadcasters that her party should be included in one of the debates.
Ms Bennett said ITV had not yet made a final decision and she hoped to hold talks with the broadcaster in the near future.
"Staging the debates without the prime minister might score a point but would not serve the public, who rightly expect the political parties and the broadcasters to find a format that is acceptable to all concerned," she wrote in a letter to the trio.
"If you indicated that you were open to the inclusion of the Greens, then I feel sure that ITV would respond."
John Curtice, professor of politics at the University of Strathclyde, said that having "nailed his colours to the mast" over the Green Party's participation, it would be difficult for Mr Cameron to shun the debates if they were invited.
The three debates in 2010 were watched by a total of 22 million people, and they have been cited as one reason for the increase in turnout at the election, which rose for the first time since 1992.
While the debates were a "good test of political leadership", Prof Curtice told the BBC that they had dominated the campaign and "not necessarily succeeded in reaching out to more marginal voters".
The SNP and Plaid Cymru, which have more seats in Parliament than either UKIP or the Green Party, also say they should be represented in any series of debates.
SNP deputy leader Stuart Hosie said: "The idea that the SNP, bigger than UKIP and the Lib Dems combined in terms of membership, should be excluded from debates which are broadcast into people's homes in Scotland is just simply quite unacceptable."
In a statement, the four broadcasters said: "The debates played an important role in informing millions of our viewers in 2010 and we will continue to work with all the parties to ensure that they happen again in 2015".
The Guardian, Daily Telegraph and YouTube are proposing a separate digital debate, which they suggest could reach up to 55 million internet users. They have invited the Greens, UKIP, the Lib Dems, Labour and the Conservatives to take part.
David Cameron and Ed Miliband have accused each other of "running scared" over election TV debates, as they clashed at Prime Minister's Questions.