The announcement came after a pact was agreed between the Socialists and centre-right Ciudadanos (Citizens).
"What has been agreed with Ciudadanos is incompatible with us," said Podemos's Inigo Errejon.
It deals a blow to Socialist leader Pedro Sanchez's hopes of becoming PM.
Between them, the Socialists and Ciudadanos command only 130 seats in the 350-seat lower house -  leaving Mr Sanchez far short of a parliamentary majority.
In a vote on 1 March, he will seek the support of deputies in a vote of confidence that the former ruling Popular Party (PP), led by acting Prime Minister Mariano Rajoy, has indicated it is likely to oppose.
Podemos now appears less likely to offer its support - overtly or tacitly, in the form of abstentions - now it has framed the pact between the Socialists (PSOE) and Ciudadanos as "an invitation to the PP".
"For the moment we have failed," Mr Errejon was quoted as saying. "The change [we sought] was not just to remove Mr Rajoy from the Moncloa [prime ministerial residence], but his policies."
Earlier, signing a deal based on a five-point plan for constitutional reform, Ciudadanos leader Albert Rivera had urged other parties to come on board.
Podemos's radical policies are at odds with the liberal-centrist slant of Ciudadanos, and it had already warned they were unlikely to join forces, but it was not clear that Podemos would withdraw from the negotiating table entirely.
Should Mr Sanchez fail to secure an absolute majority on 2 March, he could then aim for a simple majority in a second vote on 5 March.
If he fails, the PP may attempt once again to form a coalition - perhaps a "grand coalition" with the Socialists and Ciudadanos.
If that fails, a new election would have to be called, probably on 26 June.
Mr Rajoy's centre-right PP, which took the greatest number of seats in the election, has already tried to form a coalition without success.

About 200 people have been killed in the protests since November, Human Rights Watch (HRW) said.
The unrest began over government plans to expand the capital's administrative control into the region.
The government later dropped the plan, but protesters dismissed the decision as "too little too late".
Oromia is Ethiopia's largest region, surrounding the capital, Addis Ababa.
Many people in the region complain of being politically and economically marginalised in a country where the Ethiopian People's Revolutionary Democratic Front (EPRDF) coalition has ruled since 1991.
New York-based HRW said, in a report, that anti-government protests had decreased in the last few weeks, but the "crackdown continues".
"Almost daily accounts of killings and arbitrary arrests have been reported to Human Rights Watch since 2016 began," it added.
HRW said thousands of people had been detained without charge since November.
But Ethiopia's Communications Minister, Getachew Reda, told the BBC that it was a "stroke of magic" for HRW to release a report "from half way across the world".
He acknowledged that there had been trouble, but said attacks on public buildings were carried out by armed gangs "who are trying to stir up emotions in the public".
Mr Getachew said that HRW has been "churning out report after report" on Ethiopia without a presence in the country, and its allegations against the security forces were an "absolute lie".
At the last census in 2007, the Oromo made up Ethiopia's biggest ethnic group, at about 25 million people out of a population at the time of nearly 74 million.
Spain's anti-austerity party, Podemos, says it is suspending negotiations over forming a government with the Socialists as the country struggles to resolve inconclusive elections.
[NEXT_CONCEPT]
Ethiopia has dismissed allegations it is violently suppressing protests in its restive Oromia region as an "absolute lie".