The decision raises the possibility that Spain may be forced to hold a third election after seven months of uncertainty.
Mr Rajoy said he would try to form a new government if asked by the king.
The PP is the largest party in parliament but cannot form a coalition.
June's general election was the second in six months, after a poll in December was inconclusive.
Can Rajoy make new election victory count?
Podemos's Ikea-style appeal to young voters
The PP - which claimed Spain's improved economic performance showed its policies were working - won 137 seats in the 350-seat parliament, up 15 from the 122 they won in the December ballot.
But Pedro Sanchez, whose PSOE came second in June, insisted there would be no "grand coalition" of the main right- and left-wing parties, as has been seen in other European countries.
"Right now, his Popular Party has no support. Right now, the Socialist party reaffirms its vote against Rajoy," he said.
However Mr Sanchez also said he would do anything to avoid a third election, a prospect described by Mr Rajoy as "madness".
The left-wing Unidos Podemos alliance - which had been expected to become the country's main left-wing party but in fact lost more than a million votes and came third - has also rejected a deal with the PP.
Its leader Pablo Iglesias said the Socialists now had to come to an agreement with either the PP or Podemos or Spain would have to hold a third election.
Meanwhile centre-right Ciudadanos, which came fourth, said it was prepared to abstain in any vote over a possible new coalition to give it a chance of success.
MPs are due to assemble next Tuesday, after which King Felipe VI will consult party leaders and could nominate one of them to try to form a government.
December's election was a watershed for Spain, because the PP and the PSOE had previously alternated in power since the restoration of democracy in the 1970s.
There is no end in sight to Spain's political deadlock after the Socialist party leader again ruled out supporting re-election for acting PM Mariano Rajoy's People's Party (PP).