Conservative Matthew Grove defeated the Labour veteran in the battle for the Humberside Police role.
The vote went to a second round of counting with Mr Grove getting 42,164 votes and Lord Prescott 39,933.
Mr Grove said Lord Prescott was an "utter gentleman".
Both candidates had failed to get 50% of first preference votes.
Lord Prescott secured 33,282 votes, the most in the first round, with Mr Grove polling 29,440 votes.
The turnout in the election was 133,762 (19.15% of the electorate)
Speaking after the result had been declared, Lord Prescott said: "I said at the beginning, this a Tory marginal seat. It's not a safe Labour seat. It's not even a Labour seat."
Asked if he will be putting himself up for election again, he said: "I've always been in public service rather than anywhere else. And I don't think I'll be standing for election, no.
"But this was one occasion where you were required to have lived in the actual constituency, I was one of the constituency, I always wanted to make service so I put my name forward and you have the results today.
"I nearly got it, didn't I?
"It would have been nice but it wasn't so. The people have spoken."
"It's nothing to be defeated in a democracy, is it?"
He laughed when  asked if his wife, Pauline, wanted him to retire.
Mr Grove paid tribute to Lord Prescott, saying: "I've enjoyed his company, and it's with slight sadness I feel I've beaten him.
"In some ways he's given such long and dedicated service to this area, it's a shame I had to beat him, but it had to be done."
The new PCCs, who are set to replace police authorities, will set spending plans and have the power to "hire and fire" chief constables.
The Humberside police commissioner will receive a salary of Â£70,000.
The East Riding of Yorkshire council area saw the highest turnout in the region with 23.19%, while 15.65% of the electorate voted in Kingston upon Hull.
The other candidates vying for the Humberside police post were Godfrey Bloom of UKIP, Simone Butterworth from the Liberal Democrats, and independents Paul Davison, Neil Eyre and Walter Sweeney.
Turnout does not include spoiled ballots
Former deputy prime minister John Prescott says he will not stand for an election again after failing in his bid to become a police and crime commissioner.