Jurgen Kantner was abducted from his yacht in November. The Abu Sayyaf group said it was behind the incident.
The body was retrieved on the island of Sulu and will be returned to Germany, officials said.
Mr Kantner's companion, Sabine Merz, was found shot dead aboard their abandoned yacht in November.
Last month, a video was posted showing the beheading of Mr Kantner by a knife-wielding man.
The German foreign ministry said in a statement that it was "deeply shocked by the inhuman and gruesome act".
Abu Sayyaf is one of the smallest and most violent jihadist groups in the southern Philippines, known for its brutality, including beheadings.
It has pledged allegiance to so-called Islamic State.
In recent months it has been behind a series of attacks at sea between the Philippines and Malaysia, with tug boats and fishing vessels intercepted and their crews kidnapped and held for ransom.
According to Abu Sayyaf, the couple were cruising off Malaysia's Sabah state when the militants intercepted them, the Filipino military say.
Mr Kantner, 70, and his companion had been held for 52 days in 2008 by Somali pirates and released after a ransom was paid.
The body of a German hostage beheaded by Islamist militants in the Philippines this week has been recovered, military officials say.