Sir Bob Kerslake said one of the difficulties was that "easier" savings had already been made.
He said the "sense of urgency" would be reduced and the need for cuts would be hard to explain to public sector staff.
Sir Bob stepped down from his role this month and was replaced by former Cabinet Secretary Sir Jeremy Heywood.
He said the civil service had "excelled" during his time in charge.
Speaking at the Institute for Government think tank, he said it was clear that "under any government, we face up to a further five years of austerity in public sector spending".
"The first five years have been challenging but the second five years are likely to prove even harder for three reasons," he said.
"Firstly, the easier savings have already been made.
"Secondly, we are likely to be doing it against a background of a growing economy and greater competition for good staff.
"Thirdly, the sense of urgency that underpinned the first savings programme will be reduced.
"In reality, the task is not yet complete. But this will be hard to explain to those in the public sector, including our own staff, who are looking for some relief."
Sir Bob said the civil service had "excelled" in its challenge to "deliver big programme savings at the same time as undergoing huge change and reduction itself".
The roles of cabinet secretary and head of the civil service were split into two jobs when Sir Gus O'Donnell retired as both in 2012.
Sir Bob, who was already permanent secretary at the Department for Communities and Local Government (DCLG), took the role as head of the civil service at that time.
He will retire from his DCLG role early next year.

The company said it wanted to focus development "on the mobile platforms the vast majority of people use".
Facebook-owned WhatsApp, which is used by a billion people worldwide, will stop working on the named operating systems by December 2016.
But it will still work on Blackberry's latest smartphone which runs Android.
Most of the operating systems that WhatsApp is dropping support for are legacy operating systems, which are no longer updated or installed on new devices.
The exception is Blackberry 10, which was launched in January 2013 and is still being developed by Blackberry.
"We are also planning version 10.3.4 for later this year with even more security improvements," the firm said in January 2016.
However, the operating system has failed to gain traction with smartphone users and now accounts for less than 1% of the market.
There had been speculation that the handset-maker would wind down support for Blackberry 10 after it released a smartphone running Android, and closed its "Built for Blackberry" programme for app developers.
But the firm has insisted: "We're not abandoning the loyal customers who have contributed to our success."
The full list of operating systems WhatsApp will stop supporting is:
"While these mobile devices have been an important part of our story, they don't offer the kind of capabilities we need to expand our app's features in the future," WhatsApp said in a blogpost.
"When we started WhatsApp in 2009... about 70 percent of smartphones sold at the time had operating systems offered by BlackBerry and Nokia."
The firm said mobile operating systems offered by Google, Apple and Microsoft accounted for 99.5% of sales today.
The outgoing head of the civil service has predicted another five years of UK government spending cuts - and he said making the cuts would be "even harder".
[NEXT_CONCEPT]
WhatsApp is to end support for a number of operating systems including Blackberry 10, Nokia Symbian S60 and Windows Phone 7.1.