The Court of Cassation rejected Mr Sarkozy's challenge to the seizure.
The diaries were initially confiscated as part of an inquiry into alleged illegal funding during his successful 2007 presidential campaign.
That case has since been dropped, but the diaries may be used in other investigations targeting Mr Sarkozy.
The former president, who lost his re-election bid in 2012, is planning a political comeback and correspondents say the drip of allegations has harmed him.
The diaries were seized after claims surfaced in 2010 that Mr Sarkozy had taken advantage of 90-year-old L'Oreal heiress Liliane Bettencourt when he was standing for president.
It was alleged that Mrs Bettencourt had given large sums of cash to Mr Sarkozy's aides. Both Mr Sarkozy and Mrs Bettencourt denied this. Last October, Mr Sarkozy was removed from the list of defendants.
Lawyers for Mr Sarkozy argued that confiscating the diaries had been illegal.
But in Tuesday's ruling, the Court of Cassation - France's top court - decided there was no need to rule on the issue as Mr Sarkozy was no longer a suspect in the case.
The ruling comes days after Le Monde newspaper reported that Mr Sarkozy had recently had his phone tapped on orders from judges investigating alleged campaign donations from late Libyan ruler Muammar Gaddafi.
The papers said the phone taps had revealed evidence of tampering with the justice system. Mr Sarkozy denies the claims.
French media say the diaries could be used in this case, and also in an investigation into allegations that French tycoon Bernard Tapie received a huge payout in 2007 to settle a long-running legal battle with the French state.
In a separate development regarded as embarrassing for the former president, transcripts of recorded conservations taped by a former aide of Mr Sarkozy were leaked last week.
Mr Sarkozy and his wife Carla Bruni say they were recorded without their consent.

The 34-year-old Warwickshire captain has played 118 Test matches, scoring 7,727 runs at an average of 42.69, but last represented his country in 2015.
Bell, who has 18,533 runs in first-class cricket, is still ambitious to regain his England Test place.
He said he was "thrilled" to sign a new contract with the club he first joined as a 10-year-old.
"Warwickshire is the only county that I've ever wanted to play for," he added.
"The club has played a huge role in my career and it's a great honour to now captain the team and help our young, talented players fulfil their potential here at Edgbaston."
Bell has been part of an Ashes-winning team on five occasions, helping England win the urn in 2005, 2009, 2010-11, 2013 and 2015.
He has played in a total of seven series against the Australians, having also been a part of the touring side in 2006/07 and 2013/14.
In domestic cricket, Bell has won six trophies, including captaining the Bears to success in last summer's Royal London One-Day Cup in his first season as skipper.
He is the most capped international cricketer in Warwickshire's history, having played 161 one-day internationals and eight T20 matches for England, in addition to his 118 Test appearances.
Warwickshire sport director Ashley Giles said: "Ian is a true Bear who has been with Warwickshire since the age of 10 and has gone on to be one of the most decorated players in the club's history.
"He continues to be one of the leading batsmen in the country across all three formats and, as our captain, he sets exceptionally high standards for the rest of the squad."
A French court has ruled investigators can retain the seized diaries of ex-President Nicolas Sarkozy, who is battling corruption allegations.
[NEXT_CONCEPT]
England batsman Ian Bell has signed a new deal with Warwickshire which runs until the end of the 2020 season.