Tripoli residents said men with bulldozers attacked the shrine of al-Shaab al-Dahmani, unimpeded by police.
The attack came a day after hardliners were accused of damaging the tomb of a Sufi scholar in the city of Zlitan.
Hardline Salafists regard the shrines as idolatrous.
On Friday, a group attacked the tomb of 15th-Century scholar Abdel Salam al-Asmar in Zlitan, about 160km (100 miles) south-east of Tripoli. The Reuters news agency said its dome had collapsed.
Video footage showed chunks of masonry littering the floor, bullet holes pockmarking the walls and ornate Islamic tiling destroyed.
People in Tripoli say they saw bulldozers destroy part of the al-Shaab al-Dahmani mosque and Sufi shrine.
One, a student named Abdurrahman, told the BBC: "There's a large group of Salafists - they are the one with the bulldozers, and some military police are also present.
"They seem to be overseeing the process, rather than preventing it... There are some bystanders who seem to approve".
He said the Salafists were also handing out pamphlets issued by a Saudi Arabian mufti from the hardline Wahhabi school of Islam.
The destruction in Zlitan follows two days of clashes between rival local tribes which left at least three people dead.
Omar Ali, an official from the Zlitan military council, told Reuters: "The extremist Salafis took advantage [of the fact] that security officials were busy calming down the clashes and they desecrated the shrine."
Libya's Deputy Prime Minister Mustafa Abushagur tweeted: "The destruction of shrines and mosques is a crime. Those who commit these crimes will be held responsible."
In November last year, the bodies of two Muslim clerics were removed from the Sidi Nasr shrine and mosque in Tripoli and reburied according to the principles of the hardline Wahabi school of Islam.
There has recently been an international outcry over the destruction of centuries-old shrines in Timbuktu, Mali.
The Sufi sites were attacked by the al-Qaeda-linked Ansar Dine group which seized the city in April.
A shrine in the Libyan capital Tripoli venerating a Sufi Muslim saint has been partly destroyed - the latest in a series of attacks blamed on ultra-conservative Salafi Islamists.