Unesco concern at Tower of London and Westminster buildings
A team visiting the Unesco World Heritage sites the Tower of London, the Palace of Westminster and the Abbey is concerned about their "visual setting".
A World Heritage Committee official said there were no "buffer zones" between the sites and 20 ongoing or new developments to protect the views.
If the team decides the sites are threatened, it could place them on the UN's Heritage in Danger list.
The government said it could not afford to see the iconic monuments downgraded.
A spokesman for the Department for Culture, Media and Sport (DCMS) said if the sites - which include St Margaret's Church in the Westminster group - appeared to be heading for the Heritage in Danger List when the committee met in June 2012, it would bring "bad publicity" ahead of the Olympics.
There are 28 World Heritage Sites in the UK.
The delegation is on a four-day visit to the capital and has met Mayor Boris Johnson.
It will also meet with officers from Southwark and Westminster Councils.
The delegation last inspected the sites in 2006, when it said the Tower of London was becoming "overshadowed" by new buildings.
The current inspection will look at whether the 2006 recommendations were implemented and study the impact of new buildings built since then, for instance the Shard.
Patricia Alberth, an official with the committee's Europe unit, said the team would look at "proposed and current developments around the two heritage sites in London; the visual integrity of the sites and the conduction of visual impact assessments prior to development, which has not been sufficiently done in the past.
"The majority of World Heritage sites have a buffer zone which guarantees the protection of the sites, that's not the case with the two sites in London, that's a major concern of the World Heritage Committee."
The DCMS spokesman said being featured on the Heritage in Danger List "certainly downgrades their status and that is very bad news", but it will not have an impact on the permission granted for new developments or result in a fine for the councils or the government.
"The last thing the government wants is to see two iconic monuments becoming in danger of losing World Heritage status," he added.
The mayor's spokesman said: "These historic parts of London are of truly global significance and the mayor has outlined to Unesco improvements he has made to planning policy and guidance, which are designed to protect them."