Deputy Prime Minster John Prescott says he accepts the use of grace-and-favour home Dorneywood was "getting in the way" of doing his government job.
Chancellor Gordon Brown chose not to occupy Dorneywood
He came under fire last weekend when pictures of him playing croquet on the lawns of the property near Burnham, Buckinghamshire, featured on newspaper front pages.
Dorneywood was gifted to the National Trust by industrialist Lord Courtauld-Thomson in 1947 as a country home for a senior member of the government.
The upkeep of the 200-acre estate is supported by the Dorneywood Thomson Endowment Trust Fund.
The 21-room Queen Anne-style house, comprising nine bedrooms and four reception rooms, was built in 1920 on the site of a mansion damaged in a fire.
Its grounds and gardens, noted for its cottage and kitchen styles, herbaceous borders and rose displays, is tended by the National Trust.
The public can visit its gardens on four days each year but the property, which also houses a swimming pool and billiards room, remains off limits.
Dorneywood's interior reportedly contains furniture and decorations belonging to the National Trust and Government Art Collection including work by Sir John Tenniel and Rex Whistler.
The prime minister decides which secretary of state or minister can
occupy Dorneywood, with the chancellor traditionally offered the run of the estate.
But Gordon Brown decided not to take up the privilege after Labour's general election victory in 1997.
A number of former prime ministers, including John Major, James Callaghan and Sir Anthony Eden, have, however, used Dorneywood before taking up top office.
When Alec Douglas-Home became prime minister in 1963 he was said to be reluctant to forsake the property for the antique surroundings of Chequers.