Anne and her family moved back into the house, passing it to her daughter and granddaughter, the 3rd and 4th Countesses, who also lived there. On the night of 20 May 1795, the house caught fire and burned down. The cause of the fire was thought to have been a servant knocking over a candle. The 4th Countess moved out after the fire but retained the site, only selling it in 1824. After the fire of 1795, the house remained a ruin for the first quarter of the 19th century. In 1824, the estate was purchased by Sir George Warrender, 4th Baronet. To rebuild Cliveden, Warrender selected William Burn, a Scottish architect, and decided on a design for a two-storey mansion with entertaining on a grand scale in mind. Warrender died in 1849 and the house was sold to the Sutherland family, headed by the second Duke. Sutherland had been in possession of the estate for only a few months when the house burned down for the second time in its history. The cause this time appears to have been negligence on the part of the decorators. The Duke was prompt in commissioning the architect Charles Barry to rebuild Cliveden in the style of an Italianate villa. Barry, whose most famous project is arguably the Houses of Parliament, Westminster, was inspired by the outline of the two earlier houses for his design. The third (and present) house on the site was completed in 1851–52, and its exterior appearance has little changed since then. The -tall clock tower, which is actually a water tower (still working to this day) was added in 1861 by the architect Henry Clutton. Also around this time another architect, George Devey, was commissioned to build half-timbered cottages on the estate along with a dairy and boathouse. After the duke's death in 1861, his widow Harriet continued to live at the house for part of the year until her death in 1868, after which it was sold to her son-in-law Hugh Lupus, Earl Grosvenor, later 1st Duke of Westminster. Westminster was one of the wealthiest Englishmen of the period and during his ownership of the estate (1868–93), he contributed significant additions to the house and gardens including the "porte cochère" on the north front of the mansion, a new stable block and the dovecote, all designed by Henry Clutton. In 1893, the estate was purchased by an American millionaire, William Waldorf Astor (later 1st Viscount Astor), who made sweeping alterations to the gardens and the interior of the house, but lived at Cliveden as a recluse after the early death of his wife.