The house has been extended and remodelled several times by prominent architects such as James Adam, John Adam, and David Bryce. It has been described by the architectural historian Charles McKean as "one of the grandest houses in Scotland" and is designated a Category A listed building. The grounds were enlarged in the 1820s when the entire village of Cullen, save for Cullen Old Church, was demolished to make way for improvements to the grounds by Lewis Grant-Ogilvy, 5th Earl of Seafield; a new village, closer to the coast, was constructed for the inhabitants.