buller was from a prominent cornish family that included sir richard buller, general sir george buller, general sir redvers buller and the barons churston. he married mary down on 4 may 1801 and had seven sons and three daughters. he died in kensington in 1849.