Later, in the 1850s, under President Pierce, he was known to have remarked — probably the only thing he's known for — when a neighbor passed by and said, "I'd love to see the beautiful house," and Pierce said to him,