He was born into a prominent Maryland family; among his ancestors was Captain James Neale, one of the settlers of the Maryland Colony, who arrived in 1637 upon receiving a royal grant of 2000 acres in what would become Port Tobacco.