The couple had five children together. His daughter Mary married the politician and diplomat Sir Alick Downer; their son Alexander Downer also entered politics. A nephew, George Gosse, was a decorated soldier.