Colenso married Sarah Frances Bunyon in 1846, and they had five children, Harriette Emily, Frances Ellen, Robert John, Francis "Frank" Ernest, and Agnes. (In the marriage register, her name is spelt Bunyan. There had long been variations in the spelling of a surname that goes back at least to the 12th century in England and in Normandy.)