 His maternal ancestors were of Irish descent.His brother-in-law, Nathaniel G. Taylor (married to his sister, Emaline), was a Whig congressman and Unionist.Taylor's sons (Haynes's nephews) included Alfred A. Taylor (1848–1931) and Robert Love Taylor (1850–1912), who would both serve as Governor of Tennessee.