The Davises were devastated in 1854 when their first child died before the age of two. Varina Davis largely withdrew from social life for a time. In 1855, she gave birth to a healthy daughter, Margaret (1855–1909); followed by two sons, Jefferson, Jr., (1857–1878) and Joseph (1859–1864), during her husband's remaining tenure in Washington, D.C. The early losses of all four of their sons was a source of enormous grief to both the Davises.