Morgan married Connie H Morgan in 2010 and has been married three times before that. In 1953, he married Elizabeth Robbins Choate (1933–1998), the daughter of Robert Burnett Choate and the sister of Robert B. Choate Jr.[18] Before their divorce in 1957, they had: