 Willoughby's only child, Joan, married Richard Welles, 7th Baron Welles (d.1470), by whom she had a son, Sir Robert Welles, and a daughter, Joan Welles.At Willoughby's death, his lands and title descended to his daughter and son-in-law, Richard de Welles, 7th Baron Welles.Willoughby married firstly, before 21 February 1421, Elizabeth Montagu, daughter of John Montagu, 3rd Earl of Salisbury, by whom he had an only daughter, Joan Willoughby, who married Richard Welles, 7th Baron Welles.