 Kirby was born in Falls Church, Virginia, on October 22, 1939, to John Joseph Kirby, a lawyer with the federal government for over 40 years, and Rose L. Mangan Kirby, a home maker.He had two brothers, Peter Kirby and Michael Kirby, and two sisters Lisa Greissing and Cecelia Wrasse.During his younger days, Kirby worked at the United States Department of Justice as the special assistant to the head of the Civil Rights Division, John Doar, during the height of the civil rights movement in the 1960s.