He also founded the Vanderbilt chapter of the Phi Beta Kappa Society, an honor society. Additionally, Vaughn was a renowned book collector and an avid reader. He read more than ten foreign languages, including Sanskrit and Russian. He owned about 6,000 books, five hundred of which were about the French Emperor Napoleon (1769-1821); they were all donated to the Vanderbilt University library after his death. On August 17, 1865, shortly after the Civil War, Vaughn married Abbie Maria Scott (1841-1923) in Marion, Alabama. He was thirty-one and she was twenty-four. They had three children:  Vaughn died on December 1, 1912 in Nashville, Tennessee. His portrait, taken circa 1850, can be seen in the William Stanley Hoole Special Collections Library at the University of Alabama. His grandson, William S. Vaughn (1903–1996), was a prominent businessman who served on the Board of Trust of Vanderbilt University from 1952 to 1995.