Swayne was born in Frederick County, Virginia in the uppermost reaches of the Shenandoah Valley, approximately 100 miles northwest of Washington D.C. He was the youngest of nine children of Joshua Swayne and Rebecca (Smith) Swayne.