Jeffery attended Calhoun County High School in St. Matthews, South Carolina, where he was part of a four-time state championship winning (80–1 record) basketball team.[1] He also played two years of football and was widely recruited, giving a verbal commitment to the University of Southern California before switching to South Carolina.[2] Jeffery's younger brother Shamier Jeffery committed to play football for the Gamecocks in 2011, following in the footsteps of both of Jeffery's two older brothers Charles Ben and Darren Ben; both of whom also played basketball and football for Calhoun County High School.