 Gilmer was born in Birmingham, Alabama, where he attended and played high school football at Woodlawn High School.Gilmer often utilized the technique of leaping high into the air to pass the ball because, as a child, he often played pickup games with teammates who were much older and thus taller than he was; Gilmer was then one of the first players to popularize the "jump pass" when he continued using the technique at the collegiate level.After high school, Gilmer played college football at the University of Alabama, where he was the left halfback from 1944 to 1947.