Born in Hampton, Connecticut, the son and grandson of Congregational ministers, at age 14 Weld took over his father's 100 acres farm near Hartford, Connecticut, to earn money to study at Phillips Academy in Andover, Massachusetts, attending from 1820 to 1822, when failing eyesight caused him to leave.