Felton was the oldest of four children; her sister, Mary Latimer, also became prominent in women's reforms in the early 20th century. When she was 15, her father sent her to live with relatives in the town of Madison where she attended a private school within a local Presbyterian church. She then went on to attend Madison Female College, from which she received a classical liberal arts education.[4] She graduated at the top of her class at age 17 in 1852.[5]