He also served on the Lynchburg city council, and as the city's mayor in 1830 and won re-election in 1832. He worked to bring a railroad to the hill city, and later once noted that local railroad contractors preferred slave labor rather than hiring white workers because they deemed the former "equally efficient, more moral and much easier managed," an assessment with which he agreed after hiring white labor to build an addition to his house. Elijah Fletcher also helped found St. Paul's Episcopal Church in Lynchburg and Ascension Episcopal Church in Amherst, Virginia. Elijah died at Sweet Briar on February 13, 1858, and was buried on his plantation. His letters would later be published, as would those of his brother Calvin. Upon Elijah Fletcher's death in 1858, his daughter, Indiana Fletcher Williams, inherited the Sweet Briar plantation. Around 1858 in New York City, Indiana met J. Henry Williams (1831–1889), a recent graduate of General Theological Seminary, who would marry her and move to Virginia in 1865. Williams won election to in the Virginia Constitutional Convention of 1868 and later served as clerk of Amherst County. After the American Civil War ended in 1865, slaves were emancipated, but several continued to work for pay and live at Sweet Briar and other plantations. Henry and Indiana Fletcher Williams' only child, Daisy (1867–1883) predeceased her parents. When Indiana Williams died a widow in 1900, she willed the land and much of her assets to a trust, and founded Sweet Briar College for women, which opened in 1906. Elijah's eldest son, Sidney was trained at Yale as well as in Richmond and Europe. He escorted his sisters on their European tour and briefly traveled to California with his brother Lucian, as well as practiced medicine for a time. Elijah had given Sidney the deed for Tusculum plantation in 1850, where he farmed and raised livestock until his death. Although Sidney never married officially, he had a long term relationship with a mulatto woman, and devised property to her children in 1870. He later asked a New York cousin to move to Virginia and help him and his widowed sister Indiana. John Jay Williams did so, and later learned that he had inherited the plantation, and kept it in the family for several more decades. In 2003, the Association for the Preservation of Virginia Antiquities acquired the property to prevent its demolition. Tusculum was listed on the National Register for Historic Places in 2004, but the house itself was dismantled in 2006 (archeological excavations having been conducted at various times) and plans to reassemble it on campus fell through, so it still awaits restoration.