To escape racial discrimination and have more opportunities, his children passed into white society: the four surviving sons married white spouses; his unmarried daughter lived with a brother. They worked as teachers, a lawyer, and business people. Smith's unique achievements as a pioneering African-American doctor were rediscovered by 20th-century historians. They were relearned by his descendants in the twenty-first century, who identified as white and did not know about him, when a three-times-great-granddaughter took a history class and found his name in her grandmother's family bible. In 2010, several Smith descendants commissioned a new tombstone for his grave in Brooklyn. They gathered to honor him and their African-American ancestry. Smith was born into slavery in 1813 in New York City and was set free on July 4, 1827, at age 14, by the Emancipation Act of New York. That was the final date when New York officially freed its remaining slaves. His mother was an enslaved woman named Lavinia who achieved her freedom later in life; in 1855, Smith described her as a "self-emancipated woman." She was born into slavery in South Carolina and had been brought to New York as a slave. His father was Samuel Smith, a white merchant and his mother's master, who had brought her with him to New York from South Carolina. The boy grew up only with his mother. As an adult, James Smith alluded to other white ancestry through his mother's family, saying he had kin in the South, some of whom were slaveholders and others slaves. Smith attended the African Free School (AFS) #2 on Mulberry Street in Manhattan, where he was described as an "exceptionally bright student". He was among numerous boys from the school who went on to have brilliant careers, some of whom he worked with as adults in the abolitionist cause. In the course of his studies, Smith was tutored by Rev. Peter Williams, Jr., a graduate of the African Free School who had been ordained in 1826 as the second African-American priest in the Episcopal Church. Upon graduation, he applied to Columbia University and Geneva Medical College in New York State, but was denied admission due to racial discrimination. Williams encouraged Smith to attend the University of Glasgow in Scotland. He and abolitionist benefactors of the AFS provided Smith with money for his trip overseas and his education. Smith kept a journal of his sea voyage that expressed his sense of mission. After arriving in Liverpool and walking along the waterfront, he thought, "I am free!"