his paternal grandparents were dr. ezra isham and nancy (née pierpont) isham, and his maternal grandparents were dr. noadiah swift and jennet (née henderson) swift. his maternal grandmother nancy was the great-granddaughter of rev. james pierpont, the founder of yale university.