His mother was the only daughter of Eliphalet Nott, who served as the longtime president of Union College.[1] His paternal grandparents were Joseph Potter and Anne Brown (née Knight) Potter and his uncle was Horatio Potter, the bishop of the Episcopal Diocese of New York.[4]