 Sir Alfred was the son of John Stephen, a judge of the Supreme Court of New South Wales.Stephen married Virginia, daughter of Matthew Consett, who died in 1837, and Eleanor daughter of the Rev.William Bedford, who died in 1886.