Tozer was born in Plymouth, Devon, the eldest son of Capt. Aaron Tozer of the Royal Navy.[2] After graduating from Exeter College, Oxford, in 1850, he was tutor there from 1855 to 1893 and was also curator of the Taylor Institution (Oxford) from 1869 to 1893.[3] His brother-in-law was Ernest Mason Satow.