 Joseph was offered the mastership at Hull's grammar school and invited Isaac to become the institution's usher.Through the patronage of his brother, Milner was subsequently freed from his duties in Hull and entered Queens' College, Cambridge, as a sizar in 1770.He graduated with a BA degree as senior wrangler in 1774, winning the Smith's first prize.