 Milner was born on 11 January 1750 in Mabgate, Leeds.He began his education at a grammar school in Leeds in 1756, but this ended in 1760 with the death of his father.He was apprenticed as a weaver, reading the classics when time permitted, until his elder brother, Joseph Milner, provided him with an opportunity.