Cavendish was born in Douglas, Isle of Man, the son of David Cavendish also from the Isle of Man, and Adele from Yorkshire. He began riding BMX at a young age, racing at the National Sports Centre in Douglas. It was at that time that Cavendish met David Millar at a race on the Isle of Man. Cavendish said he was inspired by Millar.