caswell was born in wednesbury and began his career with walsall making his debut in the 1972–73 season. he became the regular in the "saddlers" side whilst left back was the key position he was able to play in all outfield positions and was a key member of the side which lost narrowly to liverpool in the semi-final of the league cup in 1983–84. after racking up 400 league appearances for walsall he joined billy bremner at doncaster rovers however in november 1985 bremner joined leeds united and he signed caswell for £30,000. injuries disrupted his time at elland road and after spending a short time out on loan at wolverhampton wanderers he decided to retire from playing. he then joined birmingham city's community team and then became youth coach. in 1992, he became youth coach at stoke city until 1995 when he became assistant manager to wayne clarke at telford united. he then returned to being youth coach at northampton town and shrewsbury town before leaving football in 2002 to work for bmw in birmingham.