He threatened to shoot me once when I played in the garden as a kid, and many weekends, shaven-headed National Front activists arrived at his house and emerged with scores of placards screaming that they wanted us to go back home.