What is your downside to having them? Follower account increases your propagation rate, if that is important to you. 

Also, why should you be the arbiter of others' interests? I would be inclined not to cull, unless they are using your channel to distribute some repulsive content.