Why must an oxidizer not touch rubber? Why is it that oxidizers (specifically $\ce{KClO3}$) are at risk of causing a fire when they come into contact with the rubber stopper used to contain them in their container, but can be slowly heated if they do not contact the rubber stopper?

Essentially, why is it that oxidizers are incompatible with rubber?
A lot of rubbers possess C=C double bonds. Alkenes are very easily oxidized by KMnO4 or KClO3, therefore in the case where you're not aware of the chemical nature of the rubber, it's best not to expose it to strong oxidizers.