This is not my area, but couldn’t this problem be solved by adding random noise to the input data such that the adversary can’t control what is passed into system? You would trade off accuracy for robustness.