Thanks a lot. I tried using sigmoid for squashing the patch values from 0 to 1 and now I am successfully able to attack test data with 68% accuracy. But the problem is, the patch values even after using sigmoid are either 0 or 1. And the coordinate prediction is always being done in the top left part of each image. I am attaching a sample output that my model is predicting and the patch is in almost same place for all images.
Screenshot from 2019-02-19 13-04-42.png1337×784 56.5 KB
 .
For values I am thinking about trying to add norm of patch to my loss function to keep it’s value low.