Ho ok.
Then I would say:

Find out which function is used in your case (that will depend on how you installed pytorch). And if you can, change this one the same way you changed the caffe implementation.
Use the cpp_extensions to create a custom cpp function that does what you want, and modify adam to use that instead of the plain .add.
You can also use the cpp extension to easily set a global flag in cpp to enable/disable your special code.

Would that help?