Solved by albanD in post #6 
              
              
                Ho ok. 
Then I would say: 

Find out which function is used in your case (that will depend on how you installed pytorch). And if you can, change this one the same way you changed the caffe implementation.
Use the cpp_extensions to create a custom cpp function that does what you want, and modify adam…