Solved by albanD in post #2 
              
              
                Hi, 
Could you try to define your relus not inplace by using  uprelu = nn.ReLU(False). It is possible that one of the operations just before the relu needs its output to compute its gradient and so the relu operation change it inplace.