Thanks Alban for the quick answer!
It solves some of my problems, that is perfect. I did some changes to my code to fix this. However, I know that I am still doing an in-place operation on the memory and/or hidden line of code. I dont have a clue of how to fix it.
# operations
reset = torch.sigmoid(self.r[layer](torch.cat([input_, hidden[layer]], 1)))
forget = torch.sigmoid(self.z[layer](torch.cat([input_, hidden[layer]], 1)))
memory_temp = torch.tanh(self.h[layer](torch.cat([input_, torch.mul(reset, hidden[layer])], 1)))
hidden[layer] = torch.mul((one_matrix-forget), hidden[layer]) + torch.mul(reset, memory_temp)

Would the solution to have the formula of memory_temp and hidden[layer] be dependant on the clone() of hidden would make sense? I did clone just before those operation and it seems to work. However, would that still work with backprop?
Thanks again for your help!