I was trying in a computer without GPU and it was working fine.
After some searches I found that it looks like a known pytorch bug. See this github issue 59.
Then I checked all the variables I was trying to pass to GPU if they were in fact on GPU, and some of them were NOT!
Then I forced they to be on GPU since the instantiation and my code worked fine after this.
My problem was solved