After reading a little and trying running it w/o cuda is does seem to backprop twice on a cpu. I suspect there is a bug there due to experiencing a different behavior so I reported on it pytorch github issues. I hope they would be able to address it