Okay. So this is not an inherent issue of every pytorch network? We are currently testing with more than 10 different models, and all except for 1 display this behaviour, which made us believe that they is something inherently inefficient about pytorch. Is this something unusual?
I will get back to you guys if I find anything useful.