I figured out the main problrm that was my installation. I hadn’t installed torch with cuda support and torch 1.4.1 didn’t send warnings about this. It said cuda was available which confused me, but I didn’t question it.
After I updated the torch package version 1.7.1 triggered an assertion that I had installed torch without cuda support and after reinstalling it works ok.
Now there isn’t a time difference between .to(“cpu”) calls and the normal cpu computation, which reassures me al lot (both come in at 7 seconds). If I use .to(“cuda:0”) it takes 11 seconds, which still is considerably slower then cpu, I can work with that and try out stuff.
I would appreciate if somebody with experience would comment on the way I define my net and call it in the training loop. To me it looks wrong, but I am to inexperienced to find a better way.