EDIT: just saw that the code sample uses mse loss, in that case im sorry no idea how I solved it, quite some time ago. Will post when I’m back but that is over a month away. Would recommend to search for some working pytorch implementation online and compare
Sorry, completely forgot to post the code. Unfortunately I am on summer break right now and have no access to the code.
The solution was in my case to use the mse loss instead of the (smooth) l1 loss (which was really unintuitive as the smooth l1 loss is explicitly recommend to prevent problems from the mse loss)
Never got round to actually find out why the l1 loss didn’t work for me, surely I made some implementation mistake.
Cheers, Johannes