Im going to try and figure this out when I have time. Im not too familiar with the C code that pytorch is using, thanks for suggesting what the error could be!
As the model uses custom layers the backward implementation might be flawed somewhere. The custom network (flownet2) was written in python2.7 using pytorch 0.2 which is not what I am using so maybe that can also be the source of the error?
I will update this thread if I find time to go through all the code