It is learning now, tank you. However, it seems to be extremly slow. I compared the P-Net with only 93878 parameters against a normal U-Net with  30786048 parameters (same data, same hardware). Even though the P-Net has so few parameters it is training 3x slower than the U-Net.
Why could this be the case?
Could it be because in my forward I am computing the compression first and only then it is forwared to the next block? I am currently looking into torch.cuda.Stream(). Can this be used for speeding it up on the same gpu?