I reimplemented the Network in DGL (Pytorch Backend) instead of pure Pytorch. The problem is gone, but my cpu is way more faster than gpu. In the end of the day I would like to have the Network in Pytorch.