looks like it has to do the memory footprint in some way. If I decrease the network size or the input size everything is fine. Even though in general the GPU is no where close to maxing out when looking at msi