Solved by Sharpy in post #2 
              
              
                Oops, seems like that delay was mostly caused by me not removing 
torch.autograd.set_detect_anomaly(True) 

after a debug. Might be “mostly”, as this 
class HalfReLU(nn.Module):
	def __init__(self):
		super(HalfReLU, self).__init__()

	def forward(self, x):
		half = x.shape[1]//2
		y = nn.functional…