Solved by rohun in post #2 
              
              
                It was the Dropout in the middle. 
Works fine if the Dropout is left out, if say the net is the following - 
net = nn.Sequential( nn.Conv2d(50, 30, kernel_size=1, padding=1, bias=False), nn.BatchNorm2d(30,
momentum=.95), nn.ReLU(inplace=True), nn.Conv2d(30, 19, kernel_size=1))

Need to close this th…