HI guys Pytorch newby here 
I have translated one of my models from TF Keras to Pytorch, the model matches exactly. I have used custom data augmentation that I have used with my Keras model for a number of years.
I have tested the shape x after each layer in forward and they are correct, they match the original model. At first the model seems to do quite well loss steadily decreases and accuracy slowly increases, then it hits a road block, accuracy stops increasing, and eventually loss starts going crappy.
Can anyone suggest what is happening, this model in Keras does extremely well on the augmented dataset.
Below is the network architecture and the output from training.
Net(
  (pad1): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
  (conv1): Conv2d(3, 30, kernel_size=(5, 5), stride=(1, 1))
  (pad2): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
  (conv2): Conv2d(30, 30, kernel_size=(5, 5), stride=(1, 1))
  (pool3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (flatten): Flatten(start_dim=1, end_dim=-1)
  (fc1): Linear(in_features=75000, out_features=128, bias=True)
  (relu): ReLU()
  (softmax): Softmax(dim=None)
)

Output:
Train Loss: 4.850 | Accuracy: 21.895
Train Loss: 4.847 | Accuracy: 29.684
Train Loss: 4.840 | Accuracy: 29.684
Train Loss: 4.830 | Accuracy: 29.895
Train Loss: 4.814 | Accuracy: 30.526
Train Loss: 4.787 | Accuracy: 30.947
Train Loss: 4.742 | Accuracy: 32.000
Train Loss: 4.677 | Accuracy: 34.737
Train Loss: 4.590 | Accuracy: 64.632
Train Loss: 4.494 | Accuracy: 70.316
Train Loss: 4.398 | Accuracy: 70.316
Train Loss: 4.301 | Accuracy: 70.316
Train Loss: 4.231 | Accuracy: 70.316
Train Loss: 4.199 | Accuracy: 70.316
Train Loss: 4.184 | Accuracy: 70.316
Train Loss: 4.178 | Accuracy: 70.316
Train Loss: 4.167 | Accuracy: 70.316
Train Loss: 4.166 | Accuracy: 70.316
Train Loss: 4.166 | Accuracy: 70.316
Train Loss: 4.161 | Accuracy: 70.316
Train Loss: 4.167 | Accuracy: 70.316
Train Loss: 4.161 | Accuracy: 70.316
Train Loss: 4.163 | Accuracy: 70.316
Train Loss: 4.164 | Accuracy: 70.316
Train Loss: 4.164 | Accuracy: 70.316
Train Loss: 4.161 | Accuracy: 70.316
Train Loss: 4.163 | Accuracy: 70.316
Train Loss: 4.165 | Accuracy: 70.316
Train Loss: 4.159 | Accuracy: 70.316
Train Loss: 4.165 | Accuracy: 70.316
Train Loss: 4.167 | Accuracy: 70.316
Train Loss: 4.168 | Accuracy: 70.316
Train Loss: 4.163 | Accuracy: 70.316
Train Loss: 4.170 | Accuracy: 70.316
Train Loss: 4.163 | Accuracy: 70.316
Train Loss: 4.160 | Accuracy: 70.316
Train Loss: 4.162 | Accuracy: 70.316
Train Loss: 4.162 | Accuracy: 70.316
Train Loss: 4.166 | Accuracy: 70.316
Train Loss: 4.161 | Accuracy: 70.316
Train Loss: 4.160 | Accuracy: 70.316
Train Loss: 4.165 | Accuracy: 70.316
Train Loss: 4.165 | Accuracy: 70.316
Train Loss: 4.168 | Accuracy: 70.316
Train Loss: 4.161 | Accuracy: 70.316
Train Loss: 4.165 | Accuracy: 70.316
Train Loss: 4.164 | Accuracy: 70.316
Train Loss: 4.161 | Accuracy: 70.316
Train Loss: 4.161 | Accuracy: 70.316
Train Loss: 4.165 | Accuracy: 70.316
Train Loss: 4.165 | Accuracy: 70.316
Train Loss: 4.164 | Accuracy: 70.316
Train Loss: 4.164 | Accuracy: 70.316
Train Loss: 4.160 | Accuracy: 70.316
Train Loss: 4.157 | Accuracy: 70.316
Train Loss: 4.158 | Accuracy: 70.316
Train Loss: 4.170 | Accuracy: 70.316
Train Loss: 4.159 | Accuracy: 70.316
Train Loss: 4.159 | Accuracy: 70.316
Train Loss: 4.165 | Accuracy: 70.316
Train Loss: 4.164 | Accuracy: 70.316

Thanks in advance.