You are missing an nn.ReLU() module in PyTorch after the last conv layer and should also remove the nn.Softmax(1) layer as nn.CrossEntropyLoss expects raw logits and will internally apply F.log_Softmax and F.nll_loss.