Are you using a batch size of 1 during training or just validation?
In the former case, I would recommend to remove the batch norm layers, as single samples will most likely create invalid running estimates, which would explain the bad validation accuracy.
You could try to use e.g. nn.InstanceNorm layers instead.