There might be some differences and issues in your PyTorch code:

it seems you are using half the number of filters in each conv layer compared to the Keras model
in your Keras implementation you are applying: conv - relu - pool - bn, while in Pytorch you are using: conv - bn - relu - pool
Dropout is always activated in your PyTorch model. Use the nn.Dropout module or pass training=self.training