difference in the order of 1e-5 is very reasonable to expect in a forward pass.
The difference could be one of many things:

Keras using no cudnn while pytorch is?
keras calling a different CuDNN kernel while pytorch is? (cudnn has many algorithms, direct convolution, FFT, Winograd, etc.)

A 1e-5 difference is not a big deal.
if you really wish to get precision down even further, try:
torch.backends.cudnn.enabled = False

See if that helps…