Thanks a lot for the response @ptrblck.
So the models are as follows.
Pytorch:
(cnn): Sequential(
(0): Conv2d(1, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): ReLU()
(2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(3): ReLU()
(4): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 3), padding=(1, 1))
(5): ReLU()
(6): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(7): ReLU()
(8): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(9): ReLU()
(10): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 3), padding=(1, 1))
(11): ReLU()
(12): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(13): ReLU()
(14): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(15): ReLU()
(16): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 3), padding=(1, 1))
(17): ReLU()
(18): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(19): ReLU()
(20): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(21): ReLU()
(22): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 3), padding=(1, 1))
(23): ReLU()
)
Keras:
conv1 = (Conv2D(16, (3,3), strides=(1, 1), activation=‘relu’, padding=‘same’))(re_input)
conv_out_1=conv1
conv1 = (Conv2D(16, (3,3), strides=(1, 1), activation=‘relu’, padding=‘same’))(conv1)
conv_out_2=conv1
conv1 = (Conv2D(16, (3,3), strides=(1, 3), activation=‘relu’, padding=‘same’))(conv1)
conv_out_3=conv1
    conv2 = (Conv2D(32, (3,3), strides=(1, 1), activation='relu', padding='same'))(conv1)
    conv_out_4=conv2
    conv2 = (Conv2D(32, (3,3), strides=(1, 1), activation='relu', padding='same'))(conv2)
    conv_out_5=conv2
    conv2 = (Conv2D(32, (3,3), strides=(1, 3), activation='relu', padding='same'))(conv2)
    conv_out_6=conv2
    
    conv3 = (Conv2D(64, (3,3), strides=(1, 1), activation='relu', padding='same'))(conv2)
    conv_out_7=conv3
    conv3 = (Conv2D(64, (3,3), strides=(1, 1), activation='relu', padding='same'))(conv3)
    conv_out_8=conv3
    conv3 = (Conv2D(64, (3,3), strides=(1, 3), activation='relu', padding='same'))(conv3)
    conv_out_9=conv3
    
    conv4 = (Conv2D(128, (3,3), strides=(1, 1), activation='relu', padding='same'))(conv3)
    conv_out_10=conv4
    conv4 = (Conv2D(128, (3,3), strides=(1, 1), activation='relu', padding='same'))(conv4)
    conv_out_11=conv4
    conv4 = (Conv2D(128, (3,3), strides=(1, 3), activation='relu', padding='same'))(conv4)
    conv_out_12=conv4

Messy formatting, sorry, but they should be 100% equivalent. The outputs are as follows (I print the mean and std of the output vectors at each of the 12 Conv layers):
Pytorch:
mean 0.1638 std 0.2410
mean 0.1336 std 0.2011
mean 0.1350 std 0.1917
mean 0.0607 std 0.1112
mean 0.0305 std 0.0819
mean 0.0276 std 0.0563
mean 0.0128 std 0.0345
mean 0.0222 std 0.0392
mean 0.0333 std 0.0613
mean 0.0319 std 0.0750
mean 0.1114 std 0.1933
mean 0.3793 std 0.6193
Keras:
mean 0.1639 std 0.2410
mean 0.1339 std 0.2010
mean 0.1360 std 0.1920
mean 0.0608 std 0.1117
mean 0.0305 std 0.0825
mean 0.0277 std 0.0565
mean 0.0130 std 0.0349
mean 0.0222 std 0.0395
mean 0.0337 std 0.0616
mean 0.0299 std 0.0726
mean 0.1088 std 0.1867
mean 0.3760 std 0.6399
As you can see, the difference is very small in the beginning but propagates into something larger. I also have feed the outputs of this into a BLSTM and then into FC layers so the error is propagated even further, but I want to deal with this issue first. Do you believe there is any way to fix this? Thanks!
Edit: BTW, I compared the outputs after the first layer, without activation, and they are indeed also not the same:
Pytorch:
mean -0.0057 std 0.4149
Keras:
mean -0.0054 std 0.4144