Thanks for your insight. Indeed, the problem comes from the dropout layer.
Even when I make sure the seed for both script are the same
import torch
import torch.nn as nn
import numpy as np
import random
SEED = 1234
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.cuda.manual_seed(SEED)
torch.backends.cudnn.deterministic = True

x_batch_first = torch.ones(2,3,4)
x_batch_second = torch.ones(3,2,4)

dropout_layer = nn.Dropout(p=0.5)

output1 = dropout_layer(x_batch_first)
output2 = dropout_layer(x_batch_second)
torch.equal(output1, output2.transpose(0,1))
output >>> False

When I print out both results, it gives me:
output1
tensor([[[0., 2., 0., 0.],
         [2., 0., 2., 0.],
         [0., 0., 2., 2.]],

        [[2., 0., 2., 2.],
         [0., 0., 2., 2.],
         [0., 2., 2., 0.]]])

output2
tensor([[[2., 0., 2., 0.],
         [2., 2., 0., 2.]],

        [[0., 2., 2., 2.],
         [2., 0., 2., 0.]],

        [[2., 2., 0., 2.],
         [2., 0., 0., 0.]]])

Any ways to make sure dropout layer still behaves the same when batch_first=True and batch_first=False?