I was able to get a reasonable difference (around 1e-5) between the first-layer outputs of both framework by changing the permute to [4, 3, 2, 0, 1].
But the second layer generates different outputs.
I have checked that GroupNorm with group 1 is really close to LayerNorm.
The layer just looks like this:
        self.conv1_1 = nn.Sequential(
            nn.Conv3d(in_c, nf, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
            nn.GroupNorm(1, nf)
        )
        self.conv1_2 = nn.Sequential(
            nn.Conv3d(nf, nf*2, kernel_size=3, stride=2, padding=1),
            nn.ReLU(inplace=True),
            nn.GroupNorm(1, nf*2)
        )

The big difference happens at the output of conv1_2.
Does anyone know how to fix this?