Thanks again for your reply! I was finally able to figure out my issue. As you said, trying it out on a small example did make me understand it better!
My issue was I had to change this:
x.weight = nn.Parameter(torch.from_numpy(TF_CONV1_KERNEL.transpose(3, 2, 0, 1)))
x.bias = nn.Parameter(torch.from_numpy(TF_CONV1_BIAS))
To this:
x.weight.data = torch.tensor(TF_CONV1_KERNEL.transpose(3, 2, 0, 1), dtype=torch.float)
x.bias.data = torch.tensor(TF_CONV1_BIAS, dtype=torch.float)
Meaning, you need to explicitly specify the type when assigning a tensor back to the convolutional layer in PyTorch.