Thanks again for your many suggestions. I was finally able to figure out my issue. As you said, I had to rotate the weights when transferring from TensorFlow. The rotation that I applied was transpose(3, 2, 0, 1). However, this alone was not enough, I also had to be very careful to specify the type of the tensor when assigning it to the PyTorch layer, as follows:
pytorch_layer.weight.data = torch.tensor(tf_weight, dtype=torch.float)
where tf_weight is the transposed weight from the corresponding TensorFlow layer.