The difference is most likely created by the limited precision using FP32.
If you use DoubleTensors, your difference should be smaller:
fc = torch.nn.Linear(256, 128).double()
inp = torch.rand(3, 10, 256).double()