Did you initialize the linear’s weights properly? A very simple normal init would be
fc.weight.data.normal_(0, 0.02)
fc.bias.data.zero_()

Also, ideally there should be non-linear activations between them, but i don’t think it matters too much in this case.