Currently your model uses the linear layers (fc1, fc2, fc3) without a non-linearity between them, so basically it’s just one single linear transformation. Is this on purpose or did you forget to add the relu or another activation function?
Might be unrelated to this issue, but might be worth a try as a first approach.