@JStavy
I think you might’ve figured it out by this time. But, just for the sake of it I’m going to respond. ( I’m no expert, still a practitioner).
I believe you’re better of adding activation to your lstm_output, with activation of your choice, say:
lstm_out = nn.ELU(lstm_out[-1, :, :])
You can use this output to feed to your Linear Layer. Hope it helps someone.