Your issue is that you are taking all the time steps in the series of hidden states without any aggregation. You should aggregate these outputs in a way that produces a tensor of size torch.size([20, hidden_size]), as you are trying to get a torch.size([20, 30495]) tensor as a result through a Linear layer. There are a couple of “mainstream” ways to achieve that:
>>> lstm_outputs = torch.zeros(20, 28, 512)
>>> lstm_outputs.mean(dim=1).size() # mean of hidden activations through all time steps
torch.Size([20, 512])
>>> lstm_outputs[:, -1].size() # index sequence dimension, get last hidden activation
torch.Size([20, 512])