I think it trains to a point where the gap between 1st choice and 2nd choice is super large…

Good point, I think this can easily happen.

I have a feeling my problem may have to do with the way I’m feeding the network the input and target during training or something?

Yeah, your issue may already occur during training (as opposed to “inference”). Are you drawing random chunks from your training set? If not, you probably want to do that. It looks like in
    input_vector = torch.tensor([[
        vocab.get(char, len(vocab)) 
        for char in training_data[i+b:i+b+SEQ_LEN]
    ] for b in range(BATCH_SIZE)])

that the network always gets the same inputs during training? For debugging, maybe try to print out the text chunks you are feeding the network to make sure they are not repeating.