I got this workaround. I don’t know what the problem is and how to correctly resolve it. But just posting it.
Since the weights are already there in the loaded_model, so I copied those weights into another model that I created using same arguments as in the training time:
model = charGen(n_letters, 512, hidden_dim=512).to(device)  # This is exactly how I had my model during training
model.load_state_dict(loaded_model.state_dict())

And it worked now.
predict(model, "i am happy ", device)
#'{i am happy }and away along in the act and finish a straightforward battle on the starboard, and starts to the ope'