The network is deterministic, you can’t expect it to output many different values for a single  input token. At least give it some noise. Designing this will also involve potentially changing how to train your network.