Transformers, Deep Learning, Machine Learning, Editors Pick, Data Science.
this is just sampling from a multinomial distribution. The predicted token is then appended to the input sequence and fed back into the model, until the desired number of tokens have been generated, or the model produces a stop token; a special token to signify the end of a sequence.