You can just run the model multiple times to generate a conditional sequence. You for example first give the start token as input_ids, make a prediction, add that token to the input_sequence and forward again. This will give you a pure conditionally generated sequence. It is however not very efficient as it does not make use of ‘cache’, i.e. it will compute the outputs for every time step, also if it has already outputted values for that time step in the previous generation step. You could fix that by adding a notion of cache to the code (returning previous key, values, which you can later add). GPT-2 implements this cache with variables names past and present.