oh this sounds very peculiar indeed! if i understand correctly, you’re using the EncoderDecoderModel (docs 1) to create your Roberta2Roberta model right?
one thing i notice in the code snippet example is that you need to specify the decoder_start_token_id:
generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id)

this wouldn’t explain why you get good generations with the smaller dataset, but is worth double-checking. another idea would be to swap the Roberta decoder for a genuine autoregressive model like GPT-2 - if the effect persists, this would suggest something is strange in your larger corpus
hth!