How to make a new line when using gpt2 to generate lyrics?
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I use pre-trained gpt2 to generate lyrics (text generation). I can generate a long string of lyrics. While how to break the line, I try to add "\n" into it. But it seems that is not a good idea.

Thanks!
Did you pre-train your model while keeping all the line returns or did you remove them? You can keep them during the training so that the model learns to predict them.

If you remove them during training and wish to apply them later on, I guess you can always just create the long string of lyrics and split them with line returns.<|||||>Actually, I remove them during. Do you think keeping all the line returns during training is a better way? I mean input the whole song (use encode(text)  method) instead of splitting each line into echo tokens -> ids .  

By the way, Can I input POS of text into the model? Is it necessary?<|||||>I think keeping the line returns during your training is a good idea. The model is very likely to learn their position and frequency.

You can input the `position_ids` in your forward, but it is not necessary. If no position information is provided, the model will create it on its own.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
