Training causal language models from scratch without grouping independent data samples into blocks
Trying to train a causal language model from scratch in Pytorch using run_clm_no_trainer script.
My dataset has independent sentences separated by newlines.
The current script groups these sentence (which are image captions) while training which creates erroneous results when sampling for generation 
Currently I'm trying to avoid this by adding period mark between these sentences before training and at inference splitting generated result at period. This is more of a hack and has obvious shortcomings.
Is there a way to avoid grouping of text while training. I've gone through the script and I may be missing something really obvious.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.