@mariosasko, I guess my problem of concatenation between train and validation when using load_dataset() seems to be a normal behavior if no dataset loading script is created in the files on the HF dataset repository (check this post):



Creating and uploading dataset Huggingface Hub vs Dataset creation script


If you don’t upload a dataset script, then the default dataset builder for .txt file is used (and basically it concatenates all the text data together).


However, this information is not given in the HF doc Upload from Python about how to upload a  datasets.DatasetDict  on the Hugging Face Hub in Python.
The HF doc has to be updated or the DatasetDict.push_to_hub() has to be modified?
cc @lhoestq