Hi @lhoestq,



 lhoestq:

Make sure to update datasets


You are right.
I was using the HF sagemaker notebooks where the datasets version is inferior to the 1.16 (often 1.13, cc @philschmid) where the DatasetDict.push_to_hub() method was introduced.
With this version (and the latest one 1.16.1), it works (I get the same DatasetDict with train and validation that the one I pushed to the HF datasets hub):
# !pip install datasets --upgrade
import datasets

print(datasets.__version__)
# 1.16.1

from datasets import load_dataset
raw_datasets = load_dataset(dataset_name, use_auth_token=True)
raw_datasets

raw_datasets = DatasetDict({
    train: Dataset({
        features: ['translation'],
        num_rows: 10000000
    })
    validation: Dataset({
        features: ['translation'],
        num_rows: 1000000
    })
})