mariosasko:

this week’s release of datasets will add support for directly pushing a Dataset / DatasetDict object to the Hub.


Hi @mariosasko,
I just followed the guide Upload from Python 1 to push to the datasets hub a DatasetDict with train and validation Datasets inside.
raw_datasets = DatasetDict({
    train: Dataset({
        features: ['translation'],
        num_rows: 10000000
    })
    validation: Dataset({
        features: ['translation'],
        num_rows: 1000000
    })
})

from huggingface_hub import notebook_login
notebook_login()

raw_datasets.push_to_hub(repo_id=dataset_name, private=True)

The DatasetDict.push_to_hub() works, and I have train and validation parquet files in my repository (in the folder data) but when I do a load_dataset(), I got a DatasetDict with only a Dataset train that has all the rows (11000000) from the original Dataset train (10000000) and Dataset validation (1000000) that were pushed.
import datasets
from datasets import load_dataset
raw_datasets = load_dataset(dataset_name, use_auth_token=True)
raw_datasets

DatasetDict({
    train: Dataset({
        features: ['translation'],
        num_rows: 11000000
    })
})

Strange. How can I get my original DatasetDict with load_dataset()? Thanks.