How to convert datasets.arrow_dataset.Dataset to torch.utils.data.Dataset

Hi,

a HF dataset can be converted to a Torch Dataset with a simple wrapper as follows:
```python
from torch.utils.data import Dataset

class HFDataset(Dataset):
    def __init__(self, dset):
        self.dset = dset

    def __getitem__(self, idx):
        return self.dset[idx]

    def __len__(self):
        return len(self.dset)

train_ds = HFDataset(train_ds)
```
@lhoestq Since the Arrow Dataset already provides `__getitem__` and `__len__`, I think we could use the [virtual subclass](https://docs.python.org/3/library/abc.html#abc.ABCMeta.register) mechanism from the `abc` module to elegantly solve this issue. This mechanism would allow the Arrow Dataset to be used in place of the Torch Dataset because the `isinstance(instance of Arrow Dataset, TorchDataset)` check would return True (DeepSpeed has this check [here](https://github.com/microsoft/DeepSpeed/blob/ab5534fc4c0f8ca21ada321f9730d723aa31288b/deepspeed/runtime/engine.py#L823)).

And it requires a minimal change in the `arrow_dataset.py` file:
```python
if config.TORCH_AVAILABLE:
    from torch.utils.data import Dataset as TorchDataset
    TorchDataset.register(Dataset)
```

Interesting ! Thanks for sharing this @mariosasko . I like the idea
This looks like something we should add IMO

@mariosasko
Thx for your code!
It perfectly works with a small modification for HF NLP dataset:
```
original_ds = nlp.load_dataset('scientific_papers', 'arxiv')
train_ds = HFDataset(train_ds['train']) # needs splitting
```

@lhoestq Sadly, from Python 3.7 onwards `torch.utils.data.Dataset` doesn't support the virtual subclass mechanism due to `typing.Generic` type no longer having `abc.ABCMeta` as its metaclass.

With that in mind, another option is to remove a direct type check (`isinstance(dataset, torch.utils.data.Dataset)`) in `deepspeed.initalize` and to rewrite the checks in a manner similar to `torch.utils.data.DataLoader` ([link](https://github.com/pytorch/pytorch/blob/b80c6f863f2327c712c478f67c248b94d66b65ac/torch/utils/data/dataloader.py#L197-L239)). This is exactly why the `DataLoader` works with arbitrary objects that provide `__getitem__` and `__len__` (and in our case, the `ArrowDataset`). By doing so, their code wouldn't be any stricter in comparison to the `DataLoader`.

So if you agree, I can open an issue in their repo and fix this if they like the idea.

That makes sense ! Feel free to open an issue on their repo and discuss this idea

@y-rokutan Hi, now if you install `deepspeed` from master (this feature will be available in the next official release), the code should work without subclassing. Let us know if you still have any issues.

Worth mentioning that any function that expects a `torch..Dataset` (like `torch..DataLoader`) will fail a mypy-esque typecheck if a `datasets.Dataset` is passed, even though it implements the interface correctly (I think). The virtual subclass idea was a good one- I wonder if there's another workaround given the Generic issue. What we're really talking about is something similar to the structural subtyping semantics that `typing.Protocol` defines. If `torch..DataLoader` accepted anything that supports `__getitem__` and `__len__` methods this would be much easier. Not sure if there's a way to do this without the wrapper from the perspective of `datasets`.

Hi,

I'm trying to pretraine deep-speed model using HF arxiv dataset like:
```
train_ds = nlp.load_dataset('scientific_papers', 'arxiv')
train_ds.set_format(
        type="torch",
        columns=["input_ids", "attention_mask", "global_attention_mask", "labels"],
    )
engine, _, _, _ = deepspeed.initialize(
    args=args,
    model=model,
    model_parameters=[p for p in model.parameters() if p.requires_grad],
    training_data=train_ds)
```
but deepspeed.initialize accepts torch.utils.data.Dataset only. How can I convert HF-style dataset to torch-style dataset?


Replace assertTrue(isinstance with assertIsInstance in tests

Replaces all the occurrences of the `assertTrue(isinstance(` pattern with `assertIsInstance`.


Concat only unique fields in DatasetInfo.from_merge

Hi @mariosasko,
Just came across this PR and I was wondering if we can use
`description = "\n\n".join(OrderedDict.fromkeys([info.description for info in dataset_infos]))`

This will obviate the need for `unique` and is almost as fast as `set`. We could have used `dict` inplace of `OrderedDict` but it's available 3.7+ onwards

Hi,

let's see what @lhoestq thinks. Although my approach adds more code, it's more readable IMO.

Yeah, that's true. Your approach is more readable.

I thought someone from the community with less experience would be interested in fixing this issue, but that wasn't the case.

Fixes #2103


visualization for cc100 is broken

This looks like an issue with the cc100 dataset itself but not sure
Did you try loading cc100 on your machine ?

Hi
loading works fine, but the viewer only is broken
thanks

On Wed, Apr 7, 2021 at 12:17 PM Quentin Lhoest ***@***.***>
wrote:

> This looks like an issue with the cc100 dataset itself but not sure
> Did you try loading cc100 on your machine ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2162#issuecomment-814793809>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMRUO33JSOYGT6RETWLTHQWNLANCNFSM42IUOR6Q>
> .
>

Hi
visualization through dataset viewer for cc100 is broken
https://huggingface.co/datasets/viewer/

thanks a lot


any possibility to download part of large datasets only?

Not yet but it’s on the short/mid-term roadmap (requested by many indeed).

oh, great, really awesome feature to have, thank you very much for the great, fabulous work

We'll work on dataset streaming soon. This should allow you to only load the examples you need ;)

thanks a lot Quentin, this would be really really a great feature to have

On Wed, Apr 7, 2021 at 12:14 PM Quentin Lhoest ***@***.***>
wrote:

> We'll work on dataset streaming soon. This should allow you to only load
> the examples you need ;)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2161#issuecomment-814791922>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMROD62QAKIJMAKWISTTHQWBVANCNFSM42IUI5JQ>
> .
>

Is streaming completed? On the 1.8.0 docs it is mentioned (https://huggingface.co/docs/datasets/dataset_streaming.html), but when following the example I get the following error:

```
>>> dataset2 = load_dataset("amazon_us_reviews", "Pet_Products_v1_00", split='train', streaming=True)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-1eedab26cff1> in <module>()
----> 1 en_dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)

3 frames
/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    339                 if value is not None:
    340                     if not hasattr(builder_config, key):
--> 341                         raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
    342                     setattr(builder_config, key, value)
    343

ValueError: BuilderConfig OscarConfig(name='unshuffled_deduplicated_en', version=1.0.0, data_dir=None, data_files=None, description='Unshuffled and deduplicated, English OSCAR dataset') doesn't have a 'streaming' key.
```

UPDATE: Managed to get streaming working by building from source and installing the additional `datasets[streaming]` package:

```
!pip install git+https://github.com/huggingface/datasets.git
!pip install datasets[streaming]
```

Hi ! Streaming is available on `master` only right now. We'll make a new release 1.9.0 on Monday :)

Hi
Some of the datasets I need like cc100 are very large, and then I wonder if I can download first X samples of the shuffled/unshuffled data without going through first downloading the whole data then sampling? thanks


data_args.preprocessing_num_workers almost freezes

Hi.
I cannot always reproduce this issue, and on later runs I did not see it so far. Sometimes also I set 8 processes but I see less being showed, is this normal, here only 5 are shown for 8 being set, thanks

```
#3:  11%|███████████████▊                                                                                | 172/1583 [00:46<06:21,  3.70ba/s]
#4:   9%|█████████████▏                                                                                | 143/1583 [00:46<07:46,  3.09ba/s]
#7:   6%|█████████                                                                                | 98/1583 [00:45<11:34,  2.14ba/s]
#5:   8%|███████████▍                                                                                | 124/1583 [00:46<09:03,  2.68ba/s]
#6:   7%|██████████▏
```

closing since I cannot reproduce it again, thanks

Hi @lhoestq

I am running this code from huggingface transformers https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py

to speed up tokenization, since I am running on multiple datasets, I am using data_args.preprocessing_num_workers = 4 with opus100 corpus but this moves on till a point and then this freezes almost for sometime during  tokenization steps and then this is back again, overall to me taking more time than normal case, I appreciate your advice on how I can use this option properly to speed up.

thanks


adding ccnet dataset

dataset request

Requesting to add a new dataset

closing since I think this is cc100, just the name has been changed. thanks

## Adding a Dataset
- **Name:**  ccnet

- **Description:**
Common Crawl

- **Paper:**
https://arxiv.org/abs/1911.00359

- **Data:**
https://github.com/facebookresearch/cc_net

- **Motivation:**
this is one of the most comprehensive clean monolingual datasets across a variety of languages. Quite important for cross-lingual reseach

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

thanks


viewer "fake_news_english" error

Thanks for reporting !
The viewer doesn't have all the dependencies of the datasets. We may add openpyxl to be able to show this dataset properly

When I visit the [Huggingface - viewer](https://huggingface.co/datasets/viewer/) web site, under the dataset "fake_news_english" I've got this error:

> ImportError: To be able to use this dataset, you need to install the following dependencies['openpyxl'] using 'pip install # noqa: requires this pandas optional dependency for reading xlsx files' for instance'

as well as the error Traceback.


updated user permissions based on umask

Updated user permissions based on running user's umask (#2065). Let me know if `0o666` is looking good or should I change it to `~umask` only (to give execute permissions as well)


User permissions

Updated user permissions based on running user's umask. Let me know if `0o666` is looking good or should I change it to `~umask` only (to give execute permissions as well)


Add table classes to the documentation

Just note that docstrings injected from PyArrow do not follow the same convention for formatting types in `Args` or `Returns` as we do... Not a big problem, anyway! 😄

Following #2025 , I added the table classes to the documentation

cc @albertvillanova


Adding the NorNE dataset for Norwegian POS and NER

NorNE is a manually annotated corpus of named entities which extends the annotation of the existing Norwegian Dependency Treebank. Comprising both of the official standards of written Norwegian (Bokmål and Nynorsk), the corpus contains around 600,000 tokens and annotates a rich set of entity types including persons, organizations, locations, geo-political entities, products, and events, in addition to a class corresponding to nominals derived from names.

See #1720.


load_dataset ignoring features

bug

Something isn't working

Hi ! Thanks for reporting. I opened a PR to fix this issue: #2201

Nice question which helped me a lot! I have wasted a lot of time to the `DatasetDict` creation from a csv file. Hope the document of this module add some simple examples.

Hi :) We're indeed working on tutorials that we will add to the docs !

First of all, I'm sorry if it is a repeated issue or the changes are already in master, I searched and I didn't find anything.

I'm using datasets 1.5.0

![image](https://user-images.githubusercontent.com/37592763/113114369-8f376580-920b-11eb-900d-94365b59f04b.png)

As you can see, when I load the dataset, the ClassLabels are ignored, I have to cast the dataset in order to make it work.

Code to reproduce:

```python
import datasets
data_location = "/data/prueba_multiclase"
features = datasets.Features(
        {"texto": datasets.Value("string"), "label": datasets.features.ClassLabel(names=["false", "true"])}
    )
dataset = datasets.load_dataset(
        "csv", data_files=data_location, delimiter="\t", features=features
    )
```

Dataset I used:

[prueba_multiclase.zip](https://github.com/huggingface/datasets/files/6235022/prueba_multiclase.zip) (it has to be unzipped)

Thank you! ❤


Update README.md

Updated some descriptions of Wino_Bias dataset.


Add support for axis in concatenate datasets

enhancement

New feature or request

1.6

Next minor release

@lhoestq I am going to implement the consolidation step you mentioned in #1870.

@lhoestq I was thinking that the order of the TableBlocks is not relevant, isn't it?

I mean, in order to consolidate _consecutive_ in-memory table blocks, in this case:
```
blocks = [in_memory_1, memory_mapped, in_memory_2]
```
I could reorder the list:
```
blocks = [in_memory_1, in_memory_2, memory_mapped]
```
so that the first 2 can be consolidated into a single one:
```
blocks = [in_memory_3, memory_mapped]
```

> I think the order is important, users won't expect the dataset to be "shuffled" when they add a new item

OK, therefore I leave `_consolidate_blocks` as it is, which currently keeps the order of the blocks (no shuffling).

Thank you guys for implementing this. Minor thing I noticed in the [documentation](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.concatenate_datasets): it says "Converts a list of Dataset with **the same schema** into a single Dataset". With the addition of the axis parameter, perhaps this should be reworded, no?

Add support for `axis` (0 or 1) in `concatenate_datasets`.

Close #853.


Allow pickling of big in-memory tables

This should fix issue #2134

Pickling is limited to <4GiB objects, it's not possible to pickle a big arrow table (for multiprocessing for example).
For big tables, we have to write them on disk and only pickle the path to the table.


Telugu subset missing for xtreme tatoeba dataset

Good catch ! Thanks for reporting

I just opened #2180 to fix this

from nlp import load_dataset
train_dataset = load_dataset('xtreme', 'tatoeba.tel')['validation']
ValueError: BuilderConfig tatoeba.tel not found.

but language tel is actually included in xtreme:
https://github.com/google-research/xtreme/blob/master/utils_preprocess.py
def tatoeba_preprocess(args):
  lang3_dict = {
    'afr':'af', 'ara':'ar', 'bul':'bg', 'ben':'bn',
    'deu':'de', 'ell':'el', 'spa':'es', 'est':'et',
    'eus':'eu', 'pes':'fa', 'fin':'fi', 'fra':'fr',
    'heb':'he', 'hin':'hi', 'hun':'hu', 'ind':'id',
    'ita':'it', 'jpn':'ja', 'jav':'jv', 'kat':'ka',
    'kaz':'kk', 'kor':'ko', 'mal':'ml', 'mar':'mr',
    'nld':'nl', 'por':'pt', 'rus':'ru', 'swh':'sw',
    'tam':'ta', **_'tel':'te'_**, 'tha':'th', 'tgl':'tl', <----here
    'tur':'tr', 'urd':'ur', 'vie':'vi', 'cmn':'zh',
    'eng':'en',
  }


Add configurable options to `seqeval` metric

Hi @marrodion.

Thanks for pointing this out. It would be great to incorporate this metric-specific enhancement.

Another possibility would be to require the user to input the scheme as a string `mode="strict", scheme="IOB2"` and then dynamically import the corresponding module using Python `importlib`:
```python
if scheme:
    scheme = importlib.import_module(f"seqeval.scheme.{scheme}")
```

Feel free to create a Pull Request to make this contribution.

Right now `load_metric("seqeval")` only works in the default mode of evaluation (equivalent to conll evaluation).

However, seqeval library [supports](https://github.com/chakki-works/seqeval#support-features) different evaluation schemes (IOB1, IOB2, etc.), which can be plugged in just by supporting additional kwargs in `Seqeval._compute`
https://github.com/huggingface/datasets/blob/85cf7ff920c90ca2e12bedca12b36d2a043c3da2/metrics/seqeval/seqeval.py#L109

Things that would be relevant are, for example, supporting `mode="strict", scheme=IOB2` to count only full entity match as a true positive and omit partial matches.

The only problem I see is that the spirit of `metrics` seems to not require additional imports from user. `seqeval` only supports schemes as objects, without any string aliases.

It can be solved naively with mapping like `{"IOB2": seqeval.scheme.IOB2}`. Or just left as is and require user to explicitly import scheme from `seqeval` if he wants to configure it past the default implementation.

If that makes sense, I am happy to implement the change.


Render docstring return type as inline

documentation

Improvements or additions to documentation

This documentation setting will avoid having the return type in a separate line under `Return type`.

See e.g. current docs for `Dataset.to_csv`.


Dataset file size on disk is very large with 3D Array

Hi ! In the arrow file we store all the integers as uint8.
So your arrow file should weigh around `height x width x n_channels x n_images` bytes.

What feature type do your TFDS dataset have ?

If it uses a `tfds.features.Image` type, then what is stored is the encoded data (as png or jpg for example). Since these encodings are made for compression, the resulting tfrecord is smaller that the arrow file.

We are working on adding a similar feature in `datasets`: the ability to store the encoded data instead of the raw integers for images, but also for audio data. This way, arrow files will have similar sizes as tfrecords for images.

Thanks for the prompt response. You're right about the encoding, I have the `tfds.features.Image` feature type you mentioned.
However, as described in the `dataset_info.json`, my dataset is made of 1479 (224x224x3) images. 1479 x 224 x 224 x 3 = 222630912 bytes which is far from the actual size 520803408 bytes.

Anyway I look forward to the Image feature type in `datasets`.

@lhoestq I changed the data structure so I have a  2D Array feature type instead of a 3D Array by grouping the two last dimensions ( a 224x672 2D Array instead of a 224x224x3 3D Array). The file size is now  223973964 bytes, nearly half the previous size! Which is around of what I would expect.
I found similar behavior in existing `datasets` collection, when comparing black and white vs color image, for example MNIST vs CIFAR.

Interesting !
This may be because of the offsets that are stored with the array data.

Currently the offsets are stored even if the `shape` of the arrays is fixed. This was needed because of some issues with pyarrow a few months ago. I think these issues have been addressed now, so we can probably try to remove them to make the file lighter.

Ideally in your case the floats data should be 220 MB for both Array2D and Array3D

Yeah for sure, can you be a bit more specific about where the offset is stored in the code base ? And any reference to pyarrow issues if you have some. I would be very interested in contributing to `datasets` by trying to fix this issue.

Pyarrow has two types of lists: variable length lists and fixed size lists.
Currently we store the ArrayXD data as variable length lists. They take more disk space because they must store both actual data and offsets.
In the `datasets` code this is done here:

https://github.com/huggingface/nlp/blob/dbac87c8a083f806467f5afc4ec9b401a7e4c15c/src/datasets/features.py#L346-L352

To use a fixed length list, one should use the `list_size` argument of `pyarrow.list_()`.
I believe this would work directly modulo some changes in the numpy conversion here:

https://github.com/huggingface/nlp/blob/dbac87c8a083f806467f5afc4ec9b401a7e4c15c/src/datasets/features.py#L381-L395

Hi,

I have created my own dataset using the provided dataset loading script. It is an image dataset where images are stored as 3D Array with dtype=uint8.

The actual size on disk is surprisingly large. It takes 520 MB. Here is some info from `dataset_info.json`.

`{
    "description": "",
    "citation": "",
    "homepage": "",
    "license": "",
    "features": {
        "image": {
            "shape": [224, 224, 3],
            "dtype": "uint8",
            "id": null,
            "_type": "Array3D",
        }
    },
    "post_processed": null,
    "supervised_keys": null,
    "builder_name": "shot_type_image_dataset",
    "config_name": "default",
    "version": {
        "version_str": "0.0.0",
        "description": null,
        "major": 0,
        "minor": 0,
        "patch": 0,
    },
    "splits": {
        "train": {
            "name": "train",
            "num_bytes": 520803408,
            "num_examples": 1479,
            "dataset_name": "shot_type_image_dataset",
        }
    },
    "download_checksums": {
        "": {
            "num_bytes": 16940447118,
            "checksum": "5854035705efe08b0ed8f3cf3da7b4d29cba9055c2d2d702c79785350d72ee03",
        }
    },
    "download_size": 16940447118,
    "post_processing_size": null,
    "dataset_size": 520803408,
    "size_in_bytes": 17461250526,
}`

I have created the same dataset with tensorflow_dataset and it takes only 125MB on disk.

I am wondering, is it normal behavior ? I understand `Datasets` uses Arrow for serialization wheres tf uses TF Records.

This might be a problem for large dataset.

Thanks for your help.


Implement Dataset add_column

enhancement

New feature or request

1.7

Next minor release

#2274 has been merged. You can now merge master into this branch and use `assert_arrow_metadata_are_synced_with_dataset_features(dset)` to make sure that the metadata are good :)

Implement `Dataset.add_column`.

Close #1954.


Loading wikipedia 20200501.en throws pyarrow related error

That's how I loaded the dataset
```python
from datasets import load_dataset
ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache')
```

Hi ! It looks like the arrow file in the folder
`/usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931` is corrupted.

Can you take a look and check that it's 18.3GB ?

If not, then maybe you need to redownload it:
```python
from datasets import load_dataset
ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache', download_mode="force_redownload")
```

> Hi ! It looks like the arrow file in the folder
> `/usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931` is corrupted.
>
> Can you take a look and check that it's 18.3GB ?
>
> If not, then maybe you need to redownload it:
>
> ```python
> from datasets import load_dataset
> ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache', download_mode="force_redownload")
> ```

Hi Ihoestq, thanks for the reply! Actually i think my issue is i couldn't download the dataset beyond 10.7G. It feels like the whole dataset is split into different volumes and after the first one was downloaded it crashed before proceeding to the next one. I did try 'force_redownload' mode but still got the same issue.

I just tried on my side and got no issues.
When downloading the dataset again, did it crash at 10.7GB as well ?

> I just tried on my side and got no issues.
> When downloading the dataset again, did it crash at 10.7GB as well ?

Yes i have tried it multiple times on different machines. I am wondering if you could share the screenshot of your dependency versions and i will try to make them the same as yours?

I tried using `datasets` from `master` on macos with python 3.7.2
I also have `requests==2.23.0` and `tqdm==4.45.0`.

**Problem description**
I am getting the following error when trying to load wikipedia/20200501.en dataset.

**Error log**
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, post-processed: Unknown size, total: 34.06 GiB) to /usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931...
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.6k/14.6k [00:00<00:00, 5.41MB/s]
Downloading:  59%|███████████████████████████████████████████████████████████████████████████████████████▊                                                              | 10.7G/18.3G [11:30<08:08, 15.5MB/s]
Dataset wikipedia downloaded and prepared to /usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931. Subsequent calls will reuse this data.
Traceback (most recent call last):
  File "load_wiki.py", line 2, in <module>
    ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache')
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 751, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications, in_memory=keep_in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 746, in as_dataset
    map_tuple=True,
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 204, in map_nested
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 204, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 142, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 763, in _build_single_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 835, in _as_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 215, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 236, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 171, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 302, in _get_dataset_from_filename
    pa_table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 324, in read_table
    pa_table = f.read_all()
  File "pyarrow/ipc.pxi", line 544, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Expected to be able to read 9176784 bytes for message body, got 4918712

**Detailed version info**
datasets==1.5.0
  - dataclasses [required: Any, installed: 0.8]
  - dill [required: Any, installed: 0.3.3]
  - fsspec [required: Any, installed: 0.8.7]
    - importlib-metadata [required: Any, installed: 1.7.0]
      - zipp [required: >=0.5, installed: 3.1.0]
  - huggingface-hub [required: <0.1.0, installed: 0.0.7]
    - filelock [required: Any, installed: 3.0.12]
    - importlib-metadata [required: Any, installed: 1.7.0]
      - zipp [required: >=0.5, installed: 3.1.0]
    - requests [required: Any, installed: 2.24.0]
      - certifi [required: >=2017.4.17, installed: 2020.6.20]
      - chardet [required: >=3.0.2,<4, installed: 3.0.4]
      - idna [required: >=2.5,<3, installed: 2.6]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
    - tqdm [required: Any, installed: 4.49.0]
  - importlib-metadata [required: Any, installed: 1.7.0]
    - zipp [required: >=0.5, installed: 3.1.0]
  - multiprocess [required: Any, installed: 0.70.11.1]
    - dill [required: >=0.3.3, installed: 0.3.3]
  - numpy [required: >=1.17, installed: 1.17.0]
  - pandas [required: Any, installed: 1.1.5]
    - numpy [required: >=1.15.4, installed: 1.17.0]
    - python-dateutil [required: >=2.7.3, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.2, installed: 2020.1]
  - pyarrow [required: >=0.17.1, installed: 3.0.0]
    - numpy [required: >=1.16.6, installed: 1.17.0]
  - requests [required: >=2.19.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.6]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - tqdm [required: >=4.27,<4.50.0, installed: 4.49.0]
  - xxhash [required: Any, installed: 2.0.0]


task casting via load_dataset

wip
not satisfied with the API, it means as a dataset implementer I need to write a function with boilerplate and write classes for each `<dataset><task>` "facet".


Gem V1.1

This branch updates the GEM benchmark to its 1.1 version which includes:
- challenge sets for most tasks
- detokenized TurkCorpus to match the rest of the text simplification subtasks
- fixed inputs for TurkCorpus and ASSET test sets
- 18 languages in WikiLingua

cc @sebastianGehrmann


added spans field for the wikiann datasets

Hi @lhoestq
Thanks a lot for taking time checking it. I update "dataset_infos.json", I added description to the function of _generate_samples  in wikiann.py but I was not sure about the format to write in README. thanks.

Thanks !

For the fields description in the dataset card, something like this does the job:
```
- `tokens`: a `list` of `string` features.
- `langs`: a `list` of `string` features that correspond to the language of each token.
- `ner_tags`: a `list` of classification labels, with possible values including `O` (0), `B-PER` (1), `I-PER` (2), `B-ORG` (3), `I-ORG` (4), `B-LOC` (5), `I-LOC` (6).
- `spans`: a `list` of `string` features, that is the list of named entities in the input text formatted as ``<TAG>: <mention>``
```

Also for information, I think the trailer of rick and morty season 5 is out now :)

Hi @lhoestq
thank you! This is updated now, please feel free to let me know if I need to modify something :)  thanks

Hi @lhoestq
I tried to add spans to the wikiann datasets.
Thanks a lot for kindly having a look.
This addresses https://github.com/huggingface/datasets/issues/2130.
Best regards
Rabeeh


add banking77 dataset

Intent classification/detection dataset from banking category with 77 unique intents.


TypeError when using save_to_disk in a dataset loaded with ReadInstruction split

Hi !
I think this has been fixed recently on `master`.
Can you try again by installing `datasets` from `master` ?
```
pip install git+https://github.com/huggingface/datasets.git
```

Hi!

Using that version of the code solves the issue. Thanks!

Hi,

Loading a dataset with `load_dataset` using a split defined via `ReadInstruction` and then saving it to disk results in the following error: `TypeError: Object of type ReadInstruction is not JSON serializable`.

Here is the minimal reproducible example:

```python
from datasets import load_dataset
from datasets import ReadInstruction

data_1 = load_dataset(
    "wikiann",
    "en",
    split="validation",
)

data_1.save_to_disk("temporary_path_1")

print("Save with regular split works.")

data_2 = load_dataset(
    "wikiann",
    "en",
    split=ReadInstruction("validation", to=50, unit="%"),
)

data_2.save_to_disk("temporary_path_2")
```

and the corresponding output:

```
Reusing dataset wikiann (/xxxxx/.cache/huggingface/datasets/wikiann/en/1.1.0/0b11a6fb31eea02f38ca17610657bfba3206100685283014daceb8da291c3be9)
Save with regular split works.
Reusing dataset wikiann (/xxxxx/.cache/huggingface/datasets/wikiann/en/1.1.0/0b11a6fb31eea02f38ca17610657bfba3206100685283014daceb8da291c3be9)
Traceback (most recent call last):
  File "bug.py", line 20, in <module>
    data_2.save_to_disk("temporary_path_2")
  File "/xxxxx/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 645, in save_to_disk
    json.dump(state, state_file, indent=2, sort_keys=True)
  File "/usr/lib/python3.7/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/lib/python3.7/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.7/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type ReadInstruction is not JSON serializable
```

Let me know if there is some misuse from my end.

Thanks in advance.


Add CER metric

Add Character Error Rate (CER) metric that is used in evaluation in ASR. I also have written unittests (hopefully thorough enough) but I'm not sure how to integrate them into the existed codebase.

```python
from cer import CER

cer = CER()

class TestCER(unittest.TestCase):
    def test_cer_case_senstive(self):
        refs = ['White House']
        preds = ['white house']
        # S = 2, D = 0, I = 0, N = 11, CER = 2 / 11
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.1818181818) < 1e-6)

    def test_cer_whitespace(self):
        refs = ['were wolf']
        preds = ['werewolf']
        # S = 0, D = 0, I = 1, N = 9, CER = 1 / 9
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.1111111) < 1e-6)

        refs = ['werewolf']
        preds = ['weae     wolf']
        # S = 1, D = 1, I = 0, N = 8, CER = 0.25
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.25) < 1e-6)

        # consecutive whitespaces case 1
        refs = ['were wolf']
        preds = ['were               wolf']
        # S = 0, D = 0, I = 0, N = 9, CER = 0
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.0) < 1e-6)

        # consecutive whitespaces case 2
        refs = ['were   wolf']
        preds = ['were               wolf']
        # S = 0, D = 0, I = 0, N = 9, CER = 0
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.0) < 1e-6)

    def test_cer_sub(self):
        refs = ['werewolf']
        preds = ['weaewolf']
        # S = 1, D = 0, I = 0, N = 8, CER = 0.125
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.125) < 1e-6)

    def test_cer_del(self):
        refs = ['werewolf']
        preds = ['wereawolf']
        # S = 0, D = 1, I = 0, N = 8, CER = 0.125
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.125) < 1e-6)

    def test_cer_insert(self):
        refs = ['werewolf']
        preds = ['wereolf']
        # S = 0, D = 0, I = 1, N = 8, CER = 0.125
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.125) < 1e-6)

    def test_cer_equal(self):
        refs = ['werewolf']
        char_error_rate = cer.compute(predictions=refs, references=refs)
        self.assertEqual(char_error_rate, 0.0)

    def test_cer_list_of_seqs(self):
        refs = ['werewolf', 'I am your father']
        char_error_rate = cer.compute(predictions=refs, references=refs)
        self.assertEqual(char_error_rate, 0.0)

        refs = ['werewolf', 'I am your father', 'doge']
        preds = ['werxwolf', 'I       am your father', 'doge']
        # S = 1, D = 0, I = 0, N = 28, CER = 1 / 28
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.03571428) < 1e-6)

    def test_cer_unicode(self):
        ref = [u'我能吞下玻璃而不伤身体']
        pred = [u' 能吞虾玻璃而 不霜身体啦']
        # S = 3, D = 2, I = 0, N = 11
        # CER = 5 / 11
        char_error_rate = cer.compute(predictions=pred, references=ref)
        self.assertTrue(abs(char_error_rate - 0.4545454545) < 1e-6)

        ref = [u'我能吞', u'下玻璃而不伤身体']
        pred = [u'我    能 吞 下 玻 璃', u'而不伤身体']
        # S = 0, D = 5, I = 0, N = 11
        # CER = 5 / 11
        char_error_rate = cer.compute(predictions=pred, references=ref)
        self.assertTrue(abs(char_error_rate - 0.454545454545) < 1e-6)

        ref = [u'我能吞下玻璃而不伤身体']
        char_error_rate = cer.compute(predictions=ref, references=ref)
        self.assertFalse(char_error_rate, 0.0)

    def test_cer_empty(self):
        ref = ''
        pred = 'Hypothesis'
        with self.assertRaises(ValueError):
            char_error_rate = cer.compute(predictions=pred, references=ref)

if __name__ == '__main__':
    unittest.main()
```


Fix missing infos from concurrent dataset loading

This should fix issue #2131

When calling `load_dataset` at the same time from 2 workers, one of the worker could have missing split infos when reloading the dataset from the cache.


fix dialogue action slot name and value

fix #2128


en language data from MLQA dataset is missing

Hi ! Indeed only the languages of the `translate-train` data are included...
I can't find a link to download the english train set on https://github.com/facebookresearch/MLQA though, do you know where we can download it ?

Hi @lhoestq
thank you very much for coming back to me, now I see, you are right, in the link you sent I see split of  {split}-context-{context_language}-question-{question_language}.json with context_language=question_language=en, TFDS most probably has extracted english ones from these files as en language files, but translate-train/test do not have en indeed. thanks a lot for the great explanations

I close the ticket, since I do not see any en existing, they have trained on "SQuAD V1.1" instead. Thanks.

Hi
I need mlqa-translate-train.en dataset, but it is missing from the MLQA dataset. could you have a look please? @lhoestq  thank you for your help to fix this issue.


Saving large in-memory datasets with save_to_disk crashes because of pickling

bug

Something isn't working

Hi !
Indeed `save_to_disk` doesn't call pickle anymore. Though the `OverflowError` can still appear for in-memory datasets bigger than 4GB. This happens when doing this for example:
```python
import pyarrow as pa
import pickle

arr = pa.array([0] * ((4 * 8 << 30) // 64))
table = pa.Table.from_arrays([a], names=["foo"])
pickle.dumps(table)  # fails with an OverflowError
pickle.dumps(table, 4)  # works !
```
We'll do the change to use `protocol=4`.

Moreover I've also seen other users complain about this error
```
struct.error: 'I' format requires 0 <= number <= 4294967295
```

It looks like something related to the 4GB limit as well but I'm not able to reproduce on my side.
Do you think you can provide a script that reproduces the issue ?
How big is your dataset ? (number of bytes, number of rows)

Hi!
So I've managed to created a minimum working (well technically crashing) example for the multiprocessing case, I create a huge list of zeros, like in your example, and then I try to .map(None, num_proc=2) over it, which then crashes, here's the code:

```python
from datasets import  Dataset

if __name__ == '__main__':
    ton_of_zeroes = [0] * ((12 * 8 << 30) // 64)
    large_dataset = Dataset.from_dict({'col': ton_of_zeroes})
    print("Start")
    large_dataset.map(function=None, num_proc=2)
    print("Done - should not print")
```

The amount of zeros could probably be reduced, I haven't tried to minimize it to find the breaking point, I just increased it from your code (which by quick glance I assumed tried to allocate over 4 GiB)

Running this results in the following traceback:

```
Parameter 'indices'=[        0         1         2 ... 805306365 805306366 805306367] of the transform datasets.arrow_dataset.Dataset.select couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
Traceback (most recent call last):
  File "./crash_multiproc_pickle.py", line 7, in <module>
    large_dataset.map(function=None, num_proc=2)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in map
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
    raise self._value
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 431, in _handle_tasks
    put(task)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 454, in dump
    StockPickler.dump(self, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 732, in save_bytes
    self._write_large_bytes(BINBYTES + pack("<I", n), obj)
struct.error: 'I' format requires 0 <= number <= 4294967295
```

My datasets usually have hundreds of thousands to low millions of rows, with each row containing a list of 10 strings and list of vectors of different length (the strings tokenized), which in the worst case have 10\*512\*8 = 40960 bytes (but usually it is much smaller, as the vectors tend to be shorter. I need these groups of text lines to create training data for the Inverse Cloze Task.

Anyway I don't think my particular dataset is relevant, as the tiny script I created also manages to crash.
But I think the issue is the same as the save_to_disk, from the traceback it seems that in multiprocessing, it tries to use dill to return the result of the map workers, which tries to pickle the data and can't do it, probably because it's again using the older pickle protocol. That's my guess anyway.

I just merged a fix #2150 that allows to pickle tables bigger than 4GiB
Feel free to try it on the `master` branch !

awesome! I started getting this error as well when I tried to tokenize with a longer sequence length

@prokopCerny does this fix work for you? I found that with the latest master, my container with 500GB RAM starts crashing when I try to map a large dataset using `num_proc`.

@lhoestq would it be possible to implement some logic to keep the individual cache files small (say below 100mb)? I find this helps with loading large datasets, but the "hack" I was using (increasing `num_proc` to a large number) doesn't work anymore with the latest master; my container crashes even with `num_proc=200` now

Closing since the original issue was fixed in #2150
Feel free to reopen if you are still experiencing it.
For the other problems, please open separate issues

Using Datasets 1.5.0 on Python 3.7.
Recently I've been working on medium to large size datasets (pretokenized raw text sizes from few gigabytes to low tens of gigabytes), and have found out that several preprocessing steps are massively faster when done in memory, and I have the ability to requisition a lot of RAM, so I decided to do these steps completely out of the datasets library.

 So my workflow is to do several .map() on datasets object, then for the operation which is faster in memory to extract the necessary columns from the dataset and then drop it whole, do the transformation in memory, and then create a fresh Dataset object using .from_dict() or other method.

When I then try to call save_to_disk(path) on the dataset, it crashes because of pickling, which appears to be because of using old pickle protocol which doesn't support large files (over 4 GiB).
```
Traceback (most recent call last):
  File "./tokenize_and_chunkify_in_memory.py", line 80, in <module>
    main()
  File "./tokenize_and_chunkify_in_memory.py", line 75, in main
    tokenize_and_chunkify(config)
  File "./tokenize_and_chunkify_in_memory.py", line 60, in tokenize_and_chunkify
    contexts_dataset.save_to_disk(chunked_path)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 457, in save_to_disk
    self = pickle.loads(pickle.dumps(self))
OverflowError: cannot serialize a bytes object larger than 4 GiB
```
From what I've seen this issue may be possibly fixed, as the line `self = pickle.loads(pickle.dumps(self))` does not appear to be present in the current state of the repository.

To save these datasets to disk, I've resorted to calling .map() over them with `function=None` and specifying the .arrow cache file, and then creating a new dataset using the .from_file() method, which I can then safely save to disk.

Additional issue when working with these large in-memory datasets is when using multiprocessing, is again to do with pickling. I've tried to speed up the mapping with function=None by specifying num_proc to the available cpu count, and I again get issues with transferring the dataset, with the following traceback. I am not sure if I should open a separate issue for that.
```
Traceback (most recent call last):
  File "./tokenize_and_chunkify_in_memory.py", line 94, in <module>
    main()
  File "./tokenize_and_chunkify_in_memory.py", line 89, in main
    tokenize_and_chunkify(config)
  File "./tokenize_and_chunkify_in_memory.py", line 67, in tokenize_and_chunkify
    contexts_dataset.map(function=None, cache_file_name=str(output_dir_path / "tmp.arrow"), writer_batch_size=50000, num_proc=config.threads)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in map
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
    raise self._value
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 431, in _handle_tasks
    put(task)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 454, in dump
    StockPickler.dump(self, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 732, in save_bytes
    self._write_large_bytes(BINBYTES + pack("<I", n), obj)
struct.error: 'I' format requires 0 <= number <= 4294967295Traceback (most recent call last):
  File "./tokenize_and_chunkify_in_memory.py", line 94, in <module>
    main()
  File "./tokenize_and_chunkify_in_memory.py", line 89, in main
    tokenize_and_chunkify(config)
  File "./tokenize_and_chunkify_in_memory.py", line 67, in tokenize_and_chunkify
    contexts_dataset.map(function=None, cache_file_name=str(output_dir_path / "tmp.arrow"), writer_batch_size=50000, num_proc=config.threads)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in map
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
    raise self._value
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 431, in _handle_tasks
    put(task)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 454, in dump
    StockPickler.dump(self, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 732, in save_bytes
    self._write_large_bytes(BINBYTES + pack("<I", n), obj)
struct.error: 'I' format requires 0 <= number <= 4294967295
```


bug in mlqa dataset

Hi @dorost1234.

In Python 3, strings are sequences of Unicode _code points_. Unicode is a specification that maps all characters (and emoji symbols) with its unique representation in terms of code points. That is what you see: Unicode code points (represented by a \u escaped sequence of 16-bit hex values).

Characters are usually represented (on screen and papers) with a graphical element called _glyph_. That is what you would like to see: glyphs. But Python does not care about glyphs: that is the job of the GUI or the terminal; glyphs are what you get with the `print` function (if your terminal is properly configured to display those glyphs).

You have more detailed information about Unicode in the Python documentation: https://docs.python.org/3/howto/unicode.html

thank you so much for the insightful comments.

Hi
Looking into MLQA dataset for langauge "ar":

```
 "question": [
    "\u0645\u062a\u0649 \u0628\u062f\u0627\u062a \u0627\u0644\u0645\u062c\u0644\u0629 \u0627\u0644\u0645\u062f\u0631\u0633\u064a\u0629 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645 \u0628\u0627\u0644\u0646\u0634\u0631?",
    "\u0643\u0645 \u0645\u0631\u0629 \u064a\u062a\u0645 \u0646\u0634\u0631\u0647\u0627 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?",
    "\u0645\u0627 \u0647\u064a \u0627\u0644\u0648\u0631\u0642\u0629 \u0627\u0644\u064a\u0648\u0645\u064a\u0629 \u0644\u0644\u0637\u0644\u0627\u0628 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?",
    "\u0643\u0645 \u0639\u062f\u062f \u0627\u0644\u0627\u0648\u0631\u0627\u0642 \u0627\u0644\u0627\u062e\u0628\u0627\u0631\u064a\u0629 \u0644\u0644\u0637\u0644\u0627\u0628 \u0627\u0644\u062a\u064a \u0648\u062c\u062f\u062a \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?",
    "\u0641\u064a \u0627\u064a \u0633\u0646\u0629 \u0628\u062f\u0627\u062a \u0648\u0631\u0642\u0629 \u0627\u0644\u0637\u0627\u0644\u0628 \u0627\u0644\u062d\u0633 \u0627\u0644\u0633\u0644\u064a\u0645 \u0628\u0627\u0644\u0646\u0634\u0631 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?"
  ]
```

the questions are in the wrong format, and not readable, could you please have a look? thanks @lhoestq


TydiQA dataset is mixed and is not split per language

You can filter the languages this way:
```python
tydiqa_en = tydiqa_dataset.filter(lambda x: x["language"] == "english")
```

Otherwise maybe we can have one configuration per language ?
What do you think of this for example ?

```python
load_dataset("tydiqa", "primary_task.en")
```

Hi
thank you very much for the great response, this will be really wonderful
to have one configuration per language, as one need the dataset in majority
of case per language for cross-lingual evaluations.
This becomes also then more close to TFDS format, which is separated per
language https://www.tensorflow.org/datasets/catalog/tydi_qa which will be
really awesome to have.
thanks

On Mon, Mar 29, 2021 at 6:17 PM Quentin Lhoest ***@***.***>
wrote:

> You can filter the languages this way:
>
> tydiqa_en = tydiqa_dataset.filter(lambda x: x["language"] == "english")
>
> Otherwise maybe we can have one configuration per language ?
> What do you think of this for example ?
>
> load_dataset("tydiqa", "primary_task.en")
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2132#issuecomment-809516799>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMXPW2PWSQ2RHG73O7TTGCY4LANCNFSM4Z7ER7IA>
> .
>

@lhoestq  I greatly appreciate any updates on this. thanks a lot

Hi @lhoestq
Currently TydiQA is mixed and user can only access the whole training set of all languages:
https://www.tensorflow.org/datasets/catalog/tydi_qa

for using this dataset, one need to train/evaluate in each separate language, and having them mixed, makes it hard to use this dataset. This is much convenient for user to have  them split and I appreciate your help on this.

Meanwhile, till hopefully this is split per language, I greatly appreciate telling me how I can preprocess and get data per language. thanks a lot


When training with Multi-Node Multi-GPU the worker 2 has TypeError: 'NoneType' object

Something isn't working

Hi ! Thanks for reporting
I was able to reproduce this issue. This was caused by missing split infos if a worker reloads the cache of the other worker.

I just opened https://github.com/huggingface/datasets/pull/2137 to fix this issue

The PR got merged :)
Feel free to try it out on the `master` branch

Sorry for the late reply.
Now everything just works well XD

version: 1.5.0
met a very strange error, I am training large scale language model, and need train on 2 machines(workers).
And sometimes I will get this error `TypeError: 'NoneType' object is not iterable`
This is traceback
```

71 |   | Traceback (most recent call last):
-- | -- | --
72 |   | File "run_gpt.py", line 316, in <module>
73 |   | main()
74 |   | File "run_gpt.py", line 222, in main
75 |   | delimiter="\t", column_names=["input_ids", "attention_mask", "chinese_ref"])
76 |   | File "/data/miniconda3/lib/python3.7/site-packages/datasets/load.py", line 747, in load_dataset
77 |   | use_auth_token=use_auth_token,
78 |   | File "/data/miniconda3/lib/python3.7/site-packages/datasets/builder.py", line 513, in download_and_prepare
79 |   | self.download_post_processing_resources(dl_manager)
80 |   | File "/data/miniconda3/lib/python3.7/site-packages/datasets/builder.py", line 673, in download_post_processing_resources
81 |   | for split in self.info.splits:
82 |   | TypeError: 'NoneType' object is not iterable
83 |   | WARNING:datasets.builder:Reusing dataset csv (/usr/local/app/.cache/huggingface/datasets/csv/default-1c257ebd48e225e7/0.0.0/2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2)
84 |   | Traceback (most recent call last):
85 |   | File "/data/miniconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
86 |   | "__main__", mod_spec)
87 |   | File "/data/miniconda3/lib/python3.7/runpy.py", line 85, in _run_code
88 |   | exec(code, run_globals)
89 |   | File "/data/miniconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 340, in <module>
90 |   | main()
91 |   | File "/data/miniconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 326, in main
92 |   | sigkill_handler(signal.SIGTERM, None)  # not coming back
93 |   | File "/data/miniconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
94 |   | raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)

```
On worker 1 it loads the dataset well, however on worker 2 will get this error.
And I will meet this error from time to time, sometimes it just goes well.


wikiann dataset is missing columns

good first issue

Good for newcomers

Here please find TFDS format of this dataset: https://www.tensorflow.org/datasets/catalog/wikiann
where there is a span column, this is really necessary to be able to use the data, and I appreciate your help @lhoestq

Hi !
Apparently you can get the spans from the NER tags using `tags_to_spans` defined here:

https://github.com/tensorflow/datasets/blob/c7096bd38e86ed240b8b2c11ecab9893715a7d55/tensorflow_datasets/text/wikiann/wikiann.py#L81-L126

It would be nice to include the `spans` field in this dataset as in TFDS. This could be a good first issue for new contributors !

The objective is to use `tags_to_spans` in the `_generate_examples` method [here](https://github.com/huggingface/nlp/blob/c98e4b8f23e3770c401c6d9326e243e1ffd599ec/datasets/wikiann/wikiann.py#L292-L316) to create he `spans` for each example.

Hi @lhoestq
thank you very much for the help, it would be very nice to have it included, here is the full code, one need to also convert tags to string first:

```
import datasets
from datasets import load_dataset

def tags_to_spans(tags):
  """Convert tags to spans."""
  spans = set()
  span_start = 0
  span_end = 0
  active_conll_tag = None
  for index, string_tag in enumerate(tags):
    # Actual BIO tag.
    bio_tag = string_tag[0]
    assert bio_tag in ["B", "I", "O"], "Invalid Tag"
    conll_tag = string_tag[2:]
    if bio_tag == "O":
      # The span has ended.
      if active_conll_tag:
        spans.add((active_conll_tag, (span_start, span_end)))
      active_conll_tag = None
      # We don't care about tags we are
      # told to ignore, so we do nothing.
      continue
    elif bio_tag == "B":
      # We are entering a new span; reset indices and active tag to new span.
      if active_conll_tag:
        spans.add((active_conll_tag, (span_start, span_end)))
      active_conll_tag = conll_tag
      span_start = index
      span_end = index
    elif bio_tag == "I" and conll_tag == active_conll_tag:
      # We're inside a span.
      span_end += 1
    else:
      # This is the case the bio label is an "I", but either:
      # 1) the span hasn't started - i.e. an ill formed span.
      # 2) We have IOB1 tagging scheme.
      # We'll process the previous span if it exists, but also include this
      # span. This is important, because otherwise, a model may get a perfect
      # F1 score whilst still including false positive ill-formed spans.
      if active_conll_tag:
        spans.add((active_conll_tag, (span_start, span_end)))
      active_conll_tag = conll_tag
      span_start = index
      span_end = index
  # Last token might have been a part of a valid span.
  if active_conll_tag:
    spans.add((active_conll_tag, (span_start, span_end)))
  # Return sorted list of spans
  return sorted(list(spans), key=lambda x: x[1][0])

dataset = load_dataset('wikiann', 'en', split="train")
ner_tags = {
   0:"O",
   1:"B-PER",
   2:"I-PER",
   3:"B-ORG",
   4:"I-ORG",
   5:"B-LOC",
   6:"I-LOC"
}

def get_spans(tokens, tags):
  """Convert tags to textspans."""
  spans = tags_to_spans(tags)
  text_spans = [
      x[0] + ": " + " ".join([tokens[i]
                              for i in range(x[1][0], x[1][1] + 1)])
      for x in spans
  ]
  if not text_spans:
    text_spans = ["None"]
  return text_spans

for i, d in enumerate(dataset):
   tokens = d['tokens']
   tags = d['ner_tags']
   tags = [ner_tags[i] for i in tags]
   spans = get_spans(tokens, tags)
   print("spans ", spans)
   print(d)
   if i > 10:
     break;
```
I am not sure how to contribute to the repository and how things work, could you let me know how one can access the datasets to be able to contribute to the repository? Maybe I could do it then
thanks

Cool ! Let me give you some context:

#### Contribution guide

You can find the contribution guide here:

https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md

It explains how to set up your dev environment in a few steps.

#### Dataset loading

Each Dataset is defined by a Table that have many rows (one row = one example) and columns (one column = one feature).
To change how a dataset is constructed, you have to modify its dataset script that you can find here:

https://github.com/huggingface/datasets/blob/master/datasets/wikiann/wikiann.py

It includes everything needed to load the WikiANN dataset.
You can load locally a modified version of `wikiann.py` with `load_dataset("path/to/wikiann.py")`.

#### Define a new column

Each column has a name and a type. You can see how the features of WikiANN are defined here:

https://github.com/huggingface/datasets/blob/c98e4b8f23e3770c401c6d9326e243e1ffd599ec/datasets/wikiann/wikiann.py#L245-L263

Ideally we would have one additional feature "spans":
```python
        "spans": datasets.Sequence(datasets.Value("string")),
```

#### Compute the content of each row

To build the WikiANN rows, the _generate_examples method from [here](https://github.com/huggingface/nlp/blob/c98e4b8f23e3770c401c6d9326e243e1ffd599ec/datasets/wikiann/wikiann.py#L292-L316) is used. This function `yield` one python dictionary for each example:
```python
yield guid_index, {"tokens": tokens, "ner_tags": ner_tags, "langs": langs}
```

The objective would be to return instead something like
```python
spans = spans = get_spans(tokens, tags)
yield guid_index, {"tokens": tokens, "ner_tags": ner_tags, "langs": langs, "spans": spans}
```

Let me know if you have questions !

The PR was merged. Issue should be closed.

CC: @lhoestq

Hi
Wikiann dataset needs to have "spans" columns, which is necessary to be able to use this dataset, but this column is missing from huggingface datasets, could you please have a look? thank you @lhoestq


How to train BERT model with next sentence prediction?

Hi !
We're not using `TextDatasetForNextSentencePrediction` in `datasets`.
Although you can probably use the `TextDatasetForNextSentencePrediction.create_examples_from_document` on a dataset to prepare it for next sentence prediction.

Thanks.

Do you mean that `TextDatasetForNextSentencePrediction.create_exapmles_from_document` can be applied to dataset object other than `TextDatasetForNextSentencePrediction`  e.g. a `Dataset` object which is  loaded by  `datasets.load_dataset`?

It would probably require a bit of tweaking, but you can apply it to a dataset, yes.
This should give you a new dataset with sentence pairs you can train a model on.

You can find the documentation about dataset processing here:
https://huggingface.co/docs/datasets/processing.html#processing-data-with-map

Thank you for detail information.

I'll try to apply `create_examples_from_document` to `Dataset` object.

Hello.

I'm trying to pretrain the BERT model with next sentence prediction. Is there any function that supports next sentence prediction
like ` TextDatasetForNextSentencePrediction` of `huggingface/transformers` ?


Dialogue action slot name and value are reversed in MultiWoZ 2.2

dataset bug

A bug in a dataset script provided in the library

Hi
Good catch ! Thanks for reporting

If you are interested in contributing, feel free to open a PR to fix this :)

Hi @yjernite, thank you for adding MultiWoZ 2.2 in the huggingface datasets platform. It is beneficial!

I spot an error that the order of Dialogue action slot names and values are reversed.

https://github.com/huggingface/datasets/blob/649b2c469779bc4221e1b6969aa2496d63eb5953/datasets/multi_woz_v22/multi_woz_v22.py#L251-L262


make documentation more clear to use different cloud storage

This PR extends the cloud storage documentation. To show you can use a different `fsspec` implementation.


Replace legacy torch.Tensor constructor with torch.tensor

The title says it all (motivated by [this issue](https://github.com/pytorch/pytorch/issues/53146) in the pytorch repo).


Is dataset timit_asr broken?

Hi,

thanks for the report, but this is a duplicate of #2052.

@mariosasko
Thank you for your quick response! Following #2052, I've fixed the problem.

Using `timit_asr` dataset, I saw all records are the same.

``` python
from datasets import load_dataset, load_metric

timit = load_dataset("timit_asr")

from datasets import ClassLabel
import random
import pandas as pd
from IPython.display import display, HTML

def show_random_elements(dataset, num_examples=10):
    assert num_examples <= len(dataset), "Can't pick more elements than there are in the dataset."
    picks = []
    for _ in range(num_examples):
        pick = random.randint(0, len(dataset)-1)
        while pick in picks:
            pick = random.randint(0, len(dataset)-1)
        picks.append(pick)

    df = pd.DataFrame(dataset[picks])
    display(HTML(df.to_html()))

show_random_elements(timit['train'].remove_columns(["file", "phonetic_detail", "word_detail", "dialect_region", "id",
                                                    "sentence_type", "speaker_id"]), num_examples=20)

```

`output`

I double-checked it [here](https://huggingface.co/datasets/viewer/), and met the same problem.


Adding ScaNN library to do MIPS?

I haven't played with it (yet) but it sounds really cool !

@lhoestq Hi I am thinking of adding this new google library to do the MIPS similar to **add_faiss_idex**. As the paper suggests, it is really fast when it comes to retrieving the nearest neighbors.

https://github.com/google-research/google-research/tree/master/scann

![image](https://user-images.githubusercontent.com/16892570/112738294-78ec9800-8fc6-11eb-9a5f-3d7ee5818e76.png)


Problem downloading GEM wiki_auto_asset_turk dataset

Hi,

sadly I can't replicate the problem on my Windows machine. Try to update the library to the newest version with:
```bash
pip install git+https://github.com/huggingface/datasets
```

Thanks for the answer! I updated the library but unfortunately it didn't solve the problem.

Is there an error message ?
What stacktrace do you get if you interrupt the execution of the program while downloading ?

Sorry for the long time since my last comment, I tried again and don't seem to have the problem anymore, thanks for your support!

Great ! I'm closing the issue then. Feel free to re-open if you experience this issue again

@yjernite

### Summary

I am currently working on the GEM datasets and do not manage to download the wiki_auto_asset_turk data, whereas all other datasets download well with the same code.

### Steps to reproduce
Code snippet:

from datasets import load_dataset
#dataset = load_dataset('gem', 'web_nlg_en')
dataset = load_dataset('gem', 'wiki_auto_asset_turk')

```

**Expected behavior:**

I expect the dataset to start downloading (download bar appears and progresses toward 100%)

**Actual behavior:**
Instead of seeing the download bar appearing, nothing happens; the following appears in the console as expected, but nothing more:

Downloading: 36.6kB [00:00, 37.2MB/s]
Downloading: 41.7kB [00:00, ?B/s]
Downloading and preparing dataset gem/wiki_auto_asset_turk (download: 121.37 MiB, generated: 145.69 MiB, post-processed: Unknown size, total: 267.07 MiB) to C:\Users\sfmil\.cache\huggingface\datasets\gem\wiki_auto_asset_turk\1.0.0\f252756d7f1b8f019aac71a1623b2950acfe10d25d956668ac4eae4e93c58b8d...

### Is this a regression?
No, it was the first time I was trying to download this dataset (same for the other ones).

### Debug info
- Python version: Python 3.8.2
- OS version: Windows 10 Family


Fast table queries with interpolation search

## Intro

This should fix issue #1803

Currently querying examples in a dataset is O(n) because of the underlying pyarrow ChunkedArrays implementation.
To fix this I implemented interpolation search that is pretty effective since datasets usually verifies the condition of evenly distributed chunks (the default chunk size is fixed).

## Benchmark

Here is a [benchmark](https://pastebin.com/utEXUqsR) I did on bookcorpus (74M rows):

for the current implementation
```python
>>> python speed.py
Loaded dataset 'bookcorpus', len=74004228, nbytes=4835358766

========================= Querying unshuffled bookcorpus =========================

Avg access time key=1                                                 : 0.018ms
Avg access time key=74004227                                          : 0.215ms
Avg access time key=range(74003204, 74004228)                         : 1.416ms
Avg access time key=RandIter(low=0, high=74004228, size=1024, seed=42): 92.532ms

========================== Querying shuffled bookcorpus ==========================

Avg access time key=1                                                 : 0.187ms
Avg access time key=74004227                                          : 6.642ms
Avg access time key=range(74003204, 74004228)                         : 90.941ms
Avg access time key=RandIter(low=0, high=74004228, size=1024, seed=42): 3448.456ms
```

for the new one using interpolation search:
```python
>>> python speed.py
Loaded dataset 'bookcorpus', len=74004228, nbytes=4835358766

========================= Querying unshuffled bookcorpus =========================

Avg access time key=1                                                 : 0.076ms
Avg access time key=74004227                                          : 0.056ms
Avg access time key=range(74003204, 74004228)                         : 1.807ms
Avg access time key=RandIter(low=0, high=74004228, size=1024, seed=42): 24.028ms

========================== Querying shuffled bookcorpus ==========================

Avg access time key=1                                                 : 0.061ms
Avg access time key=74004227                                          : 0.058ms
Avg access time key=range(74003204, 74004228)                         : 22.166ms
Avg access time key=RandIter(low=0, high=74004228, size=1024, seed=42): 42.757ms
```

The RandIter class is just an iterable of 1024 random indices from 0 to 74004228.

Here is also a plot showing the speed improvement depending on the dataset size:
![image](https://user-images.githubusercontent.com/42851186/112673587-32335c80-8e65-11eb-9a0c-58ad774abaec.png)

## Implementation details:
- `datasets.table.Table` objects implement interpolation search for the `slice` method
- The interpolation search requires to store the offsets of all the chunks of a table. The offsets are stored when the `Table` is initialized.
- `datasets.table.Table.slice` returns a `datasets.table.Table` using interpolation search
- `datasets.table.Table.fast_slice` returns a `pyarrow.Table` object using interpolation search. This is useful to get a part of a dataset if we don't need the indexing structure for future computations. For example it's used when querying an example as a dictionary.
- Now a `Dataset` object is always backed by a `datasets.table.Table` object. If one passes a `pyarrow.Table` to initialize a `Dataset`, then it's converted to a `datasets.table.Table`

## Checklist:

- [x] implement interpolation search
- [x] use `datasets.table.Table` in `Dataset` objects
- [x] update current tests
- [x] add tests for interpolation search
- [x] comments and docstring
- [x] add the benchmark to the CI

Fix #1803.