ArrowInvalid issue for squad v2 dataset

Hi ! This error happens when you use `map` in batched mode and then your function doesn't return the same number of values per column.

Indeed since you're using `map` in batched mode, `prepare_validation_features` must take a batch as input (i.e. a dictionary of multiple rows of the dataset), and return a batch.

However it seems like `tokenized_examples` doesn't have the same number of elements in each field. One field seems to have `1180` elements while `candidate_attention_mask` only has `1178`.

Hello, I am using the huggingface official question answering example notebook (https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb).

In the prepare_validation_features function, I made some modifications to tokenize a new set of quesions with the original contexts and save them in three different list called candidate_input_dis, candidate_attetion_mask and candidate_token_type_ids. When I try to run the next cell for dataset.map, I got the following error:

`ArrowInvalid: Column 1 named candidate_attention_mask expected length 1180 but got length 1178`

My code is as follows:

```
def generate_candidate_questions(examples):
  val_questions = examples["question"]
  candididate_questions = random.sample(datasets["train"]["question"], len(val_questions))
  candididate_questions = [x[:max_length] for x in candididate_questions]
  return candididate_questions

def prepare_validation_features(examples, use_mixing=False):
  pad_on_right = tokenizer.padding_side == "right"
  tokenized_examples = tokenizer(
      examples["question" if pad_on_right else "context"],
      examples["context" if pad_on_right else "question"],
      truncation="only_second" if pad_on_right else "only_first",
      max_length=max_length,
      stride=doc_stride,
      return_overflowing_tokens=True,
      return_offsets_mapping=True,
      padding="max_length",
  )
  if use_mixing:
    candidate_questions = generate_candidate_questions(examples)
    tokenized_candidates = tokenizer(
        candidate_questions if pad_on_right else examples["context"],
        examples["context"] if pad_on_right else candidate_questions,
        truncation="only_second" if pad_on_right else "only_first",
        max_length=max_length,
        stride=doc_stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )

  sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")

  tokenized_examples["example_id"] = []

  if use_mixing:
    tokenized_examples["candidate_input_ids"] = tokenized_candidates["input_ids"]
    tokenized_examples["candidate_attention_mask"] = tokenized_candidates["attention_mask"]
    tokenized_examples["candidate_token_type_ids"] = tokenized_candidates["token_type_ids"]

  for i in range(len(tokenized_examples["input_ids"])):
      sequence_ids = tokenized_examples.sequence_ids(i)
      context_index = 1 if pad_on_right else 0

      sample_index = sample_mapping[i]
      tokenized_examples["example_id"].append(examples["id"][sample_index])

      tokenized_examples["offset_mapping"][i] = [
          (o if sequence_ids[k] == context_index else None)
          for k, o in enumerate(tokenized_examples["offset_mapping"][i])
      ]

  return tokenized_examples

validation_features = datasets["validation"].map(
    lambda xs: prepare_validation_features(xs, True),
    batched=True,
    remove_columns=datasets["validation"].column_names
)
```

I guess this might happen because of the batched=True. I see similar issues in this repo related to arrow table length mismatch error, but in their cases, the numbers vary a lot. In my case, this error always happens when the expected length and unexpected length are very close. Thanks for the help!


Add and fix docstring for NamedSplit

Maybe we should add some other split classes?

Add and fix docstring for `NamedSplit`, which was missing.


PyTorch not available error on SageMaker GPU docker though it is installed

Hey @sivakhno,

how does your `requirements.txt` look like to install the `datasets` library and which version of it are you running? Can you try to install `datasets>=1.4.0`

Hi @philschmid - thanks for suggestion. I am using `datasets==1.4.1`.
I have also tried using `torch=1.6.0` (docker `763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.6.0-gpu-py3 `), but the error is the same.

Could paste the code you use the start your training job and the fine-tuning script you run?

@sivakhno this should be now fixed in `datasets>=1.5.0`.

@philschmid Recently released tensorflow-macos seems to be missing.

I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!


Multiprocessing windows error

Hi ! Thanks for reporting.
This looks like a bug, could you try to provide a minimal code example that reproduces the issue ? This would be very helpful !

Otherwise I can try to run the wav2vec2 code above on my side but probably not this week..

```
from datasets import load_dataset

dataset = load_dataset('glue', 'mrpc', split='train')

updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)

```

I was able to copy some of the shell
This is repeating every half second
Win 10, Anaconda with python 3.8, datasets installed from main branche
```

  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    _check_not_importing_main()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 134, in _check_not_importing_main
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    exitcode = _main(fd, parent_sentinel)
    raise RuntimeError('''
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 125, in _main
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.    return _run_module_code(code, init_globals, run_name,
    prepare(preparation_data)

  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 236, in prepare
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\test.py", line 6, in <module>
    updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1370, in map
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 119, in Pool
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 212, in __init__
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\test.py", line 6, in <module>
    self._repopulate_pool()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 303, in _repopulate_pool
    updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1370, in map
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 326, in _repopulate_pool_static
    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 119, in Pool
    w.start()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\process.py", line 121, in start
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 212, in __init__
    self._popen = self._Popen(self)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 327, in _Popen
    self._repopulate_pool()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 303, in _repopulate_pool
    return Popen(process_obj)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\popen_spawn_win32.py", line 45, in __init__
    return self._repopulate_pool_static(self._ctx, self.Process,
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 326, in _repopulate_pool_static
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 134, in _check_not_importing_main
    w.start()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\process.py", line 121, in start
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...
```

Thanks this is really helpful !
I'll try to reproduce on my side and come back to you

if __name__ == '__main__':

This line before calling the map function stops the error but the script still repeats endless

Indeed you needed `if __name__ == '__main__'` since accoding to [this stackoverflow post](https://stackoverflow.com/a/18205006):

> On Windows the subprocesses will import (i.e. execute) the main module at start. You need to insert an if __name__ == '__main__': guard in the main module to avoid creating subprocesses recursively.

Regarding the hanging issue, can you try to update `dill` and `multiprocess` ?

```
Traceback (most recent call last):
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 791, in move
    os.rename(src, real_dst)
FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\tmpx9fl_jg8' -> 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\cache-9b4f203a63742dfc.arrow'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\cvtrain.py", line 243, in <module>
    common_voice_train = common_voice_train.map(remove_special_characters, remove_columns=["sentence"])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1339, in map
    return self._map_single(
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 203, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1646, in _map_single
    shutil.move(tmp_file.name, cache_file_name)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 805, in move
    copy_function(src, real_dst)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  0%|                                                                                | 0/27771 [00:00<?, ?ex/s]
 File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
OSError: [Errno 22] Invalid argument: 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\cache-9b4f203a63742dfc.arrow'
```

I was adding freeze support before calling the mapping function like this
if __name__ == '__main__':
    freeze_support()
   dataset.map(....)

Usually OSError of an arrow file on windows means that the file is currently opened as a dataset object, so you can't overwrite it until the dataset object falls out of scope.
Can you make sure that there's no dataset object that loaded the `cache-9b4f203a63742dfc.arrow` file ?

Now I understand
The error occures because the script got restarted in another thread, so the object is already loaded.
Still don't have an idea why a new thread starts the whole script again

As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop


Fix docstring rendering of Dataset/DatasetDict.from_csv args

Fix the docstring rendering of Dataset/DatasetDict.from_csv args.


Only user permission of saved cache files, not group

enhancement

New feature or request

good first issue

Good for newcomers

Hi ! Thanks for reporting.

Currently there's no way to specify this.

When loading/processing a dataset, the arrow file is written using a temporary file. Then once writing is finished, it's moved to the cache directory (using `shutil.move` [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1646))

That means it keeps the permissions specified by the `tempfile.NamedTemporaryFile` object, i.e. `-rw-------` instead of `-rw-r--r--`. Improving this could be a nice first contribution to the library :)

Hi @lhoestq,
I looked into this and yes you're right. The `NamedTemporaryFile` is always created with mode 0600, which prevents group from reading the file. Should we change the permissions of `tmp_file.name` [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1871) and [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1590), post creation to 0644 inorder for group and others to read it?

Good idea :) we could even update the permissions after the file has been moved by shutil.move [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1899) and [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1646) actually.
Apparently they set the default 0600 for temporary files for security reasons, so let's update the umask only after the file has been moved

Would it be possible to actually set the umask based on a user provided argument? For example, a popular usecase my team has is using a shared file-system for processing datasets. This may involve writing/deleting other files, or changing filenames, which a -rw-r--r-- wouldn't fix.

Note that you can get the cache files of a dataset with the `cache_files` attributes.
Then you can `chmod` those files and all the other cache files in the same directory.

Moreover we can probably keep the same permissions after each transform. This way you just need to set the permissions once after doing `load_dataset` for example, and then all the new transformed cached files will have the same permissions.
What do you think ?

This means we'll check the permissions of other `cache_files` already created for a dataset before setting permissions for new  `cache_files`?

You can just check the permission of `dataset.cache_files[0]` imo

> This way you just need to set the permissions once after doing load_dataset for example, and then all the new transformed cached files will have the same permissions.

I was referring to this. Ensuring that newly generated `cache_files` have the same permissions

Yes exactly

I imagine users can first do `load_dataset`, then chmod on the arrow files. After that all the new cache files could have the same permissions as the first arrow files. Opinions on this ?

Sounds nice but I feel this is a sub-part of the approach mentioned by @siddk. Instead of letting the user set new permissions by itself first and then making sure newly generated files have same permissions why don't we ask the user initially only what they want? What are your thoughts?

Yes sounds good. Should this be a parameter in `load_dataset` ? Or an env variable ? Or use the value of `os.umask` ?

Ideally it should be a parameter in `load_dataset` but I'm not sure how important it is for the users (considering only important things should go into `load_dataset` parameters)

I think it's fairly important; for context, our team uses a shared file-system where many folks run experiments based on datasets that are cached by other users.

For example, I might start a training run, downloading a dataset. Then, a couple of days later, a collaborator using the same repository might want to use the same dataset on the same shared filesystem, but won't be able to under the default permissions.

Being able to specify directly in the top-level `load_dataset()` call seems important, but an equally valid option would be to just inherit from the running user's `umask` (this should probably be the default anyway).

So basically, argument that takes a custom set of permissions, and by default, use the running user's umask!

Maybe let's start by defaulting to the user's umask !
Do you want to give it a try @bhavitvyamalik ?

Yeah sure! Instead of using default `0o644` should I first extract umask of current user and then use `os.umask` on it? We can do it inside `Dataset` class so that all folders/files created during the call use running user's umask

You can get the umask using `os.umask` and then I guess you can just use `os.chmod` as in your previous PR, but with the right permissions depending on the umask.

FWIW, we have this issue with other caches - e.g. `transformers` model files. So probably will need to backport this into `transformers` as well.

thanks @thomwolf for the pointer.

Hi @stas00,
For this should we use the same umask code in the respective model directory inside `TRANSFORMERS_CACHE`?

That sounds very right to me, @bhavitvyamalik

Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?


Fix ted_talks_iwslt version error

This PR fixes the bug where the version argument would be passed twice if the dataset configuration was created on the fly.

Fixes #2059


[Common Voice] Adapt dataset script so that no manual data download is actually needed

This PR changes the dataset script so that no manual data dir is needed anymore.


docs: fix missing quotation

The json code misses a quote


Cannot load udpos subsets from xtreme dataset using load_dataset()

Good for newcomers

@lhoestq Adding "_" to the class labels in the dataset script will fix the issue.

The bigger issue IMO is that the data files are in conll format, but the examples are tokens, not sentences.

Hi ! Thanks for reporting @adzcodez

> @lhoestq Adding "_" to the class labels in the dataset script will fix the issue.
>
> The bigger issue IMO is that the data files are in conll format, but the examples are tokens, not sentences.

You're right: "_" should be added to the list of labels, and the examples must be sequences of tokens, not singles tokens.

@lhoestq Can you please label this issue with the "good first issue" label? I'm not sure I'll find time to fix this.

To resolve it, the user should:
1. add `"_"` to the list of labels
2. transform the udpos subset to the conll format (I think the preprocessing logic can be borrowed from [the original repo](https://github.com/google-research/xtreme/blob/58a76a0d02458c4b3b6a742d3fd4ffaca80ff0de/utils_preprocess.py#L187-L204))
3. update the dummy data
4. update the dataset info
5. [optional] add info about the data fields structure of the udpos subset to the dataset readme

I tried fixing this issue, but its working fine in the dev version : "1.6.2.dev0"

I think somebody already fixed it.

Hi,

after #2326, the lines with pos tags equal to `"_"` are filtered out when generating the dataset, so this fixes the KeyError described above. However, the udpos subset should be in the conll format i.e. it should yield sequences of tokens and not single tokens, so it would be great to see this fixed (feel free to borrow the logic from [here](https://github.com/google-research/xtreme/blob/58a76a0d02458c4b3b6a742d3fd4ffaca80ff0de/utils_preprocess.py#L187-L204) if you decide to work on this).

Hello,

I am trying to load the udpos English subset from xtreme dataset, but this faces an error during loading. I am using datasets v1.4.1, pip install. I have tried with other udpos languages which also fail, though loading a different subset altogether (such as XNLI) has no issue. I have also tried on Colab and faced the same error.

Reprex is:

`from datasets import load_dataset `
`dataset = load_dataset('xtreme', 'udpos.English')`

The error is:
`KeyError: '_'`

The full traceback is:
KeyError                                  Traceback (most recent call last)
<ipython-input-5-7181359ea09d> in <module>
      1 from datasets import load_dataset
----> 2 dataset = load_dataset('xtreme', 'udpos.English')

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
    738
    739     # Download and prepare data
--> 740     builder_instance.download_and_prepare(
    741         download_config=download_config,
    742         download_mode=download_mode,

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    576                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    577                     if not downloaded_from_gcs:
--> 578                         self._download_and_prepare(
    579                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    580                         )

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    654             try:
    655                 # Prepare split will record examples associated to the split
--> 656                 self._prepare_split(split_generator, **prepare_split_kwargs)
    657             except OSError as e:
    658                 raise OSError(

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _prepare_split(self, split_generator)
    977                 generator, unit=" examples", total=split_info.num_examples, leave=False, disable=not_verbose
    978             ):
--> 979                 example = self.info.features.encode_example(record)
    980                 writer.write(example)
    981         finally:

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example)
    946     def encode_example(self, example):
    947         example = cast_to_python_objects(example)
--> 948         return encode_nested_example(self, example)
    949
    950     def encode_batch(self, batch):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    840     # Nested structures: we allow dict, list/tuples, sequences
    841     if isinstance(schema, dict):
--> 842         return {
    843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in <dictcomp>(.0)
    841     if isinstance(schema, dict):
    842         return {
--> 843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }
    845     elif isinstance(schema, (list, tuple)):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    868     # ClassLabel will convert from string to int, TranslationVariableLanguages does some checks
    869     elif isinstance(schema, (ClassLabel, TranslationVariableLanguages, Value, _ArrayXD)):
--> 870         return schema.encode_example(obj)
    871     # Other object should be directly convertible to a native Arrow type (like Translation and Translation)
    872     return obj

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example_data)
    647         # If a string is given, convert to associated integer
    648         if isinstance(example_data, str):
--> 649             example_data = self.str2int(example_data)
    650
    651         # Allowing -1 to mean no label.

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in str2int(self, values)
    605                 if value not in self._str2int:
    606                     value = value.strip()
--> 607                 output.append(self._str2int[str(value)])
    608             else:
    609                 # No names provided, try to integerize

KeyError: '_'


Filtering refactor

I thought at first that the multiproc test was not relevant now that we do stuff only in memory, but I think there's something that's actually broken, my tiny benchmark on bookcorpus runs forever (2hrs+) when I add `num_proc=4` as a kwarg, will investigate 👀

I'm not familiar with the caching you describe for `.map`, I'll look it up.

turns out the multi proc issue is also on master, I won't fix it in this PR but opened #2071 to track the problem.

tracemalloc outputs from this script:

```python
import logging
import sys
import time
import tracemalloc

from datasets import load_dataset, set_caching_enabled

if __name__ == "__main__":
    set_caching_enabled(False)
    logging.basicConfig(level=logging.DEBUG)

    tracemalloc.start()
    bc = load_dataset("bookcorpus")

    now = time.time()
    try:
        snapshot1 = tracemalloc.take_snapshot()
        bc["train"].filter(lambda x: len(x["text"]) < 64, num_proc=int(sys.argv[1]))
    except Exception as e:
        print(f"cancelled: {e}")
        exit(1)
    snapshot2 = tracemalloc.take_snapshot()
    tracemalloc.stop()
    elapsed = time.time() - now

    print(elapsed)
    top_stats = snapshot2.compare_to(snapshot1, "lineno")

    print("[ Top 10 differences ]")
    for stat in top_stats[:10]:
        print(stat)

```

This branch:

```
    ssh://theo@35.205.12.130:22/home/theo/.local/share/miniconda3/envs/datasets/bin/python -u benchmark_filter.py 1
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com:443
    DEBUG:urllib3.connectionpool:https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/datasets/bookcorpus/bookcorpus.py HTTP/1.1" 200 0
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com:443
    DEBUG:urllib3.connectionpool:https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/master/datasets/bookcorpus/bookcorpus.py HTTP/1.1" 200 0
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com:443
    DEBUG:urllib3.connectionpool:https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/master/datasets/bookcorpus/dataset_infos.json HTTP/1.1" 200 0
    WARNING:datasets.builder:Reusing dataset bookcorpus (/home/theo/.cache/huggingface/datasets/bookcorpus/plain_text/1.0.0/af844be26c089fb64810e9f2cd841954fd8bd596d6ddd26326e4c70e2b8c96fc)
      0%|                                                 | 0/74005 [00:00<?, ?ba/s]2021-03-23 10:23:20.051255: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2021-03-23 10:23:20.051304: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
    100%|████████████████████████████████████| 74005/74005 [12:16<00:00, 100.54ba/s]
    815.6356580257416
    [ Top 10 differences ]
    <frozen importlib._bootstrap_external>:580: size=38.0 MiB (+33.7 MiB), count=326226 (+307928), average=122 B
    <frozen importlib._bootstrap>:219: size=7643 KiB (+7553 KiB), count=26372 (+25473), average=297 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/torch/__init__.py:427: size=1291 KiB (+1291 KiB), count=5924 (+5924), average=223 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/abc.py:85: size=1039 KiB (+1026 KiB), count=3428 (+3384), average=310 B
    <frozen importlib._bootstrap_external>:64: size=917 KiB (+891 KiB), count=5300 (+5132), average=177 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/collections/__init__.py:456: size=720 KiB (+709 KiB), count=3403 (+3349), average=217 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/tf_export.py:346: size=607 KiB (+607 KiB), count=3962 (+3962), average=157 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/linecache.py:137: size=998 KiB (+487 KiB), count=9551 (+4517), average=107 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/tf_decorator.py:241: size=367 KiB (+367 KiB), count=5225 (+5225), average=72 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/decorator_utils.py:114: size=359 KiB (+359 KiB), count=330 (+330), average=1114 B
```

On master:
```
    ssh://theo@35.205.12.130:22/home/theo/.local/share/miniconda3/envs/datasets/bin/python -u benchmark_filter.py 1
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com:443
    DEBUG:urllib3.connectionpool:https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/datasets/bookcorpus/bookcorpus.py HTTP/1.1" 200 0
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com:443
    DEBUG:urllib3.connectionpool:https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/master/datasets/bookcorpus/bookcorpus.py HTTP/1.1" 200 0
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com:443
    DEBUG:urllib3.connectionpool:https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/master/datasets/bookcorpus/dataset_infos.json HTTP/1.1" 200 0
    WARNING:datasets.builder:Reusing dataset bookcorpus (/home/theo/.cache/huggingface/datasets/bookcorpus/plain_text/1.0.0/af844be26c089fb64810e9f2cd841954fd8bd596d6ddd26326e4c70e2b8c96fc)
      0%|                                                 | 0/74005 [00:00<?, ?ba/s]2021-03-23 12:26:47.219622: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2021-03-23 12:26:47.219669: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
    100%|███████████████████████████████████| 74005/74005 [1:02:17<00:00, 19.80ba/s]
    3738.870892047882
    [ Top 10 differences ]
    <frozen importlib._bootstrap_external>:580: size=38.0 MiB (+33.7 MiB), count=326221 (+307919), average=122 B
    <frozen importlib._bootstrap>:219: size=7648 KiB (+7557 KiB), count=26455 (+25555), average=296 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/torch/__init__.py:427: size=1291 KiB (+1291 KiB), count=5924 (+5924), average=223 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/abc.py:85: size=1039 KiB (+1026 KiB), count=3429 (+3385), average=310 B
    <frozen importlib._bootstrap_external>:64: size=917 KiB (+891 KiB), count=5300 (+5132), average=177 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/collections/__init__.py:456: size=720 KiB (+709 KiB), count=3403 (+3349), average=217 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/tf_export.py:346: size=607 KiB (+607 KiB), count=3962 (+3962), average=157 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/linecache.py:137: size=1000 KiB (+489 KiB), count=9569 (+4535), average=107 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/tf_decorator.py:241: size=367 KiB (+367 KiB), count=5225 (+5225), average=72 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/decorator_utils.py:114: size=359 KiB (+359 KiB), count=330 (+330), average=1114 B
```

I'm not concluding much, it seems nothing is really happening to memory on `pyarrow::Table.filter`?

Cool ! Maybe it increases the memory a bit but what's brought in memory is not the resulting Table but something else (not sure what though).
What's the length of the resulting dataset ?
You can also take a look at `pyarrow.total_allocated_memory()` to show how much memory is being used by pyarrow

```diff
diff --git a/benchmarks/benchmark_filter.py b/benchmarks/benchmark_filter.py
index 4b9efd4e..a862c204 100644
--- a/benchmarks/benchmark_filter.py
+++ b/benchmarks/benchmark_filter.py
@@ -1,6 +1,9 @@
 import logging
 import sys
 import time
+import tracemalloc
+
+import pyarrow as pa

 from datasets import load_dataset, set_caching_enabled

@@ -9,13 +12,28 @@ if __name__ == "__main__":
     set_caching_enabled(False)
     logging.basicConfig(level=logging.DEBUG)

+    tracemalloc.start()
     bc = load_dataset("bookcorpus")

     now = time.time()
     try:
+        snapshot1 = tracemalloc.take_snapshot()
+        pamem1 = pa.total_allocated_bytes()
         bc["train"].filter(lambda x: len(x["text"]) < 64, num_proc=int(sys.argv[1]))
+        pamem2 = pa.total_allocated_bytes()
+        snapshot2 = tracemalloc.take_snapshot()
     except Exception as e:
         print(f"cancelled: {e}")
+        exit(1)
+    tracemalloc.stop()
     elapsed = time.time() - now

     print(elapsed)
+    top_stats = snapshot2.compare_to(snapshot1, "lineno")
+
+    print("[ Top 10 differences ]")
+    for stat in top_stats[:10]:
+        print(stat)
+
+    print("[ pyarrow reporting ]")
+    print(f"before: ({pamem1}) after: ({pamem2})")
```

this yields 0-0, does not seem like a good tool 😛 and the documentation is [quite mysterious.](https://arrow.apache.org/docs/python/generated/pyarrow.total_allocated_bytes.html)

Personally if I use your script to benchmark on this branch
```python
bc = load_dataset("bookcorpus", split="train[:1%]")
bc = bc.filter(lambda x: len(x["text"]) < 64)
```

then I get
```
[ pyarrow reporting ]
before: (0) after: (15300672)
```

Maybe you got 0-0 because the filter output is directly garbage collected, since you didn't do
```python
bc["train"] = bc["train"].filter(...)
```
Can you try again on your side just to make sure ?

Even if the documentation doesn't say much, `pa.total_allocated_bytes` if pretty useful, and also very consistent.
It tracks the number of bytes used for arrow data.

> Maybe you got 0-0 because the filter output is directly garbage collected, since you didn't do
>
> ```python
> bc["train"] = bc["train"].filter(...)
> ```
Nice catch! I get 1.74GB for this branch

Looks like we may need to write the filtered table on the disk then.

The other option is to slice the table to keep only the good rows and concatenate them but this is too slow at the moment since slicing is O(n) until #1803 is fixed. I'll work on this issue this afternoon

From investigation it looks like the lib's `Table.filter` cannot send its output to memorymap, asked a question on the mailing list, see [here](https://lists.apache.org/thread.html/r8cd8591ce83a967eb0097a7f31785ac2f3ee95ea371c8c5beb0720ad%40%3Cuser.arrow.apache.org%3E)

fix https://github.com/huggingface/datasets/issues/2032

benchmarking is somewhat inconclusive, currently running on `book_corpus` with:

```python
    bc = load_dataset("bookcorpus")
    now = time.time()
    bc.filter(lambda x: len(x["text"]) < 64)
    elapsed = time.time() - now
    print(elapsed)
```

this branch does it in 233 seconds, master in 1409 seconds.


Error while following docs to load the `ted_talks_iwslt` dataset

dataset bug

A bug in a dataset script provided in the library

@skyprince999 as you authored the PR for this dataset, any comments?

This has been fixed in #2064  by @mariosasko (thanks again !)

The fix is available on the master branch and we'll do a new release very soon :)

I am currently trying to load the `ted_talks_iwslt` dataset into google colab.

The [docs](https://huggingface.co/datasets/ted_talks_iwslt) mention the following way of doing so.

```python
dataset = load_dataset("ted_talks_iwslt", language_pair=("it", "pl"), year="2014")
```

Executing it results in the error attached below.

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-7dcc67154ef9> in <module>()
----> 1 dataset = load_dataset("ted_talks_iwslt", language_pair=("it", "pl"), year="2014")

4 frames
/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
    730         hash=hash,
    731         features=features,
--> 732         **config_kwargs,
    733     )
    734

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in __init__(self, writer_batch_size, *args, **kwargs)
    927
    928     def __init__(self, *args, writer_batch_size=None, **kwargs):
--> 929         super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
    930         # Batch size used by the ArrowWriter
    931         # It defines the number of samples that are kept in memory before writing them

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in __init__(self, cache_dir, name, hash, features, **config_kwargs)
    241             name,
    242             custom_features=features,
--> 243             **config_kwargs,
    244         )
    245

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    337             if "version" not in config_kwargs and hasattr(self, "VERSION") and self.VERSION:
    338                 config_kwargs["version"] = self.VERSION
--> 339             builder_config = self.BUILDER_CONFIG_CLASS(**config_kwargs)
    340
    341         # otherwise use the config_kwargs to overwrite the attributes

/root/.cache/huggingface/modules/datasets_modules/datasets/ted_talks_iwslt/024d06b1376b361e59245c5878ab8acf9a7576d765f2d0077f61751158e60914/ted_talks_iwslt.py in __init__(self, language_pair, year, **kwargs)
    219             description=description,
    220             version=datasets.Version("1.1.0", ""),
--> 221             **kwargs,
    222         )
    223

TypeError: __init__() got multiple values for keyword argument 'version'
```

How to resolve this?

PS: Thanks a lot @huggingface team for creating this great library!


Is it possible to convert a `tfds` to HuggingFace `dataset`?

I was having some weird bugs with `C4`dataset version of HuggingFace, so I decided to try to download `C4`from `tfds`. I would like to know if it is possible to convert a tfds dataset to HuggingFace dataset format :)

I can also open a new issue reporting the bug I'm receiving with `datasets.load_dataset('c4','en')` in the future if you think that it would be useful.

Thanks!


update link to ZEST dataset

Updating the link as the original one is no longer working.


issue with opus100/en-fr dataset

@lhoestq  I also deleted the cache and redownload the file and still the same issue, I appreciate any help on this. thanks

Here please find the minimal code to reproduce the issue @lhoestq  note this only happens with MT5TokenizerFast

```
from datasets import load_dataset
from transformers import MT5TokenizerFast

def get_tokenized_dataset(dataset_name, dataset_config_name, tokenizer):
    datasets = load_dataset(dataset_name, dataset_config_name, script_version="master")
    column_names = datasets["train"].column_names
    text_column_name = "translation"
    def process_dataset(datasets):
        def process_function(examples):
            lang = "fr"
            return {"src_texts": [example[lang] for example in examples[text_column_name]]}
        datasets = datasets.map(
            process_function,
            batched=True,
            num_proc=None,
            remove_columns=column_names,
            load_from_cache_file=True,
        )
        return datasets
    datasets = process_dataset(datasets)
    text_column_name = "src_texts"
    column_names = ["src_texts"]
    def tokenize_function(examples):
            return tokenizer(examples[text_column_name], return_special_tokens_mask=True)
    tokenized_datasets = datasets.map(
            tokenize_function,
            batched=True,
            num_proc=None,
            remove_columns=column_names,
            load_from_cache_file=True
    )

if __name__ == "__main__":
     tokenizer_kwargs = {
        "cache_dir": None,
        "use_fast": True,
        "revision": "main",
        "use_auth_token": None
     }
     tokenizer = MT5TokenizerFast.from_pretrained("google/mt5-small", **tokenizer_kwargs)
     get_tokenized_dataset(dataset_name="opus100", dataset_config_name="en-fr", tokenizer=tokenizer)
~
```

as per https://github.com/huggingface/tokenizers/issues/626 this looks like to be the tokenizer bug, I therefore, reported it there https://github.com/huggingface/tokenizers/issues/626 and  I am closing this one.

Hi
I am running run_mlm.py code of huggingface repo with opus100/fr-en pair, I am getting this error, note that this error occurs for only this pairs and not the other pairs. Any idea why this is occurring? and how I can solve this?

Thanks a lot  @lhoestq for your help in advance.

`
thread '<unnamed>' panicked at 'index out of bounds: the len is 617 but the index is 617', /__w/tokenizers/tokenizers/tokenizers/src/tokenizer/normalizer.rs:382:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 63%|██████████████████████████████████████████████████████████▊                                   | 626/1000 [00:27<00:16, 22.69ba/s]

Traceback (most recent call last):
  File "run_mlm.py", line 550, in <module>
    main()
  File "run_mlm.py", line 412, in main
    in zip(data_args.dataset_name, data_args.dataset_config_name)]
  File "run_mlm.py", line 411, in <listcomp>
    logger) for dataset_name, dataset_config_name\
  File "/user/dara/dev/codes/seq2seq/data/tokenize_datasets.py", line 96, in get_tokenized_dataset
    load_from_cache_file=not data_args.overwrite_cache,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/dataset_dict.py", line 448, in map
    for k, dataset in self.items()
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/dataset_dict.py", line 448, in <dictcomp>
    for k, dataset in self.items()
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1309, in map
    update_data=update_data,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 204, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1574, in _map_single
    batch, indices, check_same_num_examples=len(self.list_indexes()) > 0, offset=offset
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1490, in apply_function_on_filtered_inputs
    function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "/user/dara/dev/codes/seq2seq/data/tokenize_datasets.py", line 89, in tokenize_function
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2347, in __call__
    **kwargs,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2532, in batch_encode_plus
    **kwargs,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 384, in _batch_encode_plus
    is_pretokenized=is_split_into_words,
pyo3_runtime.PanicException: index out of bounds: the len is 617 but the index is 617

`


is there a way to override a dataset object saved with save_to_disk?

Hi
You can rename the arrow file and update the name in `state.json`

I tried this way, but when there is a mapping process to the dataset, it again uses a random cache name. atm, I am trying to use the following method by setting an exact cache file,

```
            dataset_with_embedding =csv_dataset.map(
                partial(self.embed, ctx_encoder=ctx_encoder, ctx_tokenizer=self.context_tokenizer),
                batched=True,
                batch_size=1,
                features=new_features,
                cache_file_name=cache_arrow_path,
                load_from_cache_file=False
            )
```
So here we set a cache_file_name , after this it uses the same  file name when saving again and again.

I'm not sure I understand your issue, can you elaborate ?

`cache_file_name` is indeed an argument you can set to specify the cache file that will be used for the processed dataset. By default the file is named with something like `cache-<fingerprint>.arrow` where the fingerprint is a hash.

Let's say I am updating a set of embedding in a dataset that is around 40GB inside a training loop every 500 steps (Ex: calculating the embeddings in updated ctx_encoder in RAG and saving it to the passage path).  So when we use **dataset_object.save_to_disk('passage_path_directory')** it will save the new dataset object every time with a random file name, especially when we do some transformations to dataset objects such as map or shards. This way, we keep collecting unwanted files that will eventually eat up all the disk space.

But if we can save the dataset object every time by a single name like **data_shard_1.arrow**, it will automatically remove the previous file and save the new one in the same directory.  I found the above-mentioned code snippet useful to complete this task.

Is this clear?

At the moment when I use save_to_disk, it uses the arbitrary name for the arrow file.  Is there a way to override such an object?


Could not find file for ZEST dataset

The zest dataset url was changed (allenai/zest#3) and #2057 should resolve this.

This has been fixed in #2057 by @matt-peters (thanks again !)

The fix is available on the master branch and we'll do a new release very soon :)

I am closing this issue since its fixed!

I am trying to use zest dataset from Allen AI using below code in colab,
```
!pip install -q datasets
from datasets import load_dataset
dataset = load_dataset("zest")
```

I am getting the following error,
```
Using custom data configuration default

Downloading and preparing dataset zest/default (download: 5.53 MiB, generated: 19.96 MiB, post-processed: Unknown size, total: 25.48 MiB) to /root/.cache/huggingface/datasets/zest/default/0.0.0/1f7a230fbfc964d979bbca0f0130fbab3259fce547ee758ad8aa4f9c9bec6cca...
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-18dbbc1a4b8a> in <module>()
      1 from datasets import load_dataset
      2
----> 3 dataset = load_dataset("zest")

9 frames
/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    612             )
    613         elif response is not None and response.status_code == 404:
--> 614             raise FileNotFoundError("Couldn't find file at {}".format(url))
    615         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
    616         raise ConnectionError("Couldn't reach {}".format(url))

FileNotFoundError: Couldn't find file at https://ai2-datasets.s3-us-west-2.amazonaws.com/zest/zest.zip
```


Add bAbI QA tasks

Hi @lhoestq,

Should I remove the 160 configurations? Is it too much?

EDIT:
Can you also check the task category? I'm not sure if there is an appropriate tag for the same.

Thanks for the changes !

> Should I remove the 160 configurations? Is it too much?

Yea 160 configuration is a lot.
Maybe this dataset can work with parameters `type` and `task_no` ?
You can just remove the configuration in BUILDER_CONFIGS to only keep a few ones.
Also feel free to add an example in the dataset card of how to load the other configurations
```
load_dataset("babi_qa", type="hn", task_no="qa1")
```
for example, and with a list of the possible combinations.

> Can you also check the task category? I'm not sure if there is an appropriate tag for the same.

It looks appropriate, thanks :)

Hi @lhoestq

I'm unable to test it locally using:
```python
load_dataset("datasets/babi_qa", type="hn", task_no="qa1")
```
It raises an error:
```python
TypeError: __init__() got an unexpected keyword argument 'type'
```
Will this be possible only after merging? Or am I missing something here?

Can you try adding this class attribute to `BabiQa` ?
```python
BUILDER_CONFIG_CLASS = BabiQaConfig
```
This should fix the TypeError issue you got

Hi @lhoestq

I have added the changes. Only the "qa1" task for each category is included. Also, I haven't removed the size categories and other description because I think it will still be useful. I have updated the line in README showing the example.

Thanks,
Gunjan

- **Name:** *The (20) QA bAbI tasks*
- **Description:** *The (20) QA bAbI tasks are a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human. The aim is to classify these tasks into skill sets,so that researchers can identify (and then rectify) the failings of their systems.*
- **Paper:** [arXiv](https://arxiv.org/pdf/1502.05698.pdf)
- **Data:** [Facebook Research Page](https://research.fb.com/downloads/babi/)
- **Motivation:** This is a unique dataset with story-based Question Answering. It  is a part of the `bAbI` project by Facebook Research.

**Note**: I have currently added all the 160 configs. If this seems impractical, I can keep only a few. While each `dummy_data.zip` weighs a few KBs, overall it is around 1.3MB for all configurations. This is problematic. Let me know what is to be done.

Thanks :)

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


Timit_asr dataset repeats examples

Hi,

this was fixed by #1995, so you can wait for the next release or install the package directly from the master branch with the following command:
```bash
pip install git+https://github.com/huggingface/datasets
```

Summary

When loading timit_asr dataset on datasets 1.4+, every row in the dataset is the same
Steps to reproduce

As an example, on this code there is the text from the training part:

Code snippet:
```
from datasets import load_dataset, load_metric

timit = load_dataset("timit_asr")
timit['train']['text']
#['Would such an act of refusal be useful?',
# 'Would such an act of refusal be useful?',
# 'Would such an act of refusal be useful?',
# 'Would such an act of refusal be useful?',
# 'Would such an act of refusal be useful?',
# 'Would such an act of refusal be useful?',
```
The same behavior happens for other columns

Expected behavior:

Different info on the actual timit_asr dataset

Actual behavior:

When loading timit_asr dataset on datasets 1.4+, every row in the dataset is the same. I've checked datasets 1.3 and the rows are different
Debug info

    Streamlit version: (get it with $ streamlit version)
    Python version: Python 3.6.12
    Using Conda? PipEnv? PyEnv? Pex? Using pip
    OS version: Centos-release-7-9.2009.1.el7.centos.x86_64

Additional information

You can check the same behavior on https://huggingface.co/datasets/viewer/?dataset=timit_asr


Add MDD Dataset

Hi @lhoestq,

I have added changes from review.

- **Name:** *MDD Dataset*
- **Description:** The Movie Dialog dataset (MDD) is designed to measure how well models can perform at goal and non-goal orientated dialog centered around the topic of movies (question answering, recommendation and discussion), from various movie reviews sources such as MovieLens and OMDb.
- **Paper:** [arXiv](https://arxiv.org/pdf/1511.06931.pdf)
- **Data:**  https://research.fb.com/downloads/babi/
- **Motivation:** This is one of the popular dialog datasets, a part of Facebook Research's "bAbI project".

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

**Note**: I haven't included the following from the data files: `entities` (the file containing list of all entities in the first three subtasks), `dictionary`(the dictionary of words they use in their models), `movie_kb`(contains the knowledge base of information about the movies, actors and other entities that are mentioned in the dialogs). Please let me know if those are needed, and if yes, should I make separate configurations for them?


Build custom dataset to fine-tune Wav2Vec2

dataset request

Requesting to add a new dataset

@lhoestq - We could simply use the "general" json dataset for this no?

Sure you can use the json loader
```python
data_files = {"train": "path/to/your/train_data.json", "test": "path/to/your/test_data.json"}
train_dataset = load_dataset("json", data_files=data_files, split="train")
test_dataset = load_dataset("json", data_files=data_files, split="test")
```

You just need to make sure that the data contain the paths to the audio files.
If not, feel free to use `.map()` to add them.

Many thanks! that was what I was looking for.

Thank you for your recent tutorial on how to finetune Wav2Vec2 on a custom dataset. The example you gave here (https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) was on the CommonVoice dataset. However, what if I want to load my own dataset?  I have a manifest (transcript and their audio files) in a JSON file.


Fix text-classification tags

There are different tags for text classification right now: `text-classification` and `text_classification`:
![image](https://user-images.githubusercontent.com/29076344/111042457-856bdf00-8463-11eb-93c9-50a30106a1a1.png).

This PR fixes it.


github is not always available - probably need a back up

Yesterday morning github wasn't working:

```
:/tmp$ wget https://raw.githubusercontent.com/huggingface/datasets/1.4.1/metrics/sacrebleu/sacrebleu.py--2021-03-12 18:35:59-- https://raw.githubusercontent.com/huggingface/datasets/1.4.1/metrics/sacrebleu/sacrebleu.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2021-03-12 18:36:11 ERROR 500: Internal Server Error.
```

Suggestion: have a failover system and replicate the data on another system and reach there if gh isn't reachable? perhaps gh can be a master and the replicate a slave - so there is only one true source.


Multilingual dIalogAct benchMark (miam)

Hello. All aforementioned changes have been made. I've also re-run black on miam.py. :-)

I will run isort again. Hopefully it resolves the current check_code_quality test failure.

Once the review period is over, feel free to open a PR to add all the missing information ;)

Hi! I will follow up right now with one more pull request as I have new anonymous citation information to include.

My collaborators (@EmileChapuis, @PierreColombo) and I within the Affective Computing team at Telecom Paris would like to anonymously publish the miam dataset. It is assocated with a publication currently under review. We will update the dataset with full citations once the review period is over.


add_faisis_index  gets very slow when doing it interatively

I think faiss automatically sets the number of threads to use to build the index.
Can you check how many CPU cores are being used when you build the index in `use_own_knowleldge_dataset` as compared to this script ? Are there other programs running (maybe for rank>0) ?

Hi,
 I am running the add_faiss_index during the training process of the RAG from the master process (rank 0). But at the exact moment, I do not run any other process since I do it in every 5000 training steps.

 I think what you say is correct. It depends on the number of CPU cores. I did an experiment to compare the time taken to finish the add_faiss_index process on use_own_knowleldge_dataset.py vs the training loop thing.  The training loop thing takes 40 mins more. It might be natural right?

 at the moment it uses around 40 cores of a 96 core machine (I am fine-tuning the entire process).

Can you try to set the number of threads manually ?
If you set the same number of threads for both the `use_own_knowledge_dataset.py` and RAG training, it should take the same amount of time.
You can see how to set the number of thread in the faiss wiki: https://github.com/facebookresearch/faiss/wiki/Threads-and-asynchronous-calls

Ok, I will report the details too soon. I am the first one on the list and currently add_index being computed for the 3rd time in the loop. Actually seems like the time is taken to complete each interaction is the same, but around 1 hour more compared to running it without the training loop. A the moment this takes 5hrs and 30 mins.  If there is any way to faster the process, an end-to-end rag will be perfect. So I will also try out with different thread numbers too.

![image](https://user-images.githubusercontent.com/16892570/111453464-798c5f80-8778-11eb-86d0-19d212f58e38.png)

@lhoestq  on a different note, I read about using Faiss-GPU, but the documentation says we should use it when the dataset has the ability to fit into the GPU memory. Although this might work, in the long-term this is not that practical for me.

https://github.com/matsui528/faiss_tips

@lhoestq

Hi,  I executed the **use_own_dataset.py** script independently and ask a few of my friends to run their programs in the HPC machine at the same time.

 Once there are so many other processes are running the add_index function gets slows down naturally.  So basically the speed of the add_index depends entirely on the number of CPU processes.  Then I set the number of threads as you have mentioned and got actually the same time for RAG training and independat running.  So you are correct! :)

 Then I added this [issue in Faiss repostiary](https://github.com/facebookresearch/faiss/issues/1767). I got an answer saying our current **IndexHNSWFlat** can get slow for 30 million vectors and it would be better to use alternatives. What do you think?

It's a matter of tradeoffs.
HSNW is fast at query time but takes some time to build.
A flat index is flat to build but is "slow" at query time.
An IVF index is probably a good choice for you: fast building and fast queries (but still slower queries than HSNW).

Note that for an IVF index you would need to have an `nprobe` parameter (number of cells to visit for one query, there are `nlist` in total) that is not too small in order to have good retrieval accuracy, but not too big otherwise the queries will take too much time. From the faiss documentation:
> The nprobe parameter is always a way of adjusting the tradeoff between speed and accuracy of the result. Setting nprobe = nlist gives the same result as the brute-force search (but slower).

From my experience with indexes on DPR embeddings, setting nprobe around 1/4 of nlist gives really good retrieval accuracy and there's no need to have a value higher than that (or you would need to brute-force in order to see a difference).

@lhoestq

Thanks a lot for sharing all this prior knowledge.

Just asking what would be a good nlist of parameters for 30 million embeddings?

When IVF is used alone, nlist should be between `4*sqrt(n)` and `16*sqrt(n)`.
For more details take a look at [this section of the Faiss wiki](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index#how-big-is-the-dataset)

Thanks a lot. I was lost with calling the index from class and using faiss_index_factory.

@lhoestq  Thanks a lot for the help you have given to solve this issue. As per my experiments, IVF index suits well for my case and it is a lot faster. The use of this can make the entire RAG end-to-end trainable lot faster.  So I will close this issue. Will do the final PR soon.

As the below code suggests, I want to run add_faisis_index in every nth interaction from the training loop. I have 7.2 million documents. Usually, it takes 2.5 hours (if I run an as a separate process similar to the script given in rag/use_own_knowleldge_dataset.py). Now, this takes usually 5hrs. Is this normal?   Any way to make this process faster?

@lhoestq

```
   def training_step(self, batch, batch_idx) -> Dict:

        if (not batch_idx==0) and (batch_idx%5==0):

            print("******************************************************")
            ctx_encoder=self.trainer.model.module.module.model.rag.ctx_encoder
            model_copy =type(ctx_encoder)(self.config_dpr) # get a new instance  #this will be load in the CPU
            model_copy.load_state_dict(ctx_encoder.state_dict()) # copy weights and stuff

            list_of_gpus = ['cuda:2','cuda:3']
            c_dir='/custom/cache/dir'

            kb_dataset = load_dataset("csv", data_files=[self.custom_config.csv_path], split="train", delimiter="\t", column_names=["title", "text"],cache_dir=c_dir)

            print(kb_dataset)

            n=len(list_of_gpus) #nunber of dedicated GPUs
            kb_list=[kb_dataset.shard(n, i, contiguous=True) for i in range(n)]

            #kb_dataset.save_to_disk('/hpc/gsir059/MY-Test/RAY/transformers/examples/research_projects/rag/haha-dir')

            print(self.trainer.global_rank)
            dataset_shards = self.re_encode_kb(model_copy.to(device=list_of_gpus[self.trainer.global_rank]),kb_list[self.trainer.global_rank])
            output = [None for _ in list_of_gpus]

            #self.trainer.accelerator_connector.accelerator.barrier("embedding_process")
            dist.all_gather_object(output, dataset_shards)

            #This creation and re-initlaization of the new index
            if (self.trainer.global_rank==0):  #saving will be done in the main process

                combined_dataset = concatenate_datasets(output)

                passages_path =self.config.passages_path

                logger.info("saving the dataset with  ")
                #combined_dataset.save_to_disk('/hpc/gsir059/MY-Test/RAY/transformers/examples/research_projects/rag/MY-Passage')
                combined_dataset.save_to_disk(passages_path)
                logger.info("Add faiss index to the dataset that consist of embeddings")

                embedding_dataset=combined_dataset
                index = faiss.IndexHNSWFlat(768, 128, faiss.METRIC_INNER_PRODUCT)
                embedding_dataset.add_faiss_index("embeddings", custom_index=index)

                embedding_dataset.get_index("embeddings").save(self.config.index_path)


Preserve column ordering in Dataset.rename_column

Not sure why CI isn't triggered.

@lhoestq Can you please help me with this?

I don't know how to trigger it manually, but an empty commit should do the job

Currently `Dataset.rename_column` doesn't necessarily preserve the order of the columns:
```python
>>> from datasets import Dataset
>>> d = Dataset.from_dict({'sentences': ["s1", "s2"], 'label': [0, 1]})
>>> d
Dataset({
    features: ['sentences', 'label'],
    num_rows: 2
})
>>> d.rename_column('sentences', 'text')
Dataset({
    features: ['label', 'text'],
    num_rows: 2
})
```
This PR fixes this.


Add CBT dataset

Hi @lhoestq,

I have added changes from the review.

This PR adds the [CBT Dataset](https://arxiv.org/abs/1511.02301).

Note that I have also added the `raw` dataset as a separate configuration. I couldn't find a suitable "task" for it in YAML tags.

The dummy files have one example each, as the examples are slightly big. For `raw` dataset, I just used top few lines, because they are entire books and would take up a lot of space.

Let me know in case of any issues.


Support pickle protocol for dataset splits defined as ReadInstruction

@lhoestq But we don't perform conversion to a `NamedSplit` if `_split` is not a string which means it **will** be a `ReadInstruction` after reloading.

Yes right ! I read it wrong.
Perfect then

Fixes #2022 (+ some style fixes)


Fix arrow memory checks issue in tests

The tests currently fail on `master` because the arrow memory verification doesn't return the expected memory evolution when loading an arrow table in memory.
From my experiments, the tests fail only when the full test suite is ran.
This made me think that maybe some arrow objects from other tests were not freeing their memory until they do and cause the memory verifications to fail in other tests.

Collecting the garbage collector before checking the arrow memory usage seems to fix this issue.
I added a context manager `assert_arrow_memory_increases` that we can use in tests and that deals with the gc.


Doc2dial update data_infos and data_loaders


ValueError: datasets' indices [1] come from memory and datasets' indices [0] come from disk

Hi ! To help me understand the situation, can you print the values of  `load_from_disk(PATH_DATA_CLS_A)['train']._indices_data_files` and `load_from_disk(PATH_DATA_CLS_B)['train']._indices_data_files` ?
They should both have a path to an arrow file

Also note that from #2025 concatenating datasets will no longer have such restrictions.

Sure, thanks for the fast reply!

For dataset A: `[{'filename': 'drive/MyDrive/data_target_task/dataset_a/train/cache-4797266bf4db1eb7.arrow'}]`
For dataset B: `[]`

No clue why for B it returns nothing. `PATH_DATA_CLS_B` is exactly the same in `save_to_disk` and `load_from_disk`... Also I can verify that the folder physically exists under 'drive/MyDrive/data_target_task/dataset_b/'

In the next release you'll be able to concatenate any kinds of dataset (either from memory or from disk).

For now I'd suggest you to flatten the indices of the A and B datasets. This will remove the indices mapping and you will be able to concatenate them. You can flatten the indices with
```python
dataset = dataset.flatten_indices()
```

Indeed this works. Not the most elegant solution, but it does the trick. Thanks a lot!

Hi there,

I am trying to concat two datasets that I've previously saved to disk via `save_to_disk()` like so (note that both are saved as `DataDict`, `PATH_DATA_CLS_*` are `Path`-objects):
```python
concatenate_datasets([load_from_disk(PATH_DATA_CLS_A)['train'], load_from_disk(PATH_DATA_CLS_B)['train']])
```
Yielding the following error:
```python
ValueError: Datasets' indices should ALL come from memory, or should ALL come from disk.
However datasets' indices [1] come from memory and datasets' indices [0] come from disk.
```
Been trying to solve this for quite some time now. Both `DataDict` have been created by reading in a `csv` via `load_dataset` and subsequently processed using the various `datasets` methods (i.e. filter, map, remove col, rename col). Can't figure out tho...

`load_from_disk(PATH_DATA_CLS_A)['train']` yields:
```python
Dataset({
    features: ['labels', 'text'],
    num_rows: 785
})
```
`load_from_disk(PATH_DATA_CLS_B)['train']` yields:
```python
Dataset({
    features: ['labels', 'text'],
    num_rows: 3341
})
```


Doc2dial rc

Added fix to handle the last turn that is a user turn.


outdated dataset_infos.json might fail verifications

Hi ! Thanks for reporting.

To update the dataset_infos.json you can run:
```
datasets-cli test ./datasets/doc2dial --all_configs --save_infos --ignore_verifications
```

Fixed by #2041, thanks again @songfeng !

The [doc2dial/dataset_infos.json](https://github.com/huggingface/datasets/blob/master/datasets/doc2dial/dataset_infos.json) is outdated. It would fail data_loader when verifying download checksum etc..

Could you please update this file or point me how to update this file?

Thank you.


Fix: Wikipedia - save memory by replacing root.clear with elem.clear

The error you got is minor and appeared in the last version of pyarrow, we'll fix the CI to take this into account. You can ignore it

see: https://github.com/huggingface/datasets/issues/2031

What I did:
- replace root.clear with elem.clear
- remove lines to get root element
- $ make style
- $ make test
  - some tests required some pip packages, I installed them.

test results on origin/master and my branch are same. I think it's not related on my modification, isn't it?
```
==================================================================================== short test summary info ====================================================================================
FAILED tests/test_arrow_writer.py::TypedSequenceTest::test_catch_overflow - AssertionError: OverflowError not raised
============================================================= 1 failed, 2332 passed, 5138 skipped, 70 warnings in 91.75s (0:01:31) ==============================================================
make: *** [Makefile:19: test] Error 1

```

Is there anything else I should do?


Cannot load wikitext

when I execute these codes
```
>>> from datasets import load_dataset
>>> test_dataset = load_dataset("wikitext")
```

I got an error,any help?

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 487, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/wikitext/wikitext.py
```


wiki40b/wikipedia for almost all languages cannot be downloaded

Dear @lhoestq for wikipedia dataset I also get the same error, I greatly appreciate if you could have a look into this dataset as well. Below please find the command to reproduce the error:

```
dataset = load_dataset("wikipedia", "20200501.bg")
print(dataset)
```

Your library is my only chance to be able training  the models at scale and I am grateful for your help.

Hi @dorost1234,
Try installing this library first, `pip install 'apache-beam[gcp]' --use-feature=2020-resolver` followed by loading dataset like this using beam runner.

`dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')`

 I also read in error stack trace that:

> Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc.

Worked perfectly fine after this (Ignore these warnings)

![image](https://user-images.githubusercontent.com/19718818/110908410-c7e2ce00-8334-11eb-8d10-7354359e9ec3.png)

For wikipedia dataset, looks like the files it's looking for are no longer available. For `bg`, I checked [here](https://dumps.wikimedia.org/bgwiki/). For this I think `dataset_infos.json` for this dataset has to made again? You'll have to load this dataset also using beam runner.

Hello @dorost1234,

Indeed, Wikipedia datasets need a lot of preprocessing and this is done using Apache Beam. That is the reason why it is required that you install Apache Beam in order to preform this preprocessing.

For some specific default parameters (English Wikipedia), Hugging Face has already preprocessed the dataset for you (and it is stored in the cloud). That is the reason why you do not get the error for English: the preprocessing is already done by HF and you just get the preprocessed dataset; Apache Beam is not required in that case.

Hi
I really appreciate if huggingface can kindly provide preprocessed
datasets, processing these datasets require sufficiently large resources
and I do not have unfortunately access to, and perhaps many others too.
thanks

On Fri, Mar 12, 2021 at 9:04 AM Albert Villanova del Moral <
***@***.***> wrote:

> Hello @dorost1234 <https://github.com/dorost1234>,
>
> Indeed, Wikipedia datasets need a lot of preprocessing and this is done
> using Apache Beam. That is the reason why it is required that you install
> Apache Beam in order to preform this preprocessing.
>
> For some specific default parameters (English Wikipedia), Hugging Face has
> already preprocessed the dataset for you (and it is stored in the cloud).
> That is the reason why you do not get the error for English: the
> preprocessing is already done by HF and you just get the preprocessed
> dataset; Apache Beam is not required in that case.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2035#issuecomment-797310899>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMXACFQZAGMK4VGXRETTDHDI3ANCNFSM4ZA5R2UA>
> .
>

Hi everyone
thanks for the helpful pointers, I did it as @bhavitvyamalik suggested, for me this freezes on this command for several hours,

`Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /users/dara/cache/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
`

Do you know how long this takes? Any specific requirements the machine should have? like very large memory or so? @lhoestq

thanks

HI @dorost1234,
The dataset size is 631.84 MiB so depending on your internet speed it'll take some time. You can monitor your internet speed meanwhile to see if it's downloading the dataset or not (use `nload` if you're using linux/mac to monitor the same). In my case it took around 3-4 mins. Since they haven't used `download_and_extract` here that's why there's no download progress bar.

Hi
thanks, my internet speed should be good, but this really freezes for me, this is how I try to get this dataset:

`from datasets import load_dataset
dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')`

the output I see if different also from what you see after writing  this command:

`Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /users/dara/cache/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...`

do you have any idea why it might get freezed? anything I am missing @lhoestq @bhavitvyamalik. Do I need maybe to set anything special for apache-beam?

thanks a lot

On Tue, Mar 16, 2021 at 9:03 AM Bhavitvya Malik ***@***.***>
wrote:

> HI @dorost1234 <https://github.com/dorost1234>,
> The dataset size is 631.84 MiB so depending on your internet speed it'll
> take some time. You can monitor your internet speed meanwhile to see if
> it's downloading the dataset or not (use nload if you're using linux/mac
> to monitor the same). In my case it took around 3-4 mins. Since they
> haven't used download_and_extract here that's why there's no download
> progress bar.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2035#issuecomment-800044303>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMQIHNNLM2LGG6QKZ73TD4GDJANCNFSM4ZA5R2UA>
> .
>

I tried this on another machine (followed the same procedure I've mentioned above). This is what it shows (during the freeze period) for me:
```
>>> dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')
Downloading: 5.26kB [00:00, 1.23MB/s]
Downloading: 1.40kB [00:00, 327kB/s]
Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
WARNING:apache_beam.internal.gcp.auth:Unable to find default credentials to use: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Connecting anonymously.
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
```
After around 10 minutes, here's the loading of dataset:
```
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:16<00:00, 16.42s/sources]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.12sources/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.14sources/s]
Dataset wiki40b downloaded and prepared to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f. Subsequent calls will reuse this data.
```

Hi
I honestly also now tried on another machine and nothing shows up after
hours of waiting. Are you sure you have not set any specific setting? maybe
google cloud which seems it is used here, needs some credential setting?
thanks for any suggestions on this

On Tue, Mar 16, 2021 at 10:02 AM Bhavitvya Malik ***@***.***>
wrote:

> I tried this on another machine (followed the same procedure I've
> mentioned above). This is what it shows (during the freeze period) for me:
>
> >>> dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')
> Downloading: 5.26kB [00:00, 1.23MB/s]
> Downloading: 1.40kB [00:00, 327kB/s]
> Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
> WARNING:apache_beam.internal.gcp.auth:Unable to find default credentials to use: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
> Connecting anonymously.
> WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
>
> After around 10 minutes, here's the loading of dataset:
>
> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:16<00:00, 16.42s/sources]
> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.12sources/s]
> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.14sources/s]
> Dataset wiki40b downloaded and prepared to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f. Subsequent calls will reuse this data.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2035#issuecomment-800081772>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMX6A2ZTRZUIIZVFRCDTD4NC3ANCNFSM4ZA5R2UA>
> .
>

Hi
I am trying to download the data as below:

```
from datasets import load_dataset
dataset = load_dataset("wiki40b", "cs")
print(dataset)
```

I am getting this error. @lhoestq I will be grateful if you could assist me with this error. For almost all languages except english I am getting this error.

I really need majority of languages in this dataset to be able to train my models for a deadline and your great scalable super well-written library is my only hope to train the models at scale while being low on resources.

thank you very much.

```
(fast) dara@vgne046:/user/dara/dev/codes/seq2seq$ python test_data.py
Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to temp/dara/cache_home_2/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
Traceback (most recent call last):
  File "test_data.py", line 3, in <module>
    dataset = load_dataset("wiki40b", "cs")
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/load.py", line 746, in load_dataset
    use_auth_token=use_auth_token,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/builder.py", line 579, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/builder.py", line 1105, in _download_and_prepare
    import apache_beam as beam
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/apache_beam-2.28.0-py3.7-linux-x86_64.egg/apache_beam/__init__.py", line 96, in <module>
    from apache_beam import io
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/apache_beam-2.28.0-py3.7-linux-x86_64.egg/apache_beam/io/__init__.py", line 23, in <module>
    from apache_beam.io.avroio import *
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/apache_beam-2.28.0-py3.7-linux-x86_64.egg/apache_beam/io/avroio.py", line 55, in <module>
    import avro
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/avro_python3-1.9.2.1-py3.7.egg/avro/__init__.py", line 34, in <module>
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/avro_python3-1.9.2.1-py3.7.egg/avro/__init__.py", line 30, in LoadResource
NotADirectoryError: [Errno 20] Not a directory: '/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/avro_python3-1.9.2.1-py3.7.egg/avro/VERSION.txt'
```


Fix typo

Change `ENV_XDG_CACHE_HOME ` to `XDG_CACHE_HOME `


Raise an error for outdated sacrebleu versions

The `sacrebleu` metric seem to only work for sacrecleu>=1.4.12

For example using sacrebleu==1.2.10, an error is raised (from metric/sacrebleu/sacrebleu.py):
```python
    def _compute(
        self,
        predictions,
        references,
        smooth_method="exp",
        smooth_value=None,
        force=False,
        lowercase=False,
        tokenize=scb.DEFAULT_TOKENIZER,
        use_effective_order=False,
    ):
        references_per_prediction = len(references[0])
        if any(len(refs) != references_per_prediction for refs in references):
            raise ValueError("Sacrebleu requires the same number of references for each prediction")
        transformed_references = [[refs[i] for refs in references] for i in range(references_per_prediction)]
>       output = scb.corpus_bleu(
            sys_stream=predictions,
            ref_streams=transformed_references,
            smooth_method=smooth_method,
            smooth_value=smooth_value,
            force=force,
            lowercase=lowercase,
            tokenize=tokenize,
            use_effective_order=use_effective_order,
        )

E       TypeError: corpus_bleu() got an unexpected keyword argument 'smooth_method'
/mnt/cache/modules/datasets_modules/metrics/sacrebleu/b390045b3d1dd4abf6a95c4a2a11ee3bcc2b7620b076204d0ddc353fa649fd86/sacrebleu.py:114: TypeError
```

I improved the error message when users have an outdated version of sacrebleu.
The new error message tells the user to update sacrebleu.
cc @LysandreJik


Use Arrow filtering instead of writing a new arrow file for Dataset.filter

enhancement

New feature or request

Currently the filter method reads the dataset batch by batch to write a new, filtered, arrow file on disk. Therefore all the reading + writing can take some time.

Using a mask directly on the arrow table doesn't do any read or write operation therefore it's significantly quicker.

I think there are two cases:
- if the dataset doesn't have an indices mapping, then one can simply use the arrow filtering on the main arrow table `dataset._data.filter(...)`
- if the dataset an indices mapping, then the mask should be applied on the indices mapping table `dataset._indices.filter(...)`

The indices mapping is used to map between the idx at `dataset[idx]` in `__getitem__` and the idx in the actual arrow table.

The new filter method should therefore be faster, and allow users to pass either a filtering function (that returns a boolean given an example), or directly a mask.

Feel free to discuss this idea in this thread :)

One additional note: the refactor at #2025 would make all the pickle-related stuff work directly with the arrow filtering, so that we only need to change the Dataset.filter method without having to deal with pickle.

cc @theo-m @gchhablani

related issues: #1796 #1949


wikipedia.py generator that extracts XML doesn't release memory

Hi @miyamonz
Thanks for investigating this issue, good job !
It would be awesome to integrate your fix in the library, could you open a pull request ?

I tried downloading Japanese wikipedia, but it always failed because of out of memory maybe.

I found that the generator function that extracts XML data in wikipedia.py doesn't release memory in the loop.

https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L464-L502

`root.clear()` intend to clear memory, but it doesn't.
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L490
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L494
I replaced them with `elem.clear()`, then it seems to work correctly.

here is the notebook to reproduce it.
https://gist.github.com/miyamonz/dc06117302b6e85fa51cbf46dde6bb51#file-xtract_content-ipynb


Implement Dataset from text

I am wondering why only one test of "keep_in_memory=True" fails, when there are many other tests that test the same and it happens only in pyarrow_1...

Implement `Dataset.from_text`.

Analogue to #1943, #1946.


Loading a faiss index KeyError

documentation

Improvements or additions to documentation

In your code `dataset2` doesn't contain the "embeddings" column, since it is created from the pandas DataFrame with columns "text" and "label".

Therefore when you call `dataset2[embeddings_name]`, you get a `KeyError`.

If you want the "embeddings" column back, you can create `dataset2` with
```python
dataset2 = load_from_disk(dataset_filename)
```
where `dataset_filename` is the place where you saved you dataset with the embeddings in the first place.

Ok in that case HF should fix their misleading example at https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index

I copy-pasted it here.

> When you are done with your queries you can save your index on disk:
>
> ```python
> ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')
> ```
> Then reload it later:
>
> ```python
> ds = load_dataset('crime_and_punish', split='train[:100]')
> ds.load_faiss_index('embeddings', 'my_index.faiss')
> ```

Hi !

The code of the example is valid.
An index is a search engine, it's not considered a column of a dataset.
When you do `ds.load_faiss_index("embeddings", 'my_index.faiss')`, it attaches an index named "embeddings" to the dataset but it doesn't re-add the "embeddings" column. You can list the indexes of a dataset by using `ds.list_indexes()`.

If I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.
This looks misleading indeed, and we should add a note to make it more explicit that it doesn't store the column that was used to build the index.

Feel free to open a PR to suggest an improvement on the documentation if you want to contribute :)

> If I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.
Yes. I was trying to use the dataset in RAG and it complained that the dataset didn't have the right columns. No problems when loading the dataset with `load_from_disk` and then doing `load_faiss_index`

What I learned was
1. column and index are different
2. loading the index does not create a column
3. the column is not needed to be able to use the index
4. RAG needs both the embeddings column and the index

If I can come up with a way to articulate this in the right spot in the docs, I'll open a PR

I've recently been testing out RAG and DPR embeddings, and I've run into an issue that is not apparent in the documentation.

The basic steps are:

1. Create a dataset (dataset1)
2. Create an embeddings column using DPR
3. Add a faiss index to the dataset
4. Save faiss index to a file
5. Create a new dataset (dataset2) with the same text and label information as dataset1
6. Try to load the faiss index from file to dataset2
7. Get `KeyError: "Column embeddings not in the dataset"`

I've made a colab notebook that should show exactly what I did. Please switch to GPU runtime; I didn't check on CPU.

https://colab.research.google.com/drive/1X0S9ZuZ8k0ybcoei4w7so6dS_WrABmIx?usp=sharing

Ubuntu Version
VERSION="18.04.5 LTS (Bionic Beaver)"

datasets==1.4.1
faiss==1.5.3
faiss-gpu==1.7.0
torch==1.8.0+cu101
transformers==4.3.3

NVIDIA-SMI 460.56
Driver Version: 460.32.03
CUDA Version: 11.2
Tesla K80

I was basically following the steps here: https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index

I included the exact code from the documentation at the end of the notebook to show that they don't work either.


Adding PersiNLU reading-comprehension

@lhoestq I think I have addressed all your comments.

Thanks! @lhoestq  Let me know if you want me to address anything to get this merged.


Update format columns in Dataset.rename_columns

Fixes #2026