Streaming for the Json loader

A note is that I think we should add a few indicator of status (as mentioned by @stas00 in #2649), probably at the (1) downloading, (2) extracting and (3) reading steps. In particular when loading many very large files it's interesting to know a bit where we are in the process.

I tested locally, and the builtin `json` loader is 4x slower than `pyarrow.json`. Thanks for the comment @albertvillanova !

Therefore I switched back to using `pyarrow.json`, but only on the batch that is read. This way we don't have to deal with its `block_size`, and it only loads in memory one batch at a time.

It was not using `open` in the builder. Therefore `pyarrow.json.read_json` was downloading the full file to start yielding rows.

Moreover, it appeared that `pyarrow.json.read_json` was not really suited for streaming as it was downloading too much data and failing if `block_size` was not properly configured (related to #2573).

So I switched to using `open` which is extended to support reading from remote file progressively, and I removed the pyarrow json reader which was not practical.
Instead, I'm using the classical `json.loads` from the standard library.


Streaming for the Pandas loader

It was not using open in the builder. Therefore pd.read_pickle could fail when streaming from a private repo for example.

Indeed, when streaming, open is extended to support reading from remote files and handles authentication to the HF Hub


Streaming for the CSV loader

It was not using `open` in the builder. Therefore `pd.read_csv` was downloading the full file to start yielding rows.

Indeed, when streaming, `open` is extended to support reading from remote file progressively.


Inject ASR template for lj_speech dataset

1.10

Next minor release

Related to: #2565, #2633.

cc: @lewtun


Update ASR tags

1.10

Next minor release

This PR updates the ASR tags of the 5 datasets added in #2565 following the change of task categories in #2620


add image-classification task template

Thanks for adding a new task template - great work @nateraw  🚀 !

Snippet below is the tl;dr, but you can try it out directly here:

[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/005c025d41f0e48ae3d4ee61c0f20b70/image-classification-task-template-demo.ipynb)

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'labels'],
#         num_rows: 23410
#     })
# })

ds = ds.prepare_for_task('image-classification')
# DatasetDict({
#     train: Dataset({
#         features: ['image_file_path', 'labels'],
#         num_rows: 23410
#     })
# })
```


Delete extracted files when loading dataset

Sure @stas00, it is still a draft pull request. :)

Yes, I noticed it after reviewing - my apologies.

The problem with this approach is that it also deletes the downloaded files (if they need not be extracted). 😟

> The problem with this approach is that it also deletes the downloaded files (if they need not be extracted). worried

Right! These probably should not be deleted by default, but having an option for those users who are tight on disc space?

> Right! These probably should not be deleted by default, but having an option for those users who are tight on disc space?

I propose leaving that for another PR, and leave this one handling only with "extracted" files. Is it OK for you? :)

Awesome thanks !
I just have one question: what about image/audio datasets for which we store the path to the extracted file on the arrow data ?
In this case the default should be to keep the extracted files.

So for now I would just make `keep_extracted=True` by default until we have a way to separate extracted files that can be deleted and extracted files that are actual resources of the dataset.

@lhoestq, current implementation only deletes extracted "files", not extracted "directories", as it uses: `os.remove(path)`. I'm going to add a filter on files, so that this line does not throw an exception when passed a directory.

For audio datasets, the audio files are inside the extracted "directory", so they are not deleted.

I'm still more in favor of having `keep_extracted=True` by default:
- When working with a dataset, you call `load_dataset` many times. By default we want to keep objects extracted to not extract them over and over again (it can take a long time). Then once you know what you're doing and you want to optimize disk space, you can do `keep_extracted=False`. Deleting the extracted files by default is a regression that can lead to slow downs for people calling `load_dataset` many times, which is common when experimenting
- This behavior doesn't sound natural as a default behavior. In the rest of the library, things are cached and not removed unless you explicitly say do (`map` caching for example). Moreover the function in the download manager is called `download_and_extract`, not `download_and_extract_and_remove_extracted_files`

Let me know what you think !

I think the main issue is that after doing some work users typically move on to other datasets and the amount of disc space used keeps on growing. So your logic is very sound and perhaps what's really needed is a cleansweep function that can go through **all** datasets and clean them up to the desired degree:

- delete all extracted files
- delete all sources
- delete all caches
- delete all caches that haven't been accessed  in 6 months
- delete completely old datasets that haven't been accessed  in 6 months
- more?

So a user can launch a little application, choose what they want to clean up and voila they have just freed up a huge amount of disc space. Makes me think of Ubuntu Tweak's Janitor app - very useful.

At the moment, this process of linting is very daunting and error-prone, especially due to all those dirs/files with hash names.

@stas00 I've had the same idea. Instead of the full-fledged app, a simpler approach would be to add a new command to the CLI.

oh, CLI would be perfect. I didn't mean to request a GUI-one specifically, was just using it as an example.

One could even do a crontab to delete old datasets that haven't been accesses in X months.

@lhoestq I totally agree with you. I'm addressing that change.

@stas00, @mariosasko, that could eventually be addressed in another pull request. The objective of this PR is:
- add an option to pass to `load_dataset`, so that extracted files are deleted
- do this deletion file per file, once the file has been already used to generate the cache Arrow file

I also like the idea of having a CLI tool to help users clean their cache and save disk space, good idea !

Close #2481, close #2604, close #2591.

cc: @stas00, @thomwolf, @BirgerMoell


Progress bars are not properly rendered in Jupyter notebook

bug

Something isn't working

To add my experience when trying to debug this issue:

Seems like previously the workaround given [here](https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308) worked around this issue. But with the latest version of jupyter/tqdm I still get terminal warnings that IPython tried to send a message from a forked process.

Hi @mludv, thanks for the hint!!! :)

We will definitely take it into account to try to fix this issue... It seems somehow related to `multiprocessing` and `tqdm`...

## Describe the bug
The progress bars are not Jupyter widgets; regular progress bars appear (like in a terminal).

## Steps to reproduce the bug
```python
ds.map(tokenize, num_proc=10)
```

## Expected results
Jupyter widgets displaying the progress bars.

## Actual results
Simple plane progress bars.

cc: Reported by @thomwolf


Load datasets from the Hub without requiring a dataset script

This is so cool, let us know if we can help with anything on the hub side (@Pierrci @elishowk) 🎉

As a user I would like to be able to upload my csv/json/text/parquet/etc. files in a dataset repository on the Hugging Face Hub and be able to load this dataset with `load_dataset` without having to implement a dataset script.

Moreover I would like to be able to specify which file goes into which split using the `data_files` argument.

This feature should be compatible with private repositories and dataset streaming.

This can be implemented by checking the extension of the files in the dataset repository and then by using the right dataset builder that is already packaged in the library (csv/json/text/parquet/etc.)


Use ETag of remote data files

1.10

Next minor release

Use ETag of remote data files to create config ID.

Related to #2616.


Minor fix tests with Windows paths

1.10

Next minor release

Minor fix tests with Windows paths.


Use correct logger in metrics.py

1.10

Next minor release

Fixes #2624


can't set verbosity for `metric.py`

Something isn't working

Thanks @thomas-happify for reporting and thanks @mariosasko for the fix.

## Describe the bug
```
[2021-07-10 20:13:11,528][datasets.utils.filelock][INFO] - Lock 139705371374976 acquired on /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.lock
[2021-07-10 20:13:11,529][datasets.arrow_writer][INFO] - Done writing 32 examples in 6100 bytes /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.
[2021-07-10 20:13:11,531][datasets.arrow_dataset][INFO] - Set __getitem__(key) output type to python objects for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
[2021-07-10 20:13:11,543][/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric.py][INFO] - Removing /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow
```
As you can see, `datasets` logging come from different places.
`filelock`, `arrow_writer` & `arrow_dataset` comes from `datasets.*` which are expected
However, `metric.py` logging comes from `/conda/envs/myenv/lib/python3.8/site-packages/datasets/`

So when setting `datasets.utils.logging.set_verbosity_error()`,  it still logs the last message which is annoying during evaluation.

I had to do
```
logging.getLogger("/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric").setLevel(logging.ERROR)
```
to fully mute these messages

## Expected results
it shouldn't log these messages when setting `datasets.utils.logging.set_verbosity_error()`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: tried both 1.8.0 & 1.9.0
- Platform: Ubuntu 18.04.5 LTS
- Python version: 3.8.10
- PyArrow version: 3.0.0


[Metrics] added wiki_split metrics

Looks all good to me thanks :)
Just did some minor corrections in the docstring

Fixes: #2606

This pull request adds combine metrics for the wikisplit or English sentence split task

Reviewer: @patrickvonplaten


Integration with AugLy

enhancement

New feature or request

Hi,

you can define your own custom formatting with `Dataset.set_transform()` and then run the tokenizer with the batches of augmented data as follows:
```python
dset = load_dataset("imdb", split="train")  # Let's say we are working with the IMDB dataset
dset.set_transform(lambda ex: {"text": augly_text_augmentation(ex["text"])}, columns="text", output_all_columns=True)
dataloader = torch.utils.data.DataLoader(dset, batch_size=32)
for epoch in range(5):
    for batch in dataloader:
       tokenizer_output = tokenizer(batch.pop("text"), padding=True, truncation=True, return_tensors="pt")
       batch.update(tokenizer_output)
       output = model(**batch)
       ...
```

**Is your feature request related to a problem? Please describe.**
Facebook recently launched a library, [AugLy](https://github.com/facebookresearch/AugLy) , that has a unified API for augmentations for image, video and text.

It would be pretty exciting to have it hooked up to HF libraries so that we can make NLP models robust to misspellings or to punctuation, or emojis etc. Plus, with Transformers supporting more CV use cases, having augmentations support becomes crucial.

**Describe the solution you'd like**
The biggest difference between augmentations and preprocessing is that preprocessing happens only once, but you are running augmentations once per epoch. AugLy operates on text directly, so this breaks the typical workflow where we would run the tokenizer once, set format to pt tensors and be ready for the Dataloader.

**Describe alternatives you've considered**

One possible way of implementing these is to make a custom Dataset class where getitem(i) runs the augmentation and the tokenizer every time, though this would slow training down considerably given we wouldn't even run the tokenizer in batches.


Use prefix to allow exceed Windows MAX_PATH

Does this mean the `FileNotFoundError` that avoids infinite loop can be removed?

 > Or maybe we could leave it in case a relative path exceeds the MAX_PATH limit?

What about converting relative paths to absolute?

Nice ! Have you had a chance to test it on a windows machine with the max path limit enabled ? Afaik the CI doesn't have the path limit

Sure @lhoestq: I've tested on my machine... And this fixes most of the tests... 😅

By using this prefix, you can exceed the Windows MAX_PATH limit.

See: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#win32-file-namespaces

Related to #2524, #2220.


Add speech processing tasks

> Are there any `task_categories:automatic-speech-recognition` dataset for which we should update the tags ?

Yes there's a few - I'll fix them tomorrow :)

This PR replaces the `automatic-speech-recognition` task category with a broader `speech-processing` category.

The tasks associated with this category are derived from the [SUPERB benchmark](https://arxiv.org/abs/2105.01051), and ASR is included in this set.


Add ASR task for SUPERB

1.10

Next minor release

Wait until #2620 is merged before pushing the README tags in this PR

> Thanks!
>
> One question: aren't you adding `task_templates` to the `_info` method (and to the `dataset_infos.json`?

great catch! i've now added the asr task template (along with a mapping from superb task -> template) and updated the `dataset_infos.json` :)

> Good!
>
> I have a suggested refactoring... Tell me what you think! :)

your approach is much more elegant - i've included your suggestions 🙏

This PR starts building up the SUPERB benchmark by including the ASR task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/v0.2.0/downstream#asr-automatic-speech-recognition).

Usage:

```python
from datasets import load_dataset

asr = load_dataset("superb", "asr")
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 28539
#     })
#     validation: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 2703
#     })
#     test: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 2620
#     })
# })
```

I've used the GLUE benchmark as a guide for filling out the README.

To move fast during the evaluation PoC I propose to merge one task at a time, so we can continue building the training / evaluation framework in parallel.

Note: codewise this PR is ready for review - I'll add the missing YAML tags once #2620 is merged :)


`filelock.py` Error

bug

Something isn't working

Hi @liyucheng09, thanks for reporting.

Apparently this issue has to do with your environment setup. One question: is your data in an NFS share? Some people have reported this error when using `fcntl` to write to an NFS share... If this is the case, then it might be that your NFS just may not be set up to provide file locks. You should ask your system administrator, or try these commands in the terminal:
```shell
sudo systemctl enable rpc-statd
sudo systemctl start rpc-statd
```

## Describe the bug

It seems that the `filelock.py` went error.

```
>>> ds=load_dataset('xsum')

^CTraceback (most recent call last):
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 402, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
OSError: [Errno 37] No locks available
```

According to error log, it is OSError, but there is an `except` in the `_acquire` function.

```
    def _acquire(self):
        open_mode = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_TRUNC
        try:
            fd = os.open(self._lock_file, open_mode)
        except (IOError, OSError):
            pass
        else:
            self._lock_file_fd = fd
        return None
```

I don't know why it stucked rather than `pass` directly.

I am not quite familiar with filelock operation, so any help is highly appriciated.

## Steps to reproduce the bug
```python

ds = load_dataset('xsum')
```

## Expected results
A clear and concise description of the expected results.

## Actual results
```
>>> ds=load_dataset('xsum')

^CTraceback (most recent call last):
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 402, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
OSError: [Errno 37] No locks available

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/load.py", line 818, in load_dataset
    use_auth_token=use_auth_token,
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/load.py", line 470, in prepare_module
    with FileLock(lock_path):
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 323, in __enter__
    self.acquire()
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 272, in acquire
    self._acquire()
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 402, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
KeyboardInterrupt
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.0
- Platform: Linux-4.15.0-135-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.13
- PyArrow version: 4.0.1


Fix missing EOL issue in to_json for old versions of pandas

1.10

Next minor release

Some versions of pandas don't add an EOL at the end of the output of `to_json`.
Therefore users could end up having two samples in the same line

Close https://github.com/huggingface/datasets/issues/2615


Support remote data files

enhancement

New feature or request

1.10

Next minor release

> @lhoestq maybe we could also use (if available) the ETag of the remote file in `create_config_id`?

Sure ! We can get the ETag with
```python
headers = get_authentication_headers_for_url(url, use_auth_token=use_auth_token)  # auth for private repos
etag = http_head(url, headers=headers).headers.get("ETag")
```

Since the computation of the `config_id` is done in the `DatasetBuilder.__init__`, then this means that we need to add a new parameter `use_auth_token` in `DatasetBuilder.__init__`

Does that sound good ? We can add this in a following PR

Add support for (streaming) remote data files:

```python
data_files = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{relative_file_path}"
ds = load_dataset("json", split="train", data_files=data_files, streaming=True)
```

cc: @thomwolf


Jsonlines export error

bug

Something isn't working

Thanks for reporting @TevenLeScao! I'm having a look...

(not sure what just happened on the assignations sorry)

For some reason this happens (both `datasets` version are on master) only on Python 3.6 and not Python 3.8.

@TevenLeScao we are using `pandas` to serialize the dataset to JSON Lines. So it must be due to pandas. Could you please check the pandas version causing the issue?

@TevenLeScao I have just checked it: this was a bug in `pandas` and it was fixed in version 1.2: https://github.com/pandas-dev/pandas/pull/36898

Well I though it was me who has taken on this issue... 😅

Sorry, I was also talking to teven offline so I already had the PR ready before noticing x)

I was also already working in my PR... Nevermind. Next time we should pay attention if there is somebody (self-)assigned to an issue and if he/she is still working on it before overtaking it... 😄

The fix is available on `master` @TevenLeScao , thanks for reporting

## Describe the bug
When exporting large datasets in jsonlines (c4 in my case) the created file has an error every 9999 lines: the 9999th and 10000th are concatenated, thus breaking the jsonlines format. This sounds like it is related to batching, which is by 10000 by default

## Steps to reproduce the bug
This what I'm running:

in python:

```
from datasets import load_dataset
ptb = load_dataset("ptb_text_only")
ptb["train"].to_json("ptb.jsonl")
```

then out of python:

```
head -10000 ptb.jsonl
```

## Expected results
Properly separated lines

## Actual results
The last line is a concatenation of two lines

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.1.dev0
- Platform: Linux-5.4.0-1046-gcp-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyArrow version: 4.0.1


Convert numpy scalar to python float in Pearsonr output

1.10

Next minor release

Following of https://github.com/huggingface/datasets/pull/2612


Use ndarray.item instead of ndarray.tolist

1.10

Next minor release

This PR follows up on #2612 to use `numpy.ndarray.item` instead of `numpy.ndarray.tolist` as the latter is somewhat confusing to the developer (even though it works).

Judging from the `numpy` docs, `ndarray.item` is closer to what we want: https://numpy.org/doc/stable/reference/generated/numpy.ndarray.item.html#numpy-ndarray-item

PS. Sorry for the duplicate work here. I should have read the numpy docs more carefully in #2612


Return Python float instead of numpy.float64 in sklearn metrics

1.10

Next minor release

I opened an issue on the `sklearn` repo to understand why `numpy.float64` is the default: https://github.com/scikit-learn/scikit-learn/discussions/20490

It could be surprising at first to use `tolist()` on numpy scalars but it works ^^

did the same for Pearsonr here: https://github.com/huggingface/datasets/pull/2614

This PR converts the return type of all `sklearn` metrics to be Python `float` instead of `numpy.float64`.

The reason behind this is that our Hub evaluation framework relies on converting benchmark-specific metrics to YAML ([example](https://huggingface.co/datasets/autonlp/autonlp-benchmark-raft-neelalex__raft-test-neelalex__raft-predictions-3/blob/main/README.md#L11)) and the `numpy.float64` format produces garbage like:

```python
import yaml
from datasets import load_metric

metric = load_metric("accuracy")
score = metric.compute(predictions=[0,1], references=[0,1])
print(yaml.dump(score["accuracy"])) # output below
# !!python/object/apply:numpy.core.multiarray.scalar
# - !!python/object/apply:numpy.dtype
#   args:
#   - f8
#   - false
#   - true
#   state: !!python/tuple
#   - 3
#   - <
#   - null
#   - null
#   - null
#   - -1
#   - -1
#   - 0
# - !!binary |
#   AAAAAAAA8D8=
```


More consistent naming

As per @stas00's suggestion in #2500, this PR inserts a space between the logo and the lib name (`🤗Datasets` -> `🤗 Datasets`) for consistency with the Transformers lib. Additionally, more consistent names are used for Datasets Hub, etc.


Add missing WikiANN language tags

1.10

Next minor release

Add missing language tags for WikiANN datasets.


Fix potential DuplicatedKeysError

1.10

Next minor release

Fix potential DiplicatedKeysError by ensuring keys are unique.

We should promote as a good practice, that the keys should be programmatically generated as unique, instead of read from data (which might be not unique).


Support streaming JSON files

1.10

Next minor release

Use open in JSON dataset builder, so that it can be patched with xopen for streaming.

Close #2607.


Streaming local gzip compressed JSON line files is not working

Something isn't working

Updating to pyarrow-4.0.1 didn't fix the issue

Here is an exemple dataset with 2 of these compressed JSON files: https://huggingface.co/datasets/thomwolf/github-python

Hi @thomwolf, thanks for reporting.

It seems this might be due to the fact that the JSON Dataset builder uses `pyarrow.json` (`paj.read_json`) to read the data without using the Python standard `open(file,...` (which is the one patched with `xopen` to work in streaming mode).

This has to be fixed.

Sorry for reopening this, but I'm having the same issue as @thomwolf when streaming a gzipped JSON Lines file from the hub. Or is that just not possible by definition?
I installed `datasets`in editable mode from source (so probably includes the fix from #2608 ?):
```
>>> datasets.__version__
'1.9.1.dev0'
```

```
>>> msmarco = datasets.load_dataset("webis/msmarco", "corpus", streaming=True)
Using custom data configuration corpus-174d3b7155eb68db
>>> msmarco_iter = iter(msmarco['train'])
>>> print(next(msmarco_iter))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/media/ssd/TREC/msmarco/datasets/src/datasets/iterable_dataset.py", line 338, in __iter__
    for key, example in self._iter():
  File "/media/ssd/TREC/msmarco/datasets/src/datasets/iterable_dataset.py", line 335, in _iter
    yield from ex_iterable
  File "/media/ssd/TREC/msmarco/datasets/src/datasets/iterable_dataset.py", line 78, in __iter__
    for key, example in self.generate_examples_fn(**self.kwargs):
  File "/home/christopher/.cache/huggingface/modules/datasets_modules/datasets/msmarco/eb63dff8d83107168e973c7a655a6082d37e08d71b4ac39a0afada479c138745/msmarco.py", line 96, in _generate_examples
    with gzip.open(file, "rt", encoding="utf-8") as f:
  File "/usr/lib/python3.6/gzip.py", line 53, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/usr/lib/python3.6/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'https://huggingface.co/datasets/webis/msmarco/resolve/main/msmarco_doc_00.gz'
```

Loading the dataset without streaming set to True, works fine.

Hi ! To make the streaming work, we extend `open` in the dataset builder to work with urls.

Therefore you just need to use `open` before using `gzip.open`:
```diff
- with gzip.open(file, "rt", encoding="utf-8") as f:
+ with gzip.open(open(file, "rb"), "rt", encoding="utf-8") as f:
```

You can see that it is the case for oscar.py and c4.py for example:

https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/datasets/oscar/oscar.py#L358-L358

https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/datasets/c4/c4.py#L88-L88

@lhoestq Sorry I missed that. Thank you Quentin!

## Describe the bug
Using streaming to iterate on local gzip compressed JSON files raise a file not exist error

## Steps to reproduce the bug
```python
from datasets import load_dataset

streamed_dataset = load_dataset('json', split='train', data_files=data_files, streaming=True)

next(iter(streamed_dataset))
```

## Actual results
```
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-27a664e29784> in <module>
----> 1 next(iter(streamed_dataset))

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in __iter__(self)
    336
    337     def __iter__(self):
--> 338         for key, example in self._iter():
    339             if self.features:
    340                 # we encode the example for ClassLabel feature types for example

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in _iter(self)
    333         else:
    334             ex_iterable = self._ex_iterable
--> 335         yield from ex_iterable
    336
    337     def __iter__(self):

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in __iter__(self)
     76
     77     def __iter__(self):
---> 78         for key, example in self.generate_examples_fn(**self.kwargs):
     79             yield key, example
     80

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in wrapper(**kwargs)
    282     def wrapper(**kwargs):
    283         python_formatter = PythonFormatter()
--> 284         for key, table in generate_tables_fn(**kwargs):
    285             batch = python_formatter.format_batch(table)
    286             for i, example in enumerate(_batch_to_examples(batch)):

~/Documents/GitHub/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files, original_files)
     85                         file,
     86                         read_options=self.config.pa_read_options,
---> 87                         parse_options=self.config.pa_parse_options,
     88                     )
     89                 except pa.ArrowInvalid as err:

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/_json.pyx in pyarrow._json.read_json()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/_json.pyx in pyarrow._json._get_reader()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.get_input_stream()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.get_native_file()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.OSFile.__cinit__()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.OSFile._open_readable()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

FileNotFoundError: [Errno 2] Failed to open local file 'gzip://file-000000000000.json::/Users/thomwolf/github-dataset/file-000000000000.json.gz'. Detail: [errno 2] No such file or directory
```

## Environment info
- `datasets` version: 1.9.1.dev0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyArrow version: 1.0.0


[Metrics] addition of wiki_split metrics

enhancement

New feature or request

metric request

Requesting to add a new metric

**Is your feature request related to a problem? Please describe.**
While training the model on sentence split the task in English we require to evaluate the trained model on `Exact Match`, `SARI` and `BLEU` score
like this
![image](https://user-images.githubusercontent.com/26653468/124746876-ff5a3380-df3e-11eb-9a01-4b48db7a6694.png)
While training we require metrics which can give all the output

Currently, we don't have an exact match for text normalized data

**Describe the solution you'd like**
A custom metrics for wiki_split that can calculate these three values and provide it in the form of a single dictionary
For exact match, we can refer to [this](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py)

**Describe alternatives you've considered**
Two metrics are already present one more can be added for an exact match then we can run all three metrics in training script

#self-assign


Make any ClientError trigger retry in streaming mode (e.g. ClientOSError)

1.10

Next minor release

During the FLAX sprint some users have this error when streaming datasets:
```python
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
```
This error must trigger a retry instead of directly crashing

Therefore I extended the error type that triggers the retry to be the base aiohttp error type: `ClientError`
In particular both `ClientOSError` and `ServerDisconnectedError` inherit from `ClientError`.


Add option to delete temporary files (e.g. extracted files) when loading dataset

enhancement

New feature or request

1.10

Next minor release

Hi !
If we want something more general, we could either
1. delete the extracted files after the arrow data generation automatically, or
2. delete each extracted file during the arrow generation right after it has been closed.

Solution 2 is better to save disk space during the arrow generation. Is it what you had in mind ?

The API could look like
```python
load_dataset(..., delete_extracted_files_after_usage=True)
```

In terms of implementation, here are some directions we could take for each solution:
1. get the list of the extracted files from the DownloadManager and then delete them after the dataset is processed. This can be implemented in `download_and_prepare` I guess
2. maybe wrap and mock `open` in the builder to make it delete the file when the file is closed.

Also, if I delete the extracted files they need to be re-extracted again instead of loading from the Arrow cache files

I think we already opened an issue about this topic (suggested by @stas00): duplicated of #2481?

This is in our TODO list... 😅

I think the deletion of each extracted file could be implemented in our CacheManager and ExtractManager (once merged to master: #2295, #2277). 😉

Oh yes sorry, I didn't check if this was a duplicate

Nevermind @thomwolf, I just mentioned the other issue so that both appear linked in GitHub and we do not forget to close both once we make the corresponding Pull Request... That was the main reason! 😄

Ok yes. I think this is an important feature to be able to use large datasets which are pretty much always compressed files.

In particular now this requires to keep the extracted file on the drive if you want to avoid reprocessing the dataset so in my case, this require using always ~400GB of drive instead of just 200GB (which is already significant).

Two nice features would be to:
- allow to delete the extracted files without loosing the ability to load the dataset from the cached arrow-file
- streamlined decompression when only the currently read file is extracted - this might require to read the list of files from the extracted archives before processing them?

Here is a sample dataset with 2 such large compressed JSON files for debugging: https://huggingface.co/datasets/thomwolf/github-python

Note that I'm confirming that with the current master branch of dataset, deleting extracted files (without deleting the arrow cache file) lead to **re-extracting** these files when reloading the dataset instead of directly loading the arrow cache file.

Hi ! That's weird, it doesn't do that on my side (tested on master on my laptop by deleting the `extracted` folder in the download cache directory). You tested with one of the files at https://huggingface.co/datasets/thomwolf/github-python that you have locally ?

Yes it’s when I load local compressed JSON line files with load_dataset(‘json’, data_files=…)

@thomwolf I'm sorry but I can't reproduce this problem. I'm also using:
```python
ds = load_dataset("json", split="train", data_files=data_files, cache_dir=cache_dir)
```
after having removed the extracted files:
```python
assert sorted((cache_dir / "downloads" / "extracted").iterdir()) == []
```

I get the logging message:
```shell
WARNING  datasets.builder:builder.py:531 Reusing dataset json ...
```

>
>
> Do you confirm the extracted folder stays empty after reloading?

Yes, I have the above mentioned assertion on the emptiness of the extracted folder:
```python
assert sorted((cache_dir / "downloads" / "extracted").iterdir()) == []
```

I'm loading a dataset constituted of 44 GB of compressed JSON files.

When loading the dataset with the JSON script, extracting the files create about 200 GB of uncompressed files before creating the 180GB of arrow cache tables

Having a simple way to delete the extracted files after usage (or even better, to stream extraction/delete) would be nice to avoid disk cluter.

I can maybe tackle this one in the JSON script unless you want a more general solution.


Fix DuplicatedKeysError in omp

1.10

Next minor release

Close #2598.


Remove import of transformers

1.10

Next minor release

When pickling a tokenizer within multiprocessing, check that is instance of transformers PreTrainedTokenizerBase without importing transformers.

Related to huggingface/transformers#12549 and #502.


Fix `filter` with multiprocessing in case all samples are discarded

1.10

Next minor release

Fixes #2600

Also I moved the check for `num_proc` larger than dataset size added in #2566 up so that multiprocessing is not used with one process.


Crash when using multiprocessing (`num_proc` > 1) on `filter` and all samples are discarded

bug

Something isn't working

## Describe the bug
If `filter` is applied to a dataset using multiprocessing (`num_proc` > 1) and all sharded datasets are empty afterwards (due to all samples being discarded), the program crashes.

## Steps to reproduce the bug
```python
from datasets import Dataset
data = Dataset.from_dict({'id': [0,1]})
data.filter(lambda x: False, num_proc=2)
```

## Expected results
An empty table should be returned without crashing.

## Actual results
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 185, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2143, in filter
    return self.map(
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1738, in map
    result = concatenate_datasets(transformed_shards)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3267, in concatenate_datasets
    table = concat_tables(tables_to_concat, axis=axis)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/table.py", line 853, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/table.py", line 713, in from_tables
    blocks = to_blocks(tables[0])
IndexError: list index out of range
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.12.11-300.fc34.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.10
- PyArrow version: 3.0.0


Update processing.rst with other export formats

1.10

Next minor release

Add other supported export formats than CSV in the docs.


Unable to download omp dataset

Something isn't working

Hi @erikadistefano , thanks for reporting the issue.

I have created a Pull Request that should fix it.

Once merged into master, feel free to update your installed `datasets` library (either by installing it from our GitHub master branch or waiting until our next release) to be able to load omp dataset.

## Describe the bug
The omp dataset cannot be downloaded because of a DuplicatedKeysError

## Steps to reproduce the bug
from datasets import load_dataset
omp = load_dataset('omp', 'posts_labeled')
print(omp)

## Expected results
This code should download the omp dataset and print the dictionary

## Actual results
Downloading and preparing dataset omp/posts_labeled (download: 1.27 MiB, generated: 13.31 MiB, post-processed: Unknown size, total: 14.58 MiB) to /home/erika_distefano/.cache/huggingface/datasets/omp/posts_labeled/1.1.0/2fe5b067be3bff1d4588d5b0cbb9b5b22ae1b9d5b026a8ff572cd389f862735b...
0 examples [00:00, ? examples/s]2021-07-06 09:43:55.868815: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 990, in _prepare_split
    writer.write(example, key)
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 338, in write
    self.check_duplicate_keys()
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 349, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 3326
Keys should be unique and deterministic in nature

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "hf_datasets.py", line 32, in <module>
    omp = load_dataset('omp', 'posts_labeled')
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 992, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 409, in finalize
    self.check_duplicate_keys()
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 349, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 3326
Keys should be unique and deterministic in nature

## Environment info
- `datasets` version: 1.8.0
- Platform: Ubuntu 18.04.4 LTS
- Python version: 3.6.9
- PyArrow version: 3.0.0


Remove redundant prepare_module

refactoring

Restructuring existing code without changing its external behavior

1.10

Next minor release

I have noticed that after implementing `load_dataset_builder` (#2500), there is a redundant call to `prepare_module`.


Transformer Class on dataset

enhancement

New feature or request

Example:

Merge 2 datasets into one datasets

Label extraction from dataset

dataset(text, label)
   —> dataset(text, newlabel)

TextCleaning.

For image dataset,
Transformation are easier (ie linear algebra).

> On Jul 6, 2021, at 17:39, Quentin Lhoest ***@***.***> wrote:
>
>
> Hi ! Do you have an example in mind that shows how this could be useful ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

There are already a few transformations that you can apply on a dataset using methods like `dataset.map()`.
You can find examples in the documentation here:
https://huggingface.co/docs/datasets/processing.html

You can merge two datasets with `concatenate_datasets()` or do label extraction with `dataset.map()` for example

Ok, sure.

Thanks for pointing on functional part.
My question is more
“Philosophical”/Design perspective.

There are 2 perspetive:
  Add transformation methods to
     Dataset Class

 OR Create a Transformer Class
   which operates on Dataset Class.

T(Dataset) —> Dataset

datasetnew = MyTransform.transform(dataset)
datasetNew.save(path)

What would be the difficulty
of implementing a Transformer Class
operating at dataset level ?

thanks

> On Jul 6, 2021, at 22:00, Quentin Lhoest ***@***.***> wrote:
>
>
> There are already a few transformations that you can apply on a dataset using methods like dataset.map().
> You can find examples in the documentation here:
> https://huggingface.co/docs/datasets/processing.html
>
> You can merge two datasets with concatenate_datasets() or do label extraction with dataset.map() for example
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

I can imagine that this would be a useful API to implement processing pipelines as transforms. They could be used to perform higher level transforms compared to the atomic transforms allowed by methods like map, filter, etc.

I guess if you find any transform that could be useful for text dataset processing, image dataset processing etc. we could definitely start having such transforms :)

Thanks for reply.

What would be the constraints
to have
Dataset —> Dataset consistency ?

Main issue would be
larger than memory dataset and
serialization on disk.

Technically,
one still process at atomic level
and try to wrap the full results
into Dataset…. (!)

What would you think ?

> On Jul 7, 2021, at 16:51, Quentin Lhoest ***@***.***> wrote:
>
>
> I can imagine that this would be a useful API to implement processing pipelines as transforms. They could be used to perform higher level transforms compared to the atomic transforms allowed by methods like map, filter, etc.
>
> I guess if you find any transform that could be useful for text dataset processing, image dataset processing etc. we could definitely start having such transforms :)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

We can be pretty flexible and not impose any constraints for transforms.

Moreover, this library is designed to support datasets bigger than memory. The datasets are loaded from the disk via memory mapping, without filling up RAM. Even processing functions like `map` work in a batched fashion to not fill up your RAM. So this shouldn't be an issue

Ok thanks.

But, Dataset has various flavors.
In current design of Dataset,
   how the serialization on disk is done (?)

The main issue is serialization
of   newdataset= Transform(Dataset)
 (ie thats why am referring to Out Of memory dataset…):

   Should be part of Transform or part of dataset ?

Maybe, not, since the output is aimed to feed model in memory (?)

> On Jul 7, 2021, at 18:04, Quentin Lhoest ***@***.***> wrote:
>
>
> We can be pretty flexible and not impose any constraints for transforms.
>
> Moreover, this library is designed to support datasets bigger than memory. The datasets are loaded from the disk via memory mapping, without filling up RAM. Even processing functions like map work in a batched fashion to not fill up your RAM. So this shouldn't be an issue
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

I'm not sure I understand, could you elaborate a bit more please ?

Each dataset is a wrapper of a PyArrow Table that contains all the data. The table is loaded from an arrow file on the disk.
We have an ArrowWriter and ArrowReader class to write/read arrow tables on disk or in in-memory buffers.

Just wondering if you have intenttion to create

TransformerClass :
    dataset --> dataset

and make determnistic transformation (ie not fit).


ModuleNotFoundError: No module named 'datasets.tasks' while importing common voice datasets

bug

Something isn't working

Hi @profsatwinder.

It looks like you are using an old version of `datasets`. Please update it with `pip install -U datasets` and indicate if the problem persists.

@albertvillanova Thanks for the information. I updated it to 1.9.0 and the issue is resolved. Thanks again.

Error traceback:
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-8-a7b592d3bca0> in <module>()
      1 from datasets import load_dataset, load_metric
      2
----> 3 common_voice_train = load_dataset("common_voice", "pa-IN", split="train+validation")
      4 common_voice_test = load_dataset("common_voice", "pa-IN", split="test")

9 frames
/root/.cache/huggingface/modules/datasets_modules/datasets/common_voice/078d412587e9efeb0ae2e574da99c31e18844c496008d53dc5c60f4159ed639b/common_voice.py in <module>()
     19
     20 import datasets
---> 21 from datasets.tasks import AutomaticSpeechRecognition
     22
     23

ModuleNotFoundError: No module named 'datasets.tasks'


Fix BibTeX entry

Fix BibTeX entry.


Support pandas 1.3.0 read_csv

Workaround for this issue in pandas 1.3.0 : https://github.com/pandas-dev/pandas/issues/42387

The csv reader raises an error:
```python
/usr/local/lib/python3.7/dist-packages/pandas/io/parsers/readers.py in _refine_defaults_read(dialect, delimiter, delim_whitespace, engine, sep, error_bad_lines, warn_bad_lines, on_bad_lines, names, prefix, defaults)
   1304
   1305     if names is not lib.no_default and prefix is not lib.no_default:
-> 1306         raise ValueError("Specified named and prefix; you can only specify one.")
   1307
   1308     kwds["names"] = None if names is lib.no_default else names

ValueError: Specified named and prefix; you can only specify one.
```


Add c4.noclean infos

Adding the data files checksums and the dataset size of the c4.noclean configuration of the C4 dataset


Cached dataset overflowing disk space

Hi! I'm transferring this issue over to `datasets`

I'm using the datasets concatenate dataset to combine the datasets and then train.
train_dataset = concatenate_datasets([dataset1, dataset2, common_voice_train])

Hi @BirgerMoell.

You have several options:
- to set caching to be stored on a different path location, other than the default one (`~/.cache/huggingface/datasets`):
  - either setting the environment variable `HF_DATASETS_CACHE` with the path to the new cache location
  - or by passing it with the parameter `cache_dir` when loading each of the datasets: `dataset = load_dataset(..., cache_dir=your_new_location)`

  You can get all the information in the docs: https://huggingface.co/docs/datasets/loading_datasets.html#cache-directory
- I wouldn't recommend disabling caching, because current implementation generates cache files anyway, although in a temporary directory and they are deleted when the session closes. See details here: https://huggingface.co/docs/datasets/processing.html#enable-or-disable-caching
- You could alternatively load the datasets in streaming mode. This is a new feature which allows loading the datasets without downloading the entire files. More information here: https://huggingface.co/docs/datasets/dataset_streaming.html

Hi @BirgerMoell,

We are planning to add a new feature to datasets, which could be interesting in your case: Add the option to delete temporary files (decompressed files) from the cache directory (see: #2481, #2604).

We will ping you once this feature is implemented, so that the size of your cache directory will be considerably reduced.

I'm training a Swedish Wav2vec2 model on a Linux GPU and having issues that the huggingface cached dataset folder is completely filling up my disk space (I'm training on a dataset of around 500 gb).

The cache folder is 500gb (and now my disk space is full).

Is there a way to toggle caching or set the caching to be stored on a different device (I have another drive with 4 tb that could hold the caching files).

This might not technically be a bug, but I was unsure and I felt that the bug was the closest one.

Traceback (most recent call last):
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 186, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1983, in _map_single
    writer.finalize()
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_writer.py", line 418, in finalize
    self.pa_writer.close()
  File "pyarrow/ipc.pxi", line 402, in pyarrow.lib._CRecordBatchWriter.close
  File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
OSError: [Errno 28] Error writing bytes to file. Detail: [errno 28] No space left on device
"""

The above exception was the direct cause of the following exception:


Add language tags

This PR adds some missing language tags needed for ASR datasets in #2565


Support multilabel metrics

1.10

Next minor release

Hi ! Thanks for the fix :)

If I understand correctly, `OptionalSequence` doesn't have an associated arrow type that we know in advance unlike the other feature types, because it depends on the type of the examples.

For example, I tested this and it raises an error:
```python
import datasets as ds
import pyarrow as pa

features = ds.Features({"a": ds.features.OptionalSequence(ds.Value("int32"))})
batch = {"a": [[0]]}

writer = ds.ArrowWriter(features=features, stream=pa.BufferOutputStream())
writer.write_batch(batch)
# ArrowInvalid: Could not convert [0] with type list: tried to convert to int
```
This error happens because `features.type` is `StructType(struct<a: int32>)`.

Another way to add support for multilabel would be to have several configurations for these metrics. By default it would set the features without sequences, and for the multi label configuration it would use features with sequences. Let me know what you think

Hi @lhoestq, thanks for your feedback :)

Definitely, your suggested approach is simpler. I am going to refactor all my PR unless we could envision some other use cases where an OptionalSequence might be convenient, but for now I can't think of any...

Currently, multilabel metrics are not supported because `predictions` and `references` are defined as `Value("int32")`.

This PR creates a new feature type `OptionalSequence` which can act as either `Value("int32")` or `Sequence(Value("int32"))`, depending on the data passed.

Close #2554.


Fix test_is_small_dataset

1.10

Next minor release

Remove environment variable fixture `env_max_in_memory_dataset_size`. This fixture does not work because env variable is read in datasets.config when first loading datasets, and it is never reread during tests.


Add aiohttp to tests extras require

Currently, none of the streaming tests are runned within our CI test suite, because the streaming tests require aiohttp and this is missing from our tests extras require dependencies.

Our CI test suite should be exhaustive and test all the library functionalities.


Fix misalignment in SQuAD

1.10

Next minor release

Fix misalignment between:
- the answer text and
- the answer_start within the context

by keeping original leading blank spaces in the context.

Fix #2585.


sqaud_v2 dataset contains misalignment between the answer text and the context value at the answer index

Something isn't working

Hi @mmajurski, thanks for reporting this issue.

Indeed this misalignment arises because the source dataset context field contains leading blank spaces (and these are counted within the answer_start), while our datasets loading script removes these leading blank spaces.

I'm going to fix our script so that all leading blank spaces in the source dataset are kept, and there is no misalignment between the answer text and the answer_start within the context.

If you are going to be altering the data cleaning from the source Squad dataset, here is one thing to consider.
There are occasional double spaces separating words which it might be nice to get rid of.

Either way, thank you.

## Describe the bug
The built in huggingface squad_v2 dataset that you can access via datasets.load_dataset contains mis-alignment between the answers['text'] and the characters in the context at the location specified by answers['answer_start'].

For example:
id = '56d1f453e7d4791d009025bd'
answers = {'text': ['Pure Land'], 'answer_start': [146]}
However the actual text in context at location 146 is 'ure Land,'
Which is an off-by-one error from the correct answer.

## Steps to reproduce the bug
```python
import datasets

def check_context_answer_alignment(example):
    for a_idx in range(len(example['answers']['text'])):
        # check raw dataset for answer consistency between context and answer
        answer_text = example['answers']['text'][a_idx]
        a_st_idx = example['answers']['answer_start'][a_idx]
        a_end_idx = a_st_idx + len(example['answers']['text'][a_idx])
        answer_text_from_context = example['context'][a_st_idx:a_end_idx]
        if answer_text != answer_text_from_context:
            #print(example['id'])
            return False
    return True

dataset = datasets.load_dataset('squad_v2', split='train', keep_in_memory=True)

start_len = len(dataset)
dataset = dataset.filter(check_context_answer_alignment,
                                num_proc=1,
                                keep_in_memory=True)
end_len = len(dataset)
print('{} instances contain mis-alignment between the answer text and answer index.'.format(start_len - end_len))
```

## Expected results
This code should result in 0 rows being filtered out from the dataset.

## Actual results
This filter command results in 258 rows being flagged as containing a discrepancy between the text contained within answers['text'] and the text in example['context'] at the answers['answer_start'] location.

This code will reproduce the problem and produce the following count:
"258 instances contain mis-alignment between the answer text and answer index."

## Environment info
Steps to rebuilt the Conda environment:
```
# create a virtual environment to stuff all these packages into
conda create -n round8 python=3.8 -y

# activate the virtual environment
conda activate round8

# install pytorch (best done through conda to handle cuda dependencies)
conda install pytorch torchvision torchtext cudatoolkit=11.1 -c pytorch-lts -c nvidia

pip install jsonpickle transformers datasets matplotlib
```

OS: Ubuntu 20.04
Python 3.8

Result of `conda env export`:
```
name: round8
channels:
  - pytorch-lts
  - nvidia
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=4.5=1_gnu
  - blas=1.0=mkl
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - ca-certificates=2021.5.25=h06a4308_1
  - certifi=2021.5.30=py38h06a4308_0
  - cffi=1.14.5=py38h261ae71_0
  - chardet=4.0.0=py38h06a4308_1003
  - cryptography=3.4.7=py38hd23ed53_0
  - cudatoolkit=11.1.74=h6bb024c_0
  - ffmpeg=4.2.2=h20bf706_0
  - freetype=2.10.4=h5ab3b9f_0
  - gmp=6.2.1=h2531618_2
  - gnutls=3.6.15=he1e5248_0
  - idna=2.10=pyhd3eb1b0_0
  - intel-openmp=2021.2.0=h06a4308_610
  - jpeg=9b=h024ee3a_2
  - lame=3.100=h7b6447c_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.3.0=h5101ec6_17
  - libgomp=9.3.0=h5101ec6_17
  - libidn2=2.3.1=h27cfd23_0
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libstdcxx-ng=9.3.0=hd4cf53a_17
  - libtasn1=4.16.0=h27cfd23_0
  - libtiff=4.2.0=h85742a9_0
  - libunistring=0.9.10=h27cfd23_0
  - libuv=1.40.0=h7b6447c_0
  - libvpx=1.7.0=h439df22_0
  - libwebp-base=1.2.0=h27cfd23_0
  - lz4-c=1.9.3=h2531618_0
  - mkl=2021.2.0=h06a4308_296
  - mkl-service=2.3.0=py38h27cfd23_1
  - mkl_fft=1.3.0=py38h42c9631_2
  - mkl_random=1.2.1=py38ha9443f7_2
  - ncurses=6.2=he6710b0_1
  - nettle=3.7.3=hbbd107a_1
  - ninja=1.10.2=hff7bd54_1
  - numpy=1.20.2=py38h2d18471_0
  - numpy-base=1.20.2=py38hfae3a4d_0
  - olefile=0.46=py_0
  - openh264=2.1.0=hd408876_0
  - openssl=1.1.1k=h27cfd23_0
  - pillow=8.2.0=py38he98fc37_0
  - pip=21.1.2=py38h06a4308_0
  - pycparser=2.20=py_2
  - pyopenssl=20.0.1=pyhd3eb1b0_1
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.10=h12debd9_8
  - pytorch=1.8.1=py3.8_cuda11.1_cudnn8.0.5_0
  - readline=8.1=h27cfd23_0
  - requests=2.25.1=pyhd3eb1b0_0
  - setuptools=52.0.0=py38h06a4308_0
  - six=1.16.0=pyhd3eb1b0_0
  - sqlite=3.35.4=hdfb4753_0
  - tk=8.6.10=hbc83047_0
  - torchtext=0.9.1=py38
  - torchvision=0.9.1=py38_cu111
  - typing_extensions=3.7.4.3=pyha847dfd_0
  - urllib3=1.26.4=pyhd3eb1b0_0
  - wheel=0.36.2=pyhd3eb1b0_0
  - x264=1!157.20191217=h7b6447c_0
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.9=haebb681_0
  - pip:
    - click==8.0.1
    - cycler==0.10.0
    - datasets==1.8.0
    - dill==0.3.4
    - filelock==3.0.12
    - fsspec==2021.6.0
    - huggingface-hub==0.0.8
    - joblib==1.0.1
    - jsonpickle==2.0.0
    - kiwisolver==1.3.1
    - matplotlib==3.4.2
    - multiprocess==0.70.12.2
    - packaging==20.9
    - pandas==1.2.4
    - pyarrow==3.0.0
    - pyparsing==2.4.7
    - python-dateutil==2.8.1
    - pytz==2021.1
    - regex==2021.4.4
    - sacremoses==0.0.45
    - tokenizers==0.10.3
    - tqdm==4.49.0
    - transformers==4.6.1
    - xxhash==2.0.2
prefix: /home/mmajurski/anaconda3/envs/round8
```


wi_locness: reference latest leaderboard on codalab

The dataset's author asked me to put this codalab link into the dataset's README.


Error iteration over IterableDataset using Torch DataLoader

Something isn't working

Hi ! This is because you first need to format the dataset for pytorch:

```python
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)
>>> torch_iterable_dataset = dataset.with_format("torch")
>>> assert isinstance(torch_iterable_dataset, torch.utils.data.IterableDataset)
>>> dataloader = torch.utils.data.DataLoader(torch_iterable_dataset, batch_size=4)
>>> next(iter(dataloader))
{'id': tensor([0, 1, 2, 3]), 'text': ['Mtendere Village was inspired...]}
```

This is because the pytorch dataloader expects a subclass of `torch.utils.data.IterableDataset`. Since you can't pass an arbitrary iterable to a pytorch dataloader, you first need to build an object that inherits from `torch.utils.data.IterableDataset` using `with_format("torch")` for example.

Thank you for that and the example!

What you said makes total sense; I just somehow missed that and assumed HF IterableDataset was a subclass of Torch IterableDataset.

## Describe the bug
I have an IterableDataset (created using streaming=True) and I am trying to create batches using Torch DataLoader class by passing this IterableDataset to it. This throws error which is pasted below. I can do the same by using Torch IterableDataset. One thing I noticed is that in the former case when I look at the dataloader.sampler class I get torch.utils.data.sampler.SequentialSampler while the latter one gives torch.utils.data.dataloader._InfiniteConstantSampler.

I am not sure if this is how it is meant to be used, but that's what seemed reasonable to me.

## Steps to reproduce the bug

1. Does not work.
```python
>>> from datasets import load_dataset
>>> dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)
>>> dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
>>> dataloader.sampler
<torch.utils.data.sampler.SequentialSampler object at 0x7f245a510208>
>>> for batch in dataloader:
...     print(batch)
```

2. Works.
```python
import torch
from torch.utils.data import Dataset, IterableDataset, DataLoader
class CustomIterableDataset(IterableDataset):
  'Characterizes a dataset for PyTorch'
  def __init__(self, data):
        'Initialization'
        self.data = data

  def __iter__(self):
        return iter(self.data)

data = list(range(12))
dataset = CustomIterableDataset(data)
dataloader = DataLoader(dataset, batch_size=4)
print("dataloader: ", dataloader.sampler)
for batch in dataloader:
    print(batch)
```

## Expected results
To get batches of data with the batch size as 4. Output from the latter one (2) though Datasource is different here so actual data is different.
dataloader:  <torch.utils.data.dataloader._InfiniteConstantSampler object at 0x7f1cc29e2c50>
tensor([0, 1, 2, 3])
tensor([4, 5, 6, 7])
tensor([ 8,  9, 10, 11])

## Actual results
<torch.utils.data.sampler.SequentialSampler object at 0x7f245a510208>

...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 474, in _next_data
    index = self._next_index()  # may raise StopIteration
  File "/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 427, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 227, in __iter__
    for idx in self.sampler:
  File "/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 67, in __iter__
    return iter(range(len(self.data_source)))
TypeError: object of type 'IterableDataset' has no len()

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: '1.8.1.dev0'
- Platform: Linux
- Python version: Python 3.6.8
- PyArrow version: '3.0.0'


Add skip and take

@lhoestq looks good. I tried with https://huggingface.co/datasets/vblagoje/wikipedia_snippets_streamed and it worked nicely. I would add more unit tests for edge cases. What happens if the n is larger than the total number of samples? Just to make sure these cases are handled properly.

Yup I'll add the tests thanks ;)

Moreover, I just noticed something in your wiki snippets code. FYI you're using `++passage_counter ` at https://huggingface.co/datasets/vblagoje/wikipedia_snippets_streamed/blob/main/wikipedia_snippets_streamed.py#L102 but in python this doesn't increment the value @vblagoje

Thanks @lhoestq - not easy to convert after 10+ years of Java

As discussed in https://github.com/huggingface/datasets/pull/2375#discussion_r657084544 I added the `IterableDataset.skip` and `IterableDataset.take` methods that allows to do basic splitting of iterable datasets.

You can create new dataset with the first `n` examples using `IterableDataset.take()`, or you can get a dataset with the rest of the examples by skipping the first `n` examples with `IterableDataset.skip()`

One implementation detail:

Using `take` (or `skip`) prevents future dataset shuffling from shuffling the dataset shards, otherwise the taken examples could come from other shards. In this case it only uses the shuffle buffer.
I would have loved to allow the shards of the taken examples to be shuffled anyway, but since we don't know in advance the length of each shard we don't know what shards to take or skip.
I think this is ok though since users can shuffle before doing take or skip. I mentioned this in the documentation

cc @vblagoje @lewtun


Faster search_batch for ElasticsearchIndex due to threading

1.10

Next minor release

Hey,
I think it makes sense to perform search_batch threaded, so ES can perform search in parallel.
Cheers!


Fix Counter import

Import from `collections` instead of `typing`.


Fix BibTeX entry

Add missing contributor to BibTeX entry.

cc: @abhishekkrthakur @thomwolf


Support Zstandard compressed files

> What if people want to run some tests without having zstandard ?
> Usually what we do is add a decorator @require_zstandard for example

@lhoestq I think I'm missing something here...

Tests are a *development* tool (to ensure we deliver a good quality lib), not something we offer to the end users of the lib. Users of the lib just `pip install datasets` and no tests are delivered with the lib (`tests` directory is outside the `src` code dir).

On the contrary, developers (contributors) of the lib do need to be able to run tests (TDD). And because of that, they are required to install datasets differently: `pip install -e .[dev]`, so that all required developing (and testing) dependencies are properly installed (included `zstandard`).

Apart from `zsatandard`, there are many other dev/test required dependencies for running tests, and we do not have a `@require_toto` for each and every of these dependencies in our tests:
- `pytest` and `absl-py` (they are not dependencies in install_requires, but only in TEST_REQUIRE extras_require),
- `boto3` (in test_filesystem.py),
- `seqeval` (in test_metric_common.py),
- `bs4` (used by eli5 and tested in test_hf_gcp.py)
- ...

So IMHO, to run tests you should previously install datasets with dev or tests dependencies: either `pip install -e .[dev]` or `pip install -e .[tests]` (the latter to be used in CI testing-only part of the development cycle). And the tests should be written accordingly, assuming all tests dependencies are installed.

Hi !
I was saying that because the other dependencies you mentioned are only required for _some_ tests. While here zstd is required for _all_ tests since it's imported in the conftest.py
Feel free to keep it as it is right now, or maybe move the fixture to test_file_utils.py to allow users without zstd to run tests for their builders, dataset card etc. without issues

> @lhoestq does this mean that the pile could have streaming support in the future? Afaik streaming doesnt support zstandard compressed type

just for reference, i tried to stream one of the `.zst` files from [the pile](https://the-eye.eu/public/AI/pile/) using

```python
data_files = ["https://the-eye.eu/public/AI/pile/train/00.jsonl.zst"]
streamed_dataset = load_dataset('json', split='train', data_files=data_files, streaming=True)
```

and got the following error:

```
Using custom data configuration default-4e71acadc389c254
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
/tmp/ipykernel_1187680/10848115.py in <module>
      1 data_files = ["https://the-eye.eu/public/AI/pile/train/00.jsonl.zst"]
      2
----> 3 streamed_dataset = load_dataset('json', split='train', data_files=data_files, streaming=True)
      4

~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
    835         # this extends the open and os.path.join functions for data streaming
    836         extend_module_for_streaming(builder_instance.__module__, use_auth_token=use_auth_token)
--> 837         return builder_instance.as_streaming_dataset(
    838             split=split,
    839             use_auth_token=use_auth_token,

~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/builder.py in as_streaming_dataset(self, split, base_path, use_auth_token)
    922             data_dir=self.config.data_dir,
    923         )
--> 924         splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
    925         # By default, return all splits
    926         if split is None:

~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/packaged_modules/json/json.py in _split_generators(self, dl_manager)
     50         if not self.config.data_files:
     51             raise ValueError(f"At least one data file must be specified, but got data_files={self.config.data_files}")
---> 52         data_files = dl_manager.download_and_extract(self.config.data_files)
     53         if isinstance(data_files, (str, list, tuple)):
     54             files = data_files

~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in download_and_extract(self, url_or_urls)
    140
    141     def download_and_extract(self, url_or_urls):
--> 142         return self.extract(self.download(url_or_urls))

~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in extract(self, path_or_paths)
    115
    116     def extract(self, path_or_paths):
--> 117         urlpaths = map_nested(self._extract, path_or_paths, map_tuple=True)
    118         return urlpaths
    119

~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types)
    202         num_proc = 1
    203     if num_proc <= 1 or len(iterable) <= num_proc:
--> 204         mapped = [
    205             _single_map_nested((function, obj, types, None, True))
    206             for obj in utils.tqdm(iterable, disable=disable_tqdm)

~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/py_utils.py in <listcomp>(.0)
    203     if num_proc <= 1 or len(iterable) <= num_proc:
    204         mapped = [
--> 205             _single_map_nested((function, obj, types, None, True))
    206             for obj in utils.tqdm(iterable, disable=disable_tqdm)
    207         ]

~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/py_utils.py in _single_map_nested(args)
    141     # Singleton first to spare some computation
    142     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 143         return function(data_struct)
    144
    145     # Reduce logging to keep things readable in multiprocessing with tqdm

~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in _extract(self, urlpath)
    119
    120     def _extract(self, urlpath):
--> 121         protocol = self._get_extraction_protocol(urlpath)
    122         if protocol is None:
    123             # no extraction

~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in _get_extraction_protocol(self, urlpath)
    137         elif path.endswith(".zip"):
    138             return "zip"
--> 139         raise NotImplementedError(f"Extraction protocol for file at {urlpath} is not implemented yet")
    140
    141     def download_and_extract(self, url_or_urls):

NotImplementedError: Extraction protocol for file at https://the-eye.eu/public/AI/pile/train/00.jsonl.zst is not implemented yet
```

i'm not sure whether @Shashi456 is referring to a fundamental limitation with "streaming" zstandard compression files or simply that we need to support the protocol in the streaming api of `datasets`

@lewtun our streaming mode patches the Python `open` function. I could have a look tomorrow if it is easily implementable for this case.

@lewtun, I have tested and yes, it is easily implementable. I've created a draft Pull Request with an implementation proposal: #2786.

thanks a lot @albertvillanova - now i can stream the pile :)

Close #2572.

cc: @thomwolf


Add mC4

AllenAI is now hosting the processed C4 and mC4 dataset in this repo: https://huggingface.co/datasets/allenai/c4
Thanks a lot to them !

In this PR I added the mC4 dataset builder. It supports 108 languages

You can load it with
```python
from datasets import load_dataset

en_mc4 = load_dataset("mc4", "en")
fr_mc4 = load_dataset("mc4", "fr")
en_and_fr_mc4 = load_dataset("mc4", languages=["en", "fr"])
```

It also supports streaming, if you don't want to download hundreds of GB of data:
```python
en_mc4 = load_dataset("mc4", "en", streaming=True)
```

Regarding the dataset_infos.json, I will add them once I have them.

Also we can work on the dataset card at that will be at https://huggingface.co/datasets/mc4
For now I just added a link to https://huggingface.co/datasets/allenai/c4 as well as a few sections


Add C4

The old code for the C4 dataset was to generate the C4 with Apache Beam, as in Tensorflow Datasets.
However AllenAI is now hosting the processed C4 dataset in this repo: https://huggingface.co/datasets/allenai/c4
Thanks a lot to them for their amazing work !

In this PR I changed the script to download and prepare the data directly from this repo.
It has 4 variants: en, en.noblocklist, en.noclean, realnewslike

You can load it with
```python
from datasets import load_dataset

c4 = load_dataset("c4", "en")
```

It also supports streaming, if you don't want to download hundreds of GB of data:
```python
c4 = load_dataset("c4", "en", streaming=True)
```

Regarding the dataset_infos.json, I haven't added the infos for en.noclean. I will add them once I have them.

Also we can work on the dataset card at https://huggingface.co/datasets/c4
For now I just added a link to https://huggingface.co/datasets/allenai/c4 as well as a few sections


Add streaming in load a dataset docs

Mention dataset streaming on the "loading a dataset" page of the documentation


Finding right block-size with JSON loading difficult for user

Something isn't working

This was actually a second error arising from a too small block-size in the json reader.

Finding the right block size is difficult for the layman user

As reported by @thomwolf, while loading a JSON Lines file with "json" loading script, he gets
> json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 383)


Support Zstandard compressed files

enhancement

New feature or request

Add support for Zstandard compressed files: https://facebook.github.io/zstd/


Filter expected warning log from transformers

I think the failing test has nothing to do with my PR...

Close #2569.


Minor fix docs format for bertscore

Minor fix docs format for bertscore:
- link to README
- format of KWARGS_DESCRIPTION


Weights of model checkpoint not initialized for RobertaModel for Bertscore

bug

Something isn't working

Hi @suzyahyah, thanks for reporting.

The message you get is indeed not an error message, but a warning coming from Hugging Face `transformers`. The complete warning message is:
```
Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.layer_norm.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

In this case, this behavior IS expected and you can safely ignore the warning message.

The reason is that you are just using RoBERTa to get the contextual embeddings of the input sentences/tokens, thus leaving away its head layer, whose weights are ignored.

Feel free to reopen this issue if you need further explanations.

Hi @suzyahyah, I have created a Pull Request to filter out that warning message in this specific case, since the behavior is as expected and the warning message can only cause confusion for users (as in your case).

When applying bertscore out of the box,

```Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.weight']```

Following the typical usage from https://huggingface.co/docs/datasets/loading_metrics.html

```
from datasets import load_metric
metric = load_metric('bertscore')

# Example of typical usage
for batch in dataset:
    inputs, references = batch
    predictions = model(inputs)
    metric.add_batch(predictions=predictions, references=references)
score = metric.compute(lang="en")
#score = metric.compute(model_type="roberta-large") # gives the same error
```

I am concerned about this because my usage shouldn't require any further fine-tuning and most people would expect to use BertScore out of the box? I realised the huggingface code is a wrapper around https://github.com/Tiiiger/bert_score, but I think this repo is anyway relying on the model code and weights from huggingface repo....

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.0-1041-aws-x86_64-with-glibc2.27
- Python version:  3.9.5
- PyArrow version: 3.0.0


### Add interleave_datasets for map-style datasets

Add support for map-style datasets (i.e. `Dataset` objects) in `interleave_datasets`.
It was only supporting iterable datasets (i.e. `IterableDataset` objects).

### Implementation details

It works by concatenating the datasets and then re-order the indices to make the new dataset.

### TODO
- [x] tests
- [x] docs

Close #2563


Add ASR task and new languages to resources

This PR adds a new `automatic-speech-recognition` task to the list of supported tasks in `tasks.json` and also includes a few new languages missing from `common_voice`.

Note: I used the [Papers with Code list](https://www.paperswithcode.com/area/speech/speech-recognition) as inspiration for the ASR subtasks


fix Dataset.map when num_procs > num rows

closes #2470

## Testing notes
To run updated tests:
```sh
pytest tests/test_arrow_dataset.py -k "BaseDatasetTest and test_map_multiprocessing" -s
```
With Python code (to view warning):
```python
from datasets import Dataset

dataset = Dataset.from_dict({"x": ["sample"]})
print(len(dataset))
dataset.map(lambda x: x, num_proc=10)
```


Inject templates for ASR datasets

Wait until #2567 is merged so we can benefit from the tagger :)

thanks for the feedback @lhoestq! i've added the new language codes and this PR should be ready for a merge :)

This PR adds ASR templates for 5 of the most common speech datasets on the Hub, where "common" is defined by the number of models trained on them.

I also fixed a bunch of the tags in the READMEs 😎


concatenate_datasets for iterable datasets

It is probably worth noting here that the [documentation](https://huggingface.co/docs/datasets/process#concatenate) is misleading (indicating that it does work for IterableDatasets):

> You can also mix several datasets together by taking alternating examples from each one to create a new dataset. This is known as interleaving, and you can use it with [interleave_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.interleave_datasets). **Both [interleave_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.interleave_datasets) and [concatenate_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.concatenate_datasets) will work with regular [Dataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.Dataset) and [IterableDataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.IterableDataset) objects**. Refer to the [Stream](https://huggingface.co/docs/datasets/stream#interleave) section for an example of how it’s used.

Currently `concatenate_datasets` only works for map-style `Dataset`.

It would be nice to have it work for `IterableDataset` objects as well.

It would simply chain the iterables of the iterable datasets.


interleave_datasets for map-style datasets

Currently the `interleave_datasets` functions only works for `IterableDataset`.
Let's make it work for map-style `Dataset` objects as well.

It would work the same way: either alternate between the datasets in order or randomly given probabilities specified by the user.


Minor fix in loading metrics docs

Make some minor fixes in "Loading metrics" docs.


Existing cache for local dataset builder file updates is ignored with `ignore_verifications=True`

Something isn't working

Hi ! I just tried to reproduce what you said:
- create a local builder class
- use `load_dataset`
- update the builder class code
- use `load_dataset` again (with or without `ignore_verifications=True`)
And it creates a new cache, as expected.

What modifications did you do to your builder's code ?

Hi @lhoestq. Thanks for your reply. I just did minor modifications for which it should not regenerate cache (for e.g. Adding a print statement). Overall, regardless of cache miss, there should be an explicit option to allow reuse of existing cache if author knows cache shouldn't be affected.

The cache is based on the hash of the dataset builder's code, so changing the code makes it recompute the cache.

You could still rename the cache directory of your previous computation to the new expected cache directory if you want to avoid having to recompute it and if you're sure that it would generate the exact same result.

The verifications are data integrity verifications: it checks the checksums of the downloaded files, as well as the size of the generated splits.

Hi @apsdehal,

If you decide to follow @lhoestq's suggestion to rename the cache directory of your previous computation to the new expected cache directory, you can do the following to get the name of the new expected cache directory once #2500 is merged:
```python
from datasets import load_dataset_builder
dataset_builder = load_dataset_builder("path/to/your/dataset")
print(dataset_builder.cache_dir)
```

This way, you don't have to recompute the hash of the dataset script yourself each time you modify the script.

## Describe the bug
If i have local file defining a dataset builder class and I load it using `load_dataset` functionality, the existing cache is ignored whenever the file is update even with `ignore_verifications=True`. This slows down debugging and cache generator for very large datasets.

## Steps to reproduce the bug

- Create a local dataset builder class
- load the local builder class file using `load_dataset` and let the cache build
- update the file's content
- The cache should rebuilt.

## Expected results

With `ignore_verifications=True`, `load_dataset` should pick up existing cache.

## Actual results

Creates new cache.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyArrow version: 3.0.0


Hi ! Thanks for fixing this :)

Looks like you have tons of changes due to code formatting.
We're using `black` for this, with a custom line length. To run our code formatting, you just need to run
```
make style
```

Then for the windows error in the CI, I'm looking into it. It's probably just a file that isn't properly closed

CI is all green now ! Thanks :)

There are still many code formatting changes in your PR - probably due to the first commit you did.
To avoid conflicts with future PRs it would be nice to only have the changes related to the `num_proc` warning, and not have all those code formatting changes,

Could you try remove those code formatting changes ?

If it's easier for you, you can make a new branch from `master` if needed

Thanks, @lhoestq! Apologies for the half-baked commits yesterday! I wasn’t able to step back in to resolve those CI issues until this morning.

Also, I’m surprised that `make style` isn’t resolving the formatting changes. I’m a bit stumped on that, so I’m going to re-apply on a new branch and open a PR as you suggested.


Memory usage consistently increases when processing a dataset with `.map`

Something isn't working

Hi ! Can you share the function you pass to `map` ?
I know you mentioned it would be hard to share some code but this would really help to understand what happened

## Describe the bug

I have a HF dataset with image paths stored in it and I am trying to load those image paths using `.map` with `num_proc=80`. I am noticing that the memory usage consistently keeps on increasing with time. I tried using `DEFAULT_WRITER_BATCH_SIZE=10` in the builder to decrease arrow writer's batch size but that doesn't seem to help.

## Steps to reproduce the bug

Providing code as it is would be hard. I can provide a MVP if that helps.

## Expected results

Memory usage should become consistent after some time following the launch of processing.

## Actual results

Memory usage keeps on increasing.

## Environment info

- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyArrow version: 3.0.0


Update: WebNLG - update checksums

The master branch changed so I computed the new checksums.

I also pinned a specific revision so that it doesn't happen again in the future.

Fix https://github.com/huggingface/datasets/issues/2553


Fix `fever` keys

The keys has duplicates since they were reset to 0 after each file.

I fixed it by taking into account the file index as well.


Better DuplicateKeysError error to help the user debug the issue

enhancement

New feature or request

good first issue

Good for newcomers

excuse me, my `datasets` version is `2.2.2`, but I also just see the error info like
```
DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature
```

Hi ! for which dataset do you have this error ?

Also note that this issue is just about improving the error message, which is not very friendly x)

As mentioned in https://github.com/huggingface/datasets/issues/2552 it would be nice to improve the error message when a dataset fails to build because there are duplicate example keys.

The current one is
```python
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 48
Keys should be unique and deterministic in nature
```

and we could have something that guides the user to debugging the issue:
```python
DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.
Please fix the dataset script at <path/to/the/dataset/script>
```


Fix code_search_net keys

There were duplicate keys in the `code_search_net` dataset, as reported in https://github.com/huggingface/datasets/issues/2552

I fixed the keys (it was an addition of the file and row indices, which was causing collisions)

Fix #2552.


Multilabel metrics not supported

bug

Something isn't working

Hi @GuillemGSubies, thanks for reporting.

I have made a PR to fix this issue and allow metrics to be computed also for multilabel classification problems.

Sorry for reopening but I just noticed that the `_compute` method for the F1 metric is still not good enough for multilabel problems:

https://github.com/huggingface/datasets/blob/92a3ee549705aa0a107c9fa5caf463b3b3da2616/metrics/f1/f1.py#L115

Somehow we should be able to change the parameter `average` at least

@GuillemGSubies, the parameter `average` passed to `_compute` is then passed to `f1_score`. This is right.

When I try to use a metric like F1 macro I get the following error:

```
TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'
```
There is an explicit casting here:

https://github.com/huggingface/datasets/blob/fc79f61cbbcfa0e8c68b28c0a8257f17e768a075/src/datasets/features.py#L274

And looks like this is because here

https://github.com/huggingface/datasets/blob/fc79f61cbbcfa0e8c68b28c0a8257f17e768a075/metrics/f1/f1.py#L88

the features can only be integers, so we cannot use that F1 for multilabel. Instead, if I create the following F1 (ints replaced with sequence of ints), it will work:

```python
class F1(datasets.Metric):
    def _info(self):
        return datasets.MetricInfo(
            description=_DESCRIPTION,
            citation=_CITATION,
            inputs_description=_KWARGS_DESCRIPTION,
            features=datasets.Features(
                {
                    "predictions": datasets.Sequence(datasets.Value("int32")),
                    "references": datasets.Sequence(datasets.Value("int32")),
                }
            ),
            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html"],
        )

    def _compute(self, predictions, references, labels=None, pos_label=1, average="binary", sample_weight=None):
        return {
            "f1": f1_score(
                references,
                predictions,
                labels=labels,
                pos_label=pos_label,
                average=average,
                sample_weight=sample_weight,
            ),
        }
```


load_dataset("web_nlg") NonMatchingChecksumError

Something isn't working

Hi ! Thanks for reporting. This is due to the WebNLG repository that got updated today.
I just pushed a fix at #2558 - this shouldn't happen anymore in the future.

This is fixed on `master` now :)
We'll do a new release soon !

Hi! It seems the WebNLG dataset gives a NonMatchingChecksumError.

## Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset('web_nlg', name="release_v3.0_en", split="dev")
```

Gives

```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://gitlab.com/shimorina/webnlg-dataset/-/archive/master/webnlg-dataset-master.zip']
```

## Environment info
- `datasets` version: 1.8.0
- Platform: macOS-11.3.1-x86_64-i386-64bit
- Python version: 3.9.4
- PyArrow version: 3.0.0

Also tested on Linux, with python 3.6.8


Keys should be unique error on code_search_net

Something isn't working

Two questions:
- with `datasets-cli env` we don't have any information on the dataset script version used. Should we give access to this somehow? Either as a note in the Error message or as an argument with the name of the dataset to `datasets-cli env`?
- I don't really understand why the id is duplicated in the code of `code_search_net`, how can I debug this actually?

Thanks for reporting. There was indeed an issue with the keys. The key was the addition of the file id and row id, which resulted in collisions. I just opened a PR to fix this at https://github.com/huggingface/datasets/pull/2555

To help users debug this kind of errors we could try to show a message like this
```python
DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.
Please fix the dataset script at <path/to/the/dataset/script>
```

This way users who what to look for if they want to debug this issue. I opened an issue to track this: https://github.com/huggingface/datasets/issues/2556

and are we sure there are not a lot of datasets which are now broken with this change?

Thanks to the dummy data, we know for sure that most of them work as expected.
`code_search_net` wasn't caught because the dummy data only have one dummy data file while the dataset script can actually load several of them using `os.listdir`. Let me take a look at all the other datasets that use `os.listdir` to see if the keys are alright

I found one issue on `fever` (PR here: https://github.com/huggingface/datasets/pull/2557)
All the other ones seem fine :)

Hi! Got same error when loading other dataset:
```python3
load_dataset('wikicorpus', 'raw_en')
```

tb:
```pytb
---------------------------------------------------------------------------
DuplicatedKeysError                       Traceback (most recent call last)
/opt/conda/lib/python3.8/site-packages/datasets/builder.py in _prepare_split(self, split_generator)
   1109                     example = self.info.features.encode_example(record)
-> 1110                     writer.write(example, key)
   1111             finally:

/opt/conda/lib/python3.8/site-packages/datasets/arrow_writer.py in write(self, example, key, writer_batch_size)
    341             if self._check_duplicates:
--> 342                 self.check_duplicate_keys()
    343                 # Re-intializing to empty list for next batch

/opt/conda/lib/python3.8/site-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
    352             if hash in tmp_record:
--> 353                 raise DuplicatedKeysError(key)
    354             else:

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 519
Keys should be unique and deterministic in nature
```

Version: datasets==1.11.0

The wikicorpus issue has been fixed by https://github.com/huggingface/datasets/pull/2844

We'll do a new release of `datasets` soon :)

## Describe the bug
Loading `code_search_net` seems not possible at the moment.

## Steps to reproduce the bug
```python
>>> load_dataset('code_search_net')
Downloading: 8.50kB [00:00, 3.09MB/s]
Downloading: 19.1kB [00:00, 10.1MB/s]
No config specified, defaulting to: code_search_net/all
Downloading and preparing dataset code_search_net/all (download: 4.77 GiB, generated: 5.99 GiB, post-processed: Unknown size, total: 10.76 GiB) to /Users/thomwolf/.cache/huggingface/datasets/code_search_net/all/1.0.0/b3e8278faf5d67da1d06981efbeac3b76a2900693bd2239bbca7a4a3b0d6e52a...
Traceback (most recent call last):
  File "/Users/thomwolf/Documents/GitHub/datasets/src/datasets/builder.py", line 1067, in _prepare_split
    writer.write(example, key)
  File "/Users/thomwolf/Documents/GitHub/datasets/src/datasets/arrow_writer.py", line 343, in write
    self.check_duplicate_keys()
  File "/Users/thomwolf/Documents/GitHub/datasets/src/datasets/arrow_writer.py", line 354, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 48
Keys should be unique and deterministic in nature
```

## Environment info
- `datasets` version: 1.8.1.dev0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 2.0.0


Fix FileSystems documentation

### What this fixes:
This PR resolves several issues I discovered in the documentation on the `datasets.filesystems` module ([this page](https://huggingface.co/docs/datasets/filesystems.html)).

### What were the issues?
When I originally tried implementing the code examples I faced several bugs attributed to:

- out of date [botocore](https://github.com/boto/botocore) call signatures
- capitalization errors in the `S3FileSystem` class name (written as `S3Filesystem` in one place)
- call signature errors for the `S3FileSystem` class constructor (uses parameter `sessions` instead of `session` in some places) (see [`s3fs`](https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem) for where this constructor signature is defined)

### Testing/reviewing notes
Instructions for generating the documentation locally: [here](https://github.com/huggingface/datasets/tree/master/docs#generating-the-documentation).