Calling `cast_column`/`remove_columns` and a sequence of `map` operations ends up making `faiss` fail with `ValueError`

Something isn't working

It works if we either remove the `ds = ds.cast_column("id", Value("int32"))` line from the code above, or if instead calling `ds.remove_columns()` we remove the columns inside each mapping as `ds.map(..., remove_columns=[...])` instead of right after the mapping.

Both of those solutions seem to fix the issue, so the root cause of it may be around that. Sorry I cannot provide you more insights, in case I get to fix it I'll submit a PR, in the meanwhile the code that I'm using as a workaround is the following:

```python
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
import torch

torch.set_grad_enabled(False)
ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

from datasets import load_dataset, Value

ds = load_dataset("csv", data_files=["sample.csv"], split="train")
ds = ds.cast_column("id", Value("int32"))
ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])}, remove_columns=["title", "summary"])

def generate_embeddings(x):
    return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

ds = ds.map(generate_embeddings, remove_columns=["inputs"])
ds.add_faiss_index(column="embeddings")
```

FYI the main reason I want to use `dataset.remove_columns` rather than the function inside `dataset.map` is because according to the 🤗 Datasets documentation, it's faster.

"🤗 Datasets also has a [Dataset.remove_columns()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.Dataset.remove_columns) method that is functionally identical, but faster, because it doesn’t copy the data of the remaining columns."

More information at https://huggingface.co/docs/datasets/process#map

Here I'm presenting all the scenarios so that you can further investigate the issue:

- ✅ `cast_column` -> `map` with `remove_columns` -> `map` with `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])}, remove_columns=["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings, remove_columns=["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ❌ `cast_column` -> `map` -> `remove_columns` -> `map` -> `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
  ds = ds.remove_columns(["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings)
  ds = ds.remove_columns(["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ❌ `cast_column` -> `map` with `remove_columns` -> `map` -> `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])}, remove_columns=["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings)
  ds = ds.remove_columns(["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ✅ `cast_column` -> `map` -> `remove_columns` -> `map` with `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
  ds = ds.remove_columns(["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings, remove_columns=["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ✅ `map` -> `remove_columns` -> `map` -> `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
  ds = ds.remove_columns(["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings)
  ds = ds.remove_columns(["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

So on, I've created #4411 so as to fix the bug with `remove_columns` under certain conditions before `add_faiss_index`, which means that the scenarios not working above are already working fine.

First of all, sorry in advance for the unclear title, but this bug is weird to explain (at least for me), so I tried my best to summarize all the information in this issue.

## Describe the bug

Calling a certain combination of operations over a 🤗 `Dataset` and then trying to calculate the `faiss` index with `.add_faiss_index` ends up throwing an exception while trying to set the format back of a previously removed column. But this just happens over certain conditions... I'll present some scenarios below!

## Steps to reproduce the bug

Assuming the following dataset named `sample.csv` with some IMDb data:

```csv
id,title,summary
1877830,"The Batman","When a sadistic serial killer begins murdering key political figures in Gotham, Batman is forced to investigate the city's hidden corruption and question his family's involvement."
9419884,"Doctor Strange in the Multiverse of Madness","Doctor Strange teams up with a mysterious teenage girl from his dreams who can travel across multiverses, to battle multiple threats, including other-universe versions of himself, which threaten to wipe out millions across the multiverse. They seek help from Wanda the Scarlet Witch, Wong and others."
11138512,"The Northman","From visionary director Robert Eggers comes The Northman, an action-filled epic that follows a young Viking prince on his quest to avenge his father's murder."
1745960,"Top Gun: Maverick","After more than thirty years of service as one of the Navy's top aviators, Pete Mitchell is where he belongs, pushing the envelope as a courageous test pilot and dodging the advancement in rank that would ground him."
```

We'll be able to reproduce the bug using the following piece of code:

```python
# Sample code to reproduce the bug
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
import torch

torch.set_grad_enabled(False)
ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

from datasets import load_dataset, Value

ds = load_dataset("csv", data_files=["sample.csv"], split="train")
ds = ds.cast_column("id", Value("int32")) # from `int64` to `int32`
ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
ds = ds.remove_columns(["title", "summary"])

def generate_embeddings(x):
    return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

ds = ds.map(generate_embeddings)
ds = ds.remove_columns("inputs")
ds.add_faiss_index(column="embeddings") # It fails here!
```

The code above is an adaptation of https://huggingface.co/docs/datasets/faiss_es, for the sake of presenting the bug with a simple example.

## Expected results

Ideally, the `faiss` index should be calculated over the 🤗 `Dataset` and no exception should be triggered.

## Actual results

But what happens instead is that a `ValueError: Columns ['inputs'] not in the dataset. Current columns in the dataset: ['id', 'embeddings']`, which makes no sense as that column has been previously dropped.

## Environment info

<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-1074-azure-x86_64-with-glibc2.31
- Python version: 3.9.5
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Fix dependency on dill version

_The documentation is not available anymore as the PR was closed or merged._

We had to make a hotfix by pinning dill:
- #4380

because from version 0.3.5, our custom `save_function` pickling function was raising an exception:
- #4379

This PR fixes this by implementing our custom `save_function` depending on the version of dill.

CC: @anivegesana

This PR needs first being merged:
- [x] #4384
  - so that a circular import is fixed

It is also convenient to merge first:
- [x] #4385


Fix URL in gem dataset for totto config

As commented in:
- https://github.com/huggingface/datasets/issues/4386#issuecomment-1134902372

CC: @StevenTang1998


Add Pascal VOC dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4395). All of your documentation changes will be reflected on that endpoint.

This PR adds the Pascal VOC dataset in the same way TFDS has it added. I believe we can iterate on this dataset and in future versions include more data, such as segmentation masks, but for now I think it is a good idea to just add it the same way as TFDS to get a solid first version out there.


trainer became extremely slow after reload dataset by `load_from_disk`

bug

Something isn't working

I tried to make the dataset much more smaller (100000 rows) , then the speed became `33.88it/s` from`8.62s/it`.  It's nearly 200 times... Do you have any idea? Thank you!

Similar issue: https://github.com/huggingface/transformers/issues/8818

I changed `RandomSampler` to  `SequentialSampler` in the `trainer.py`, but the speed didn't become faster.

I changed
```
tokenized_datasets = load_from_disk(
        "/pathto/dataset"
    )
```
to
```
tokenized_datasets = load_from_disk(
        "/pathto/dataset", keep_in_memory=True
    )
```
and obtained normal speed. It's seems that the problem is on the os's speed limit.

Hi ! Currently `save_to_disk` saves one big Arrow file, which causes some slow downs. This has been discussed in #3735 and we'll implement sharding pretty soon to solve this

For now you can try splitting and saving your dataset in several Arrow files. Then you can load them one by one and use `concatenate_datasets` to have your big dataset again and hopefully with a better speed

## Describe the bug
Due to memory problem, I need to save my tokenized datasets locally by CPU and reload it by multi GPU for running training script. However, after I reload it by `load_from_disk` and start training, the speed is extremely slow. It says I need about 1500 hours with 8 A100 cards. Before this, I can run the whole script in one day with a single A100 card.
Since I am try to pre-train a BERT, **my dataset is very large(29058165 rows)**

## Steps to reproduce the bug
```python
tokenized_datasets.save_to_disk(
        "/pathto/dataset"
    )
tokenized_datasets = load_from_disk(
        "/pathto/dataset"
    )
trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_datasets["train"] if training_args.do_train else None,
        eval_dataset=tokenized_datasets["validation"]
        if training_args.do_eval
        else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )
 train_result = trainer.train(resume_from_checkpoint=checkpoint)
```

## Expected results
Without the save and reload process, I only need about one day to run the whole script with one A100 card.

## Actual results
```
[INFO|trainer.py:1290] 2022-05-23 22:49:46,266 >> ***** Running training *****
[INFO|trainer.py:1291] 2022-05-23 22:49:46,266 >>   Num examples = 29058165
[INFO|trainer.py:1292] 2022-05-23 22:49:46,266 >>   Num Epochs = 5
[INFO|trainer.py:1293] 2022-05-23 22:49:46,266 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:1294] 2022-05-23 22:49:46,266 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:1295] 2022-05-23 22:49:46,266 >>   Gradient Accumulation steps = 2
[INFO|trainer.py:1296] 2022-05-23 22:49:46,266 >>   Total optimization steps = 567540
  0%|          | 1/567540 [00:09<1544:49:04,  9.80s/it]
  0%|          | 2/567540 [00:17<1320:00:17,  8.37s/it]
  0%|          | 3/567540 [00:26<1393:10:17,  8.84s/it]
  0%|          | 4/567540 [00:34<1344:56:33,  8.53s/it]
  0%|          | 5/567540 [00:43<1359:36:12,  8.62s/it]
```

## Environment info
```
torch              1.11.0+cu113
torchaudio         0.11.0+cu113
torchvision        0.12.0+cu113
transformers       4.18.0
datasets           2.2.2
```


Update CI deprecated legacy image

Now our CI still uses a deprecated legacy image:
> You’re using a [deprecated Docker convenience image.](https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034) Upgrade to a next-gen Docker convenience image.

This PR updates to next-generation convenience image.

Related to:
- #2955


remove int documentation from logging docs

Removes the `int` documentation from the [logging section](https://huggingface.co/docs/datasets/package_reference/logging_methods#levels) of the docs.


Refactor column mappings for question answering datasets

> Thanks.
>
> I have no visibility about this, but if you say it is more useful for AutoTrain this way...

Thanks for the review @albertvillanova ! Yes, I need some way to reconstruct the original column names with a period because that's how they appear after we flatten the nested columns. In any case, we can adjust this later if needed :)

> Does that mean that we need to change the metadata?

Yes, but this PR takes care of it :)

This PR tweaks the keys in the metadata that are used to define the column mapping for question answering datasets. This is needed in order to faithfully reconstruct column names like `answers.text` and `answers.answer_start` from the keys in AutoTrain.

As observed in https://github.com/huggingface/datasets/pull/4367 we cannot use periods `.` in the keys of the YAML tags, so a decision was made to use a flat mapping with underscores. For QA datasets, however, it's handy to be able to reconstruct the nesting -- hence this PR.

cc @sashavor


Fix metadata validation

Since Python 3.8, the typing module:
- raises an AttributeError when trying to access `__args__` on any type, e.g.: `List.__args__`
- provides the `get_args` function instead: `get_args(List)`

This PR implements a fix for Python >=3.8 whereas maintaining backward compatibility.


Fix bug in gem dataset for wiki_auto_asset_turk config

This PR fixes some URLs.

Fix #4386.


Set builder name from module instead of class

Now the builder name attribute is set from from the builder class name.

This PR sets the builder name attribute from the module name instead. Some motivating reasons:
- The dataset ID is relevant and unique among all datasets and this is directly related to the repository name, i.e., the name of the directory containing the dataset
- The name of the module (i.e. the file containing the loading loading script) is already relevant for loading: it must have the same name as its containing directory (related to the dataset ID), as we search for it using its directory name
- On the other hand, the name of the builder class is not relevant for loading: in our code, we just search for a class which is subclass of `DatasetBuilder` (independently of its name). We do not put any constraint on the naming of the builder class and indeed it can have a name completely different from its module/direcotry/dataset_id

IMO it makes more sense to align the caching directory name with the dataset_id/directory/module name instead of the builder class name.

Fix #4381.


device/google/accessory/adk2012 - Git at Google

"git clone https://android.googlesource.com/device/google/accessory/adk2012"
 https://android.googlesource.com/device/google/accessory/adk2012/#:~:text=git%20clone%20https%3A//android.googlesource.com/device/google/accessory/adk2012


Bug for wiki_auto_asset_turk from GEM

Something isn't working

Thanks for reporting, @StevenTang1998.

I'm looking into it.

Hi @StevenTang1998,

We have fixed the issue:
- #4389

The fix will be available in our next `datasets` library release. In the meantime, you can incorporate that fix by installing `datasets` from our GitHub repo:
```
pip install git+https://github.com/huggingface/datasets#egg=datasets
```

Thanks for your reply!!
And the totto dataset has the same problem. The url should be change to [https://storage.googleapis.com/totto-public/totto_data.zip](https://storage.googleapis.com/totto-public/totto_data.zip).

Hi again @StevenTang1998,

I don't see any problem when loading `totto` dataset:
```python
In [4]: import datasets
   ...: ds = datasets.load_dataset("totto")
Downloading builder script: 5.58kB [00:00, 5.33MB/s]
Downloading metadata: 2.78kB [00:00, 2.96MB/s]
Using custom data configuration default
Downloading and preparing dataset totto/default (download: 179.03 MiB, generated: 706.59 MiB, post-processed: Unknown size, total: 885.62 MiB) to .../.cache/huggingface/datasets/totto/default/1.0.0/263c85871e5451bc892c65ca0306c0629eb7beb161e0eb998f56231562335dd2...
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 188M/188M [00:32<00:00, 5.77MB/s]
Dataset totto downloaded and prepared to .../.cache/huggingface/datasets/totto/default/1.0.0/263c85871e5451bc892c65ca0306c0629eb7beb161e0eb998f56231562335dd2. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 147.95it/s]

In [5]: ds
Out[5]:
DatasetDict({
    train: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 120761
    })
    validation: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 7700
    })
    test: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 7700
    })
})
```

Sorry, I didn't express it clearly. It's the totto dataset from gem.
datasets.load_dataset('gem', 'totto')

## Describe the bug
The script of wiki_auto_asset_turk for GEM may be out of date.

## Steps to reproduce the bug
```python
import datasets
datasets.load_dataset('gem', 'wiki_auto_asset_turk')
```

## Actual results
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/load.py", line 1731, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 640, in download_and_prepare
    self._download_and_prepare(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 1158, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 707, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/tangtianyi/.cache/huggingface/modules/datasets_modules/datasets/gem/982a54473b12c6a6e40d4356e025fb7172a5bb2065e655e2c1af51f2b3cf4ca1/gem.py", line 538, in _split_generators
    dl_dir = dl_manager.download_and_extract(_URLs[self.config.name])
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 416, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 294, in download
    downloaded_path_or_paths = map_nested(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 351, in map_nested
    mapped = [
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 352, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 288, in _single_map_nested
    return function(data_struct)
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 320, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 234, in cached_path
    output_path = get_from_cache(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 579, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://github.com/facebookresearch/asset/raw/master/dataset/asset.test.orig
```


Test dill

I should point out that the hash will be the same if computed twice with the same code on the same version of dill (after adding huggingface's code that removes line numbers and file names, and sorts globals.) My changes in dill 0.3.5 and ones that I will make in 0.3.6 will result in different pickles than the ones dill 0.3.4 was making. This should still be fine for caching.

Just some comments @lhoestq:

The best practice for testing is to have a `test_<filename>.py` for each `<filename>.py`. Therefore in order to have the filenames aligned, I would propose:
- either renaming `fingerprint.py` to `caching.py`
- or renaming `test_caching.py` to `test_fingerprint.py`

On the other hand, my idea when implementing this test was not to test all the functionalities of the `Hasher`, but just to have a regression test that fails if dill version is > 0.3.4 and the pin in our `setup.py` is not present. Just recall that we had no failing test in our CI when the issue with dill was found on `transformers`.

The objective of this PR is just to have a regression test for that case: I tested and I got `AttributeError: module 'dill._dill' has no attribute 'stack'`

For this regression test, I took into account this comment by @gugarosa: https://github.com/huggingface/datasets/issues/4379#issuecomment-1133131825

There is no equivalent test in `test_caching.py` because our CI did not fail before pinning dill.

Ok I see, renaming it to `test_fingerprint.py` sounds like a good idea :)

Regression test for future releases of `dill`.

Related to #4379.


Refactor download

_The documentation is not available anymore as the PR was closed or merged._

This looks like a breaking change no ?
Also could you explain why it would be better this way ?

The might be only there to help type checkers, but I am not too familiar with the code base to know for sure. I think this might be useful:

https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING

> This looks like a breaking change no ?
> Also could you explain why it would be better this way ?

Sorry, @lhoestq, I naively thought it was obvious. I have tried to give some arguments in the motivation of this PR (see above). I can give additional arguments if needed.

This PR performs a refactoring of the download functionalities, by proposing a modular solution and moving them to their own package "download". Some motivating arguments:
- understandability: from a logical partitioning of the library, it makes sense to have all download functionalities grouped together instead of scattered in a much larger directory containing many more different functionalities
- abstraction: the level of abstraction of "download" (higher) is not the same as "utils" (lower); putting different levels of abstraction together, makes dependencies more intricate (potential circular dependencies) and the system more tightly coupled; when the levels of abstraction are clearly separated, the dependencies flow in a neat direction from higher to lower
- architectural: "download" is a domain-specific functionality of our library/application (a dataset builder performs several actions: download, generate dataset and cache it); these functionalities are at the core of our library; on the other hand, "utils" are always a low-level set of functionalities, not directly related to our domain/business core logic (all libraries have "utils"), thus at the periphery of our lib architecture

Also note that when a library is not architecturally designed following simple, neat, clean principles, this has a negative impact on extensibility, making more and more difficult to make enhancements.

As a concrete example in this case, please see: https://app.circleci.com/pipelines/github/huggingface/datasets/12185/workflows/ff25a790-8e3f-45a1-aadd-9d79dfb73c4d/jobs/72860
- After an extension, a circular import is found
- Diving into the cause of this circular import, see the dependency flow, which should be from higher to lower levels of abstraction:
```
ImportError while loading conftest '/home/circleci/datasets/tests/conftest.py'.
tests/conftest.py:12: in <module>
    import datasets
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/__init__.py:37: in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/arrow_dataset.py:59: in <module>
    from . import config
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/config.py:8: in <module>
    from .utils.logging import get_logger
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/utils/__init__.py:30: in <module>
    from .download_manager import DownloadConfig, DownloadManager, DownloadMode
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/utils/download_manager.py:39: in <module>
    from .py_utils import NestedDataStructure, map_nested, size_str
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/utils/py_utils.py:608: in <module>
    if config.DILL_VERSION < version.parse("0.3.5"):
E   AttributeError: module 'datasets.config' has no attribute 'DILL_VERSION'
```

Imports:
- datasets
  - Dataset: lower level than datasets
    - config: lower level than Dataset
      - logger: lower level than config
        - DownloadManager: !!! HIGHER level of abstraction than logger!!

Why when importing logger we require importing DownloadManager?!?
- Logically, it does not make sense
- This is due to an error in the design/architecture of our library:
  - To import the logger, we need to import it from `.utils.logging`
  - To import `.utils.logging` we need to import `.utils`
  - The import of `.utils` require the import of all its submodules defined in `utils.__init__.py`, among them: `.utils.download_manager`!

When putting `logging` and `download_manager` both inside `utils`, in order to import `logging` we need to import `download_manager` first: this is a strong coupling between modules and moreover between modules at different levels of abstraction (to import a lower level module, we require to import a higher level module). Additionally, it is clear that is makes no sense that in order to import `logging` we require to import `download_manager` first.


L

Something isn't working

## Describe the   L
L
## Expected   L
A clear and concise lmll
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


First time trying

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Bug in caching 2 datasets both with the same builder class name

bug

Something isn't working

Hi @NouamaneTazi, thanks for reporting.

Please note that both datasets are cached in the same directory because their loading builder classes have the same name: `class MTOP(datasets.GeneratorBasedBuilder)`.

You should name their builder classes differently, e.g.:
- `MtopDomain`
- `MtopIntent`

Hi @NouamaneTazi, please note that after our fix:
- #4388

we do not consider the class name anymore, but the name of the file where the loading builder class is implemented.

## Describe the bug
The two datasets `mteb/mtop_intent` and `mteb/mtop_domain `use both the same cache folder `.cache/huggingface/datasets/mteb___mtop`. So if you first load `mteb/mtop_intent` then datasets will not load `mteb/mtop_domain`.
If you delete this cache folder and flip the order how you load the two datasets , you will get the opposite datasets loaded (difference is here in terms of the label and label_text).

## Steps to reproduce the bug
```python
import datasets

dataset = datasets.load_dataset("mteb/mtop_intent", "en")
print(dataset['train'][0])
dataset = datasets.load_dataset("mteb/mtop_domain", "en")
print(dataset['train'][0])
```

## Expected results
```
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop_intent/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 920.14it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 1, 'label_text': 'GET_MESSAGE'}
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop_domain/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1307.59it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 0, 'label_text': 'messaging'}
```

## Actual results
```
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 920.14it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 1, 'label_text': 'GET_MESSAGE'}
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1307.59it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 1, 'label_text': 'GET_MESSAGE'}
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.1
- Platform: macOS-12.1-arm64-arm-64bit
- Python version: 3.9.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Pin dill

Hotfix #4379.

CC: @sgugger


Latest dill release raises exception

Something isn't working

Just an additional insight, the latest dill (either 0.3.5 or 0.3.5.1) also broke the hashing/fingerprinting of any mapping function.

For example:
```
from datasets import load_dataset

d = load_dataset("rotten_tomatoes")
d.map(lambda x: x)
```

Returns the standard non-dillable error:
```
Parameter 'function'=<function <lambda> at 0x7fe7d18c9560> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly....
```

@albertvillanova ExamplesTests.test_run_speech_recognition_seq2seq is in which file?

Thanks a lot @gugarosa for the insight: we will incorporate it in our CI as regression testing for future dill releases.

Hi @anivegesana, that test is in `transformers` library:
- https://github.com/huggingface/transformers/blob/main/examples/pytorch/test_pytorch_examples.py#L449
- https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py

@albertvillanova

I did a deep dive into @gugarosa's problem and found the issue and it might be related to the one @sgugger discovered. In dill 0.3.5(.1), I created a new `save_function` that fixes a bug in dill that prevented the pickling of recursive inner functions. It was a more complete solution to the problem that `dill._dill.stack` tried to solve in the internal API of dill. Since `dill._dill.stack` was no longer needed, I removed it. Since datasets copies the `save_function` directly from the dill API, it stops working with the new dill version since `dill._dill.stack` is no longer present and the `save_function` has been updated with new code.

https://github.com/huggingface/datasets/blob/95193ae61e92aa537d0c65d37a1fd9d2393aae89/src/datasets/utils/py_utils.py#L607-L678

~If the dill version is below 0.3.5, you should keep this function. If it is after, you would need to update your copy of `save_function` to use the code I introduced, or manually add a `stack` variable to `dill._dill` if it doesn't exist. Fortunately, in any version of Python 3.7+, dictionaries are always in insertion order and dill no longer supports Python 3.6 or older. So, any globals dictionary saved by dill 0.3.5+ will be deterministic given that the version of dill is held constant and this save_function is unnecessary for newer versions of dill.~

Ah. I see what is happening. I guess a different copy of the function code is needed that sorts the global variables by name.

```py
if dill.__version__.split('.') < ['0', '3', '5']:
    # current save_function code inside here
else:
    # new save_function code inside here with the following line inserted after creating the globals
    globs = {k: globs[k] for k in sorted(globs.keys())}
```

Will look into the test case @sgugger pointed out after that and verify if this is causing the problem.

I am actually looking into rewriting the global variables code in uqfoundation/dill#466 and will keep this in mind and will try to create an easy way to modify the global variables in dill 0.3.6 (for example, sort them by key like datasets does).

Thanks a lot for your investigation @anivegesana.

Yes, we copied-pasted the old `save_function` function from `dill`, just adding a line to make deterministic the order of global variables `globs`.

However, this function has changed a lot from version 0.3.5, after your PR (thank you for the fix in recursiveness, indeed):
- uqfoundation/dill#443

We have to address this change.

If finally your PR to sort global variables is merged into dill 0.3.6, that will make our life easier, as the tweak will no longer be necessary. ;)

I have included a regression test so that we are sure future releases of dill do not break `datasets`:
- #4385

I should note that because Python 3.6 and older are now deprecated and Python 3.7 has insertion order dictionaries, the globals in dill will have a deterministic order, just not sorted. I would still keep it sorted like you have it to help with stability (for example, if someone reorders variables in a file, then sorting the globals would not invalidate the cache.)

It seems that the order is not quite deterministic in IPython. Huggingface datasets seems to do well in Jupyter regardless, so it is not a good idea to remove the sorting. uqfoundation/dill#19

## Describe the bug
As reported by @sgugger, latest dill release is breaking things with Datasets.

```
______________ ExamplesTests.test_run_speech_recognition_seq2seq _______________

self = <multiprocess.pool.ApplyResult object at 0x7fa5981a1cd0>, timeout = None

    def get(self, timeout=None):
        self.wait(timeout)
        if not self.ready():
            raise TimeoutError
        if self._success:
            return self._value
        else:
>           raise self._value
E           TypeError: '>' not supported between instances of 'NoneType' and 'float'
```


Tidy up license metadata for google_wellformed_query, newspop, sick

Amend three licenses on datasets to fit naming convention (lower case, cc licenses include sub-version number). I think that's it - everything else on datasets looks great & super-searchable now!


Fix checksum and bug in irc_disentangle dataset

There was a bug in filepath segment:
- wrong: `jkkummerfeld-irc-disentanglement-fd379e9`
- right: `jkkummerfeld-irc-disentanglement-35f0a40`

Also there was a bug in the checksum of the downloaded file.

This PR fixes these issues.

Fix partially #4376.


irc_disentagle viewer error

DUPLICATED comment from https://github.com/huggingface/datasets/issues/3807:

my code:
```
from datasets import load_dataset

dataset = load_dataset("irc_disentangle", download_mode="force_redownload")
```
however, it produces the same error
```
[38](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=37) if len(bad_urls) > 0:
     [39](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=38)     error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> [40](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=39)     raise NonMatchingChecksumError(error_msg + str(bad_urls))
     [41](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=40) logger.info("All the checksums matched successfully" + for_verification_name)

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/jkkummerfeld/irc-disentanglement/tarball/master']
```
I attempted to use the `ignore_verifications' as such:

```
ds = datasets.load_dataset('irc_disentangle', download_mode="force_redownload", ignore_verifications=True)

Downloading builder script: 12.0kB [00:00, 5.92MB/s]
Downloading metadata: 7.58kB [00:00, 3.48MB/s]
No config specified, defaulting to: irc_disentangle/ubuntu
Downloading and preparing dataset irc_disentangle/ubuntu (download: 112.98 MiB, generated: 60.05 MiB, post-processed: Unknown size, total: 173.03 MiB) to /Users/laylabouzoubaa/.cache/huggingface/datasets/irc_disentangle/ubuntu/1.0.0/0f24ab262a21d8c1d989fa53ed20caa928f5880be26c162bfbc02445dbade7e5...
Downloading data: 118MB [00:09, 12.1MB/s]

Dataset irc_disentangle downloaded and prepared to /Users/laylabouzoubaa/.cache/huggingface/datasets/irc_disentangle/ubuntu/1.0.0/0f24ab262a21d8c1d989fa53ed20caa928f5880be26c162bfbc02445dbade7e5. Subsequent calls will reuse this data.
100%|██████████| 3/3 [00:00<00:00, 675.38it/s]
```
but, this returns an empty set?
```
DatasetDict({
    train: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
    test: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
})
```
not sure what else to try at this point?
Thanks in advanced🤗

Thanks for reporting, @labouz. I'm addressing it.

The issue with checksum and empty dataset has been fixed by:
- #4377

To load the dataset, you should force the re-generation of the dataset from the downloaded file by passing `download_mode="reuse_cache_if_exists"` to `load_dataset`.

In relation with the issue with the dataset viewer, first the dataset should be refactored to support streaming.

the dataviewer shows this message for "ubuntu" - "train", "test", and "validation" splits:
```
Server error
Status code:   400
Exception:     ValueError
Message:       Cannot seek streaming HTTP file

```
it appears to give the same message for the "channel_two" data as well.

I get a Checksums error when using `load_data()` with this dataset. Even with the `download_mode` and `ignore_verifications` options set. i referenced the issue here: https://github.com/huggingface/datasets/issues/3807


Support DataLoader with num_workers > 0 in streaming mode

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4375). All of your documentation changes will be reflected on that endpoint.

Alright this is finally ready for review ! It's quite long I'm sorry, but it's not easy to disentangle everything ^^'

The main additions are in
- src/datasets/formatting/dataset_wrappers/torch_iterable_dataset.py
- src/datasets/iterable_dataset.py
- src/datasets/utils/patching.py

### Issue

It's currently not possible to properly stream a dataset using multiple `torch.utils.data.DataLoader` workers:

- the `TorchIterableDataset` can't be pickled and passed to the subprocesses: https://github.com/huggingface/datasets/issues/3950
- streaming extension is failing: https://github.com/huggingface/datasets/issues/3951
- `fsspec` doesn't work out of the box in subprocesses

### Solution in this PR

I fixed these to enable passing an `IterableDataset` to a  `torch.utils.data.DataLoader` with `num_workers > 0`.

I also had to shard the `IterableDataset` to give each worker a shard, otherwise data would be duplicated. This is implemented in `TorchIterableDataset.__iter__` and uses the new `IterableDataset._iter_shard(shard_idx)` method

I also had to do a few changes the patching that enable streaming in dataset scripts:
- the patches are now always applied - not just for streaming mode. They're applied when a builder is instantiated
- I improved it to also check for renamed modules or attributes (ex: pandas vs pd)
- I grouped all the patches of pathlib.Path into a class `xPath`, so that `Path` outside of dataset scripts stay unchanged - otherwise I didn't change the content of the extended Path methods for streaming
- I fixed a bug with the `pd.read_csv` patch, opening the file in "rb" mode was missing and causing some datasets to not work in streaming mode, and compression inference was missing

### A few details regarding `fsspec` in multiprocessing

From https://github.com/fsspec/filesystem_spec/pull/963#issuecomment-1131709948 :
> Non-async instances might be safe in the forked child, if they hold no open files/sockets etc.; I'm not sure any implementations pass this test!
> If any async instance has been created, the newly forked processes must:
> 1. discard references to locks, threads and event loops and make new ones
> 2. not use any async fsspec instances from the parent process
> 3. clear all class instance caches

Therefore in a DataLoader's worker, I clear the reference to the loop and thread (1). We should be fine for 2 and 3 already since we don't use fsspec class instances from the parent process.

Fix https://github.com/huggingface/datasets/issues/3950
Fix https://github.com/huggingface/datasets/issues/3951

TODO:
- [x] fix tests


extremely slow processing when using a custom dataset

Something isn't working

question

Further information is requested

Hi !

My guess is that some examples in your dataset are bigger than your RAM, and therefore loading them in RAM to pass them to `remove_non_indic_sentences` takes forever because it might use SWAP memory.

Maybe several examples in your dataset are grouped together, can you check `len(lang_dataset["train"])` and `lang_dataset["train"].data.nbytes` of both datasets please ? It can also be helpful to check the distribution of lengths of each examples in your dataset.

## processing a custom dataset loaded as .txt file  is extremely slow,  compared to a dataset of similar volume from the hub

I have a large .txt file of 22 GB which i load into HF dataset

`lang_dataset = datasets.load_dataset("text", data_files="hi.txt")`

further i use a pre-processing function to clean the dataset

 `lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)`

the following processing takes astronomical time to process, while hoging all the ram.

similar dataset of same size that's available in the huggingface hub works completely fine. which runs the same processing function and has the same amount of data.
`lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi",  use_auth_token=True)`

the hours predicted to preprocess are as follows:

huggingface hub dataset:  6.5 hrs
custom loaded dataset: 7000 hrs

note: both the datasets are almost actually same, just provided by different sources with has +/- some samples, only one is hosted on the HF hub and the other is downloaded in a text format.

## Steps to reproduce the bug
```
import datasets
import psutil
import sys
import glob
from fastcore.utils import listify
import re
import gc

def remove_non_indic_sentences(example):
    tmp_ls = []
    eng_regex = r'[. a-zA-Z0-9ÖÄÅöäå _.,!"\'\/$]*'
    for e in listify(example['text']):
        matches = re.findall(eng_regex, e)
        for match in (str(match).strip() for match in matches if match not in [""," ", "  ", ",", " ,", ", ", " , "]):
            if len(list(match.split(" "))) > 2:
                e = re.sub(match," ",e,count=1)
        tmp_ls.append(e)
        gc.collect()
    example['clean_text'] = tmp_ls
    return example

lang_dataset = datasets.load_dataset("text", data_files="hi.txt")

lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)

## same thing work much faster when loading similar dataset from hub

lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi", split="train", use_auth_token=True)

lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)

```

## Actual results

similar dataset of same size that's available in the huggingface hub works completely fine. which runs the same processing function and has the same amount of data.
`lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi", use_auth_token=True)

**the hours predicted to preprocess are as follows:**
huggingface hub dataset:  6.5 hrs
custom loaded dataset: 7000 hrs

**i even tried the following:**

-  sharding the large 22gb text files into smaller files and loading
- saving the file to disk and then loading
- using lesser num_proc
- using smaller batch size
- processing without batches ie :  without `batched=True`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2.dev0
- Platform: Ubuntu 20.04 LTS
- Python version: 3.9.7
- PyArrow version:8.0.0


Remove links in docs to old dataset viewer

_The documentation is not available anymore as the PR was closed or merged._

Remove the links in the docs to the no longer maintained dataset viewer.


Check if dataset features match before push in `DatasetDict.push_to_hub`

Fix #4211


Add missing language tags for udhr dataset

Related to #4362.


Add redirect to dataset script in the repo structure page

Following https://github.com/huggingface/hub-docs/pull/146 I added a redirection to the dataset scripts documentation in the repository structure page.


Add long answer candidates to natural questions dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4368). All of your documentation changes will be reflected on that endpoint.

Once we have added `long_answer_candidates` maybe it would be worth to also add the missing `candidate_index` (inside `long_answer`). What do you think, @seirasto ?

Also note the "Data Fields" section in the README is missing the `long_answer` field.

Moreover, there is no instance example in "Data Instances" section.

We could either make these fixes in this PR or in a subsequent PR.

@albertvillanova I've added the missing fields and updated the README to include a data instance and some other things.

Great! I've made the updates to align the README. Please let me know if I missed anything.

As there were many minor little fixes, I thought it would be easier to fix them directly.

I think the loading script is OK now. If it is also validated by another datasets maintainer, I could run the generation of the pre-processed data and then merge this PR into master (once all the tests are green).

CC: @lhoestq

This is a modification of the Natural Questions dataset to include missing information specifically related to long answer candidates. (See here: https://github.com/google-research-datasets/natural-questions#long-answer-candidates). This information is important to ensure consistent comparison with prior work. It does not disturb the rest of the format . @lhoestq @albertvillanova


Remove config names as yaml keys

I included the change from https://github.com/huggingface/datasets/pull/4302 directly in this PR, this way the datasets will be updated right away in the CI (the CI is only triggered when a dataset card is changed)

Alright it's ready now :)

Here is an example for the `ade_corpus_v2` dataset card. Notice the new `configs` key:

https://github.com/huggingface/datasets/blob/76d9a141740a03f6836feb251f6059894b8d8046/datasets/ade_corpus_v2/README.md#L1-L78

CI failures are only related to dataset cards missing some content.

Many datasets have dots in their config names. However it causes issues with the YAML tags of the dataset cards since we can't have dots in YAML keys.

I fix this, I removed the tags separations per config name completely, and have a single flat YAML for all configurations. Dataset search doesn't use this info anyway. I removed all the config names used as YAML keys, and I moved them in under a new `config:` key.

This is related to https://github.com/huggingface/datasets/pull/2362 (internal https://github.com/huggingface/moon-landing/issues/946).

Also removing the dots in the YAML keys would allow us to do as in https://github.com/huggingface/datasets/pull/4302 which removes a hack that replaces all the dots by underscores in the YAML tags.

I also added a test in the CI that checks that all the YAML tags to make sure that:

- they can be parsed using a YAML parser
- they contain only valid YAML tags like languages or task_ids


duplicate

This issue or pull request already exists

Duplicate of:
- #3956

I think you should report that issue to `elasticsearch` library: https://github.com/elastic/elasticsearch-py

  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"

when I run the order:
nohup python3 custom_service.pyc > service.log 2>&1&

the log:
nohup: 忽略输入
Traceback (most recent call last):
  File "/home/xfz/p3_custom_test/custom_service.py", line 55, in <module>
  File "/home/xfz/p3_custom_test/custom_service.py", line 48, in doInitialize
  File "custom_impl.py", line 286, in custom_setup
  File "custom_impl.py", line 127, in create_es_index
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/__init__.py", line 345, in __init__
    ssl_show_warn=ssl_show_warn,
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 105, in client_node_configs
    node_configs = hosts_to_node_configs(hosts)
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 154, in hosts_to_node_configs
    node_configs.append(host_mapping_to_node_config(host))
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 221, in host_mapping_to_node_config
    return NodeConfig(**options)  # type: ignore
TypeError: __init__() missing 1 required positional argument: 'scheme'
[1]+  退出 1                nohup python3 custom_service.pyc > service.log 2>&1

custom_service_pyc can't running


Remove dots in config names

Closing in favor of https://github.com/huggingface/datasets/pull/4367

20+ datasets have dots in their config names. However it causes issues with the YAML tags of the dataset cards since we can't have dots in YAML keys.

This is related to https://github.com/huggingface/datasets/pull/2362 (internal https://github.com/huggingface/moon-landing/issues/946).

Also removing the dots in the config names would allow us to merge https://github.com/huggingface/datasets/pull/4302 which removes a hack that replaces all the dots by underscores in the YAML tags.

I also added a test in the CI that checks that all the YAML tags to make sure that:
- they can be parsed using a YAML parser
- they contain only valid YAML tags like `languages` or `task_ids`
- they contain valid config names (no invalid characters `<>:/\|?*.`)


Support complex feature types as `features` in packaged loaders

This PR adds `table_cast` to the packaged loaders to fix casting to the `Image`/`Audio`, `ArrayND` and `ClassLabel` types. If these types are not present in the `builder.config.features` dictionary, the built-in `pa.Table.cast` is used for better performance. Additionally, this PR adds `cast_storage` to `ClassLabel` to support the string to int conversion in `table_cast` and ensure that integer labels are in a valid range.

Fix https://github.com/huggingface/datasets/issues/4210

This PR is also a solution for these (popular) discussions: https://discuss.huggingface.co/t/converting-string-label-to-int/2816 and https://discuss.huggingface.co/t/class-labels-for-custom-datasets/15130/2

TODO:
* [x] tests


The dataset preview is not available for this split.

dataset-viewer

Related to the dataset viewer on huggingface.co

Hi!  A dataset has to be streamable to work with the viewer. I did a quick test, and yours is, so this might be a bug in the viewer. cc @severo

Looking at it. The message is now:

```
Message:       cannot cache function '__shear_dense': no locator available for file '/src/services/worker/.venv/lib/python3.9/site-packages/librosa/util/utils.py'
```

so possibly it's related to the libraries versions?

Maybe this SO thread can help: https://stackoverflow.com/questions/59290386/runtimeerror-at-cannot-cache-function-shear-dense-no-locator-available-fo

Same error for https://huggingface.co/datasets/LIUM/tedlium/viewer/release1/test. cc @sanchit-gandhi . I'm on it

Fixed in the datasets viewer, by setting the `NUMBA_CACHE_DIR` env var to a writable directory.

https://huggingface.co/datasets/Roh/ryanspeech/viewer/male/train

<img width="1538" alt="Capture d’écran 2022-06-08 à 11 30 08" src="https://user-images.githubusercontent.com/1676121/172583285-4cd49a0f-5715-423b-95dd-5f6ace3b2416.png">

https://huggingface.co/datasets/LIUM/tedlium/viewer/

<img width="1538" alt="Capture d’écran 2022-06-08 à 14 31 52" src="https://user-images.githubusercontent.com/1676121/172616897-fbcb7df7-0308-4d09-a17d-48826bc91374.png">

I have uploaded the corpus developed by our lab in the speech domain to huggingface [datasets](https://huggingface.co/datasets/Roh/ryanspeech). You can read about the companion paper accepted in interspeech 2021 [here](https://arxiv.org/abs/2106.08468). The dataset works fine but I can't make the dataset preview work. It gives me the following error that I don't understand. Can you help me to begin debugging it?

```
Status code:   400
Exception:     AttributeError
Message:       'NoneType' object has no attribute 'split'
```


Update dataset_infos for UDHN/udhr dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4362). All of your documentation changes will be reflected on that endpoint.

Thanks for contributing @leondz.

The checksums of the files have changed because more languages have been added:
- the new language codes need to be added to the dataset card (README file)
- I think the dataset version number should also be increased, so that users who had previously cached it, get a new dataset download (with the additional languages)

Yep! All done (also fixed the language tags in the README which were iso639-3 instead of the expected bcp47)

I guess the language code CI failure is due to languages.json being a subset of bcp47 (see issue #4304), happy to contribute a solution here, e.g. autogeneration of the lang list from the relevant isos and the ietf bcp47 subtag register or full code for validation

> Thanks again for your contribution, @leondz.
>
> Yes, I think it is OK to set version 1.0.0 (as previous was 0.0.0).
>
> One of the CI failures is related to dummy data: once you have updated the dataset version, the dummy_data ZIP file should be moved from "dummy/0.0.0/dummy_data.zip" to "dummy/1.0.0/dummy_data.zip".

Oh, thanks, I missed that one

> Other CI failure is related to missing languages in our resources file. This has been addressed in this PR:
>
> * #4371
>
> You should merge master branch into your feature branch to incorporate that fix.

Yeah, I saw this :) I already have the merge, thanks. I'm talking about the longer-term picture: every time another language code comes up (e.g. da-bornholm or es-VE), the json will need updating, because the current approach is non-exhaustive manual whitelisting instead of relying on the established bcp standard.

Checksum update to `udhr` for issue #4361


Use config_id to check split sizes instead of config name

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4463). All of your documentation changes will be reflected on that endpoint.

Fix https://github.com/huggingface/datasets/issues/4462


NonMatchingSplitsSizesError when passing a dataset configuration parameter

bug

Something isn't working

As noticed in https://github.com/huggingface/datasets/pull/4125 when a dataset config class has a parameter that reduces the number of examples (e.g. named `max_examples`), then loading the dataset and passing `max_examples` raises `NonMatchingSplitsSizesError`.

This is because it will check for expected the number of examples of the config with the same name without taking into account the `max_examples` parameter. This can be fixed by checking the expected number of examples using the **config id** instead of name. Indeed the config id corresponds to the config name + an optional suffix that depends on the config parameters


AttributeError: module 'datasets' has no attribute 'load_dataset'

Something isn't working

## Describe the bug
I have piped install datasets, but this package doesn't have these attributes: load_dataset, load_metric.

## Environment info
- `datasets` version: 1.9.0
- Platform: Linux-5.13.0-44-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.6.13
- PyArrow version: 6.0.1


Drop Python 3.6 support

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4460). All of your documentation changes will be reflected on that endpoint.

Remove the fallback imports/checks in the code needed for Python 3.6 and update the requirements/CI files.


Add and fix language tags for udhr dataset

_The documentation is not available anymore as the PR was closed or merged._

Related to #4362.


First draft of the docs for TF + Datasets

I might cc a few of the other TF people to take a look when this is closer to being finished, but it's still a draft for now.


Workflow for Tabular data

enhancement

New feature or request

generic discussion

Generic discussion on the library

Tabular data are treated very differently than data for NLP, audio, vision, etc. and therefore the worflow for tabular data in `datasets` is not ideal.

For example for tabular data, it is common to use pandas/spark/dask to process the data, and then load the data into X and y (X is an array of features and y an array of labels), then train_test_split and finally feed the data to a machine learning model.

In `datasets` the workflow is different: we use load_dataset, then map, then train_test_split (if we only have a train split) and we end up with columnar dataset splits, not formatted as X and y.

Right now, it is already possible to convert a dataset from and to pandas, but there are still many things that could improve the workflow for tabular data:
- be able to load the data into X and y
- be able to load a dataset from the output of spark or dask (as far as I know it's usually csv or parquet files on S3/GCS/HDFS etc.)
- support "unsplit" datasets explicitly, instead of putting everything in "train" by default

cc @adrinjalali @merveenoyan feel free to complete/correct this :)

Feel free to also share ideas of APIs that would be super intuitive in your opinion !


Update data URLs in fever dataset

As stated in their website, data owners updated their URLs on 28/04/2022.

This PR updates the data URLs.

Fix #4452.


Dataset Viewer issue for Yaxin/SemEval2015

Closing since it's a duplicate of https://github.com/huggingface/datasets/issues/4453

### Link

_No response_

### Description

the link could not visit

### Owner

_No response_


I understand that the issue is that a remote file (URL) is being loaded as a local file. Right @albertvillanova @lhoestq?

```
Message:       [Errno 2] No such file or directory: 'https://raw.githubusercontent.com/YaxinCui/ABSADataset/main/SemEval2015Task12Corrected/train/restaurants_train.xml'
```

`xml.dom.minidom.parse` is not supported in streaming mode. I opened a PR here to fix it:
https://huggingface.co/datasets/Yaxin/SemEval2015/discussions/1

Please review the PR @WithYouTo and let me know if it works !

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Trying to load FEVER dataset results in NonMatchingChecksumError

bug

Something isn't working

Thanks for reporting @santhnm2. We are fixing it.

Data owners updated their URLs recently. We have to align with them, otherwise you do not download anything (that is why ignore_verifications does not work).

## Describe the bug
Trying to load the `fever` dataset fails with `datasets.utils.info_utils.NonMatchingChecksumError`.

I tried with `download_mode="force_redownload"` but that did not fix the error. I also tried with `ignore_verification=True` but then that raised a `json.decoder.JSONDecodeError`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset('fever', 'v1.0') # Fails with NonMatchingChecksumError
dataset = load_dataset('fever', 'v1.0', download_mode="force_redownload") # Fails with NonMatchingChecksumError
dataset = load_dataset('fever', 'v1.0', ignore_verification=True)` # Fails with JSONDecodeError
```

## Expected results
I expect this call to return with no error raised.

## Actual results
With `ignore_verification=False`:
```
*** datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://s3-eu-west-1.amazonaws.com/fever.public/train.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_dev.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_dev_public.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_test.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/paper_dev.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/paper_test.jsonl']
```
With `ignore_verification=True`:
```
*** json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.3.dev0
- Platform: Linux-4.15.0-50-generic-x86_64-with-glibc2.10
- Python version: 3.8.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Use newer version of multi-news with fixes

Closes #4430.


Update README.md of fquad


Rj

import android.content.DialogInterface;
import android.database.Cursor;
import android.os.Bundle;
import android.view.View;
import android.widget.ArrayAdapter;
import android.widget.Button;
import android.widget.EditText;
import android.widget.Toast;

import androidx.appcompat.app.AlertDialog;
import androidx.appcompat.app.AppCompatActivity;

public class MainActivity extends AppCompatActivity {

    private EditText editTextID;
    private EditText editTextName;
    private EditText editTextNum;

    private String name;
    private int number;
    private String ID;

    private dbHelper db;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        db = new dbHelper(this);

        editTextID = findViewById(R.id.editText1);
        editTextName = findViewById(R.id.editText2);
        editTextNum = findViewById(R.id.editText3);

        Button buttonSave = findViewById(R.id.button);
        Button buttonRead = findViewById(R.id.button2);
        Button buttonUpdate = findViewById(R.id.button3);
        Button buttonDelete = findViewById(R.id.button4);
        Button buttonSearch = findViewById(R.id.button5);
        Button buttonDeleteAll = findViewById(R.id.button6);

        buttonSave.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {

                name = editTextName.getText().toString();

                String num = editTextNum.getText().toString();

                if (name.isEmpty() || num.isEmpty()) {

                    Toast.makeText(MainActivity.this, "Cannot Submit Empty Fields", Toast.LENGTH_SHORT).show();
                } else {
                    number = Integer.parseInt(num);

                    try {
                        // Insert Data
                        db.insertData(name, number);

                        // Clear the fields
                        editTextID.getText().clear();
                        editTextName.getText().clear();
                        editTextNum.getText().clear();

                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                }

            }
        });

        buttonRead.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {

                final ArrayAdapter<String> adapter = new ArrayAdapter<>(MainActivity.this, android.R.layout.simple_list_item_1);
                String name;
                String num;
                String id;

                try {

                    Cursor cursor = db.readData();
                    if (cursor != null && cursor.getCount() > 0) {

                        while (cursor.moveToNext()) {

                            id = cursor.getString(0); // get data in column index 0
                            name = cursor.getString(1); // get data in column index 1
                            num = cursor.getString(2); // get data in column index 2

                            // Add SQLite data to listView
                            adapter.add("ID :- " + id + "\n" +
                                    "Name :- " + name + "\n" +
                                    "Number :- " + num + "\n\n");

                        }

                    } else {

                        adapter.add("No Data");
                    }
                    cursor.close();

                } catch (Exception e) {
                    e.printStackTrace();
                }

                // show the saved data in alertDialog
                AlertDialog.Builder builder = new AlertDialog.Builder(MainActivity.this);
                builder.setTitle("SQLite saved data");
                builder.setIcon(R.mipmap.app_icon_foreground);
                builder.setAdapter(adapter, new DialogInterface.OnClickListener() {
                    @Override
                    public void onClick(DialogInterface dialog, int which) {

                    }
                });

                builder.setPositiveButton("OK", new DialogInterface.OnClickListener() {
                    @Override
                    public void onClick(DialogInterface dialog, int which) {

                        dialog.cancel();
                    }
                });

                AlertDialog dialog = builder.create();
                dialog.show();

            }
        });

        buttonUpdate.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {

                name = editTextName.getText().toString();

                String num = editTextNum.getText().toString();
                ID = editTextID.getText().toString();

                if (name.isEmpty() || num.isEmpty() || ID.isEmpty()) {

                    Toast.makeText(MainActivity.this, "Cannot Submit Empty Fields", Toast.LENGTH_SHORT).show();
                } else {
                    number = Integer.parseInt(num);

                    try {
                        // Update Data
                        db.updateData(ID, name, number);

                        // Clear the fields
                        editTextID.getText().clear();
                        editTextName.getText().clear();
                        editTextNum.getText().clear();

                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                }

            }
        });

        buttonDelete.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {

                ID = editTextID.getText().toString();

                if (ID.isEmpty()) {

                    Toast.makeText(MainActivity.this, "Please enter the ID", Toast.LENGTH_SHORT).show();
                } else {

                    try {
                        // Delete Data
                        db.deleteData(ID);

                        // Clear the fields
                        editTextID.getText().clear();
                        editTextName.getText().clear();
                        editTextNum.getText().clear();

                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                }

            }
        });

        buttonDeleteAll.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {

                // Delete all data
                // You can simply delete all the data by calling this method --> db.deleteAllData();
                // You can try this also
                AlertDialog.Builder builder = new AlertDialog.Builder(MainActivity.this);
                builder.setIcon(R.mipmap.app_icon_foreground);
                builder.setTitle("Delete All Data");
                builder.setCancelable(false);
                builder.setMessage("Do you really need to delete your all data ?");
                builder.setPositiveButton("Yes", new DialogInterface.OnClickListener() {
                    @Override
                    public void onClick(DialogInterface dialog, int which) {

                        // User confirmed , now you can delete the data
                        db.deleteAllData();

                        // Clear the fields
                        editTextID.getText().clear();
                        editTextName.getText().clear();
                        editTextNum.getText().clear();
                    }
                });
                builder.setNegativeButton("No", new DialogInterface.OnClickListener() {
                    @Override
                    public void onClick(DialogInterface dialog, int which) {

                        // user not confirmed
                        dialog.cancel();
                    }
                });

                AlertDialog dialog = builder.create();
                dialog.show();

            }
        });

        buttonSearch.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {

                ID = editTextID.getText().toString();

                if (ID.isEmpty()) {

                    Toast.makeText(MainActivity.this, "Please enter the ID", Toast.LENGTH_SHORT).show();
                } else {

                    try {
                        // Search data
                        Cursor cursor = db.searchData(ID);
                        if (cursor.moveToFirst()) {

                            editTextName.setText(cursor.getString(1));
                            editTextNum.setText(cursor.getString(2));
                            Toast.makeText(MainActivity.this, "Data successfully searched", Toast.LENGTH_SHORT).show();

                        } else {
                            Toast.makeText(MainActivity.this, "ID not found", Toast.LENGTH_SHORT).show();

                            editTextNum.setText("ID Not found");
                            editTextName.setText("ID not found");

                        }

                        cursor.close();

                    } catch (Exception e) {
                        e.printStackTrace();

                    }
                }

            }
        });
    }
}


New Preprocessing Feature  - Deduplication [Request]

enhancement

New feature or request

Hi! The [datasets_sql](https://github.com/mariosasko/datasets_sql) package lets you easily find distinct rows in a dataset (an example with `SELECT DISTINCT` is in the readme). Deduplication is (still) not part of the official API because it's hard to implement for datasets bigger than RAM while only using the native PyArrow ops.

(Btw, this is a duplicate of https://github.com/huggingface/datasets/issues/2514)

**Is your feature request related to a problem? Please describe.**
Many large datasets are full of duplications and it has been shown that deduplicating datasets can lead to better performance while training, and more truthful evaluation at test-time.

A feature that allows one to easily deduplicate a dataset can be cool!

**Describe the solution you'd like**
We can define a function and keep only the first/last data-point that yields the value according to this function.

**Describe alternatives you've considered**
The clear alternative is to repeat a clear boilerplate every time someone want to deduplicate a dataset.


Minor fixes/improvements in `scene_parse_150` card

Add `paperswithcode_id` and fix some links in the `scene_parse_150` card.


Add missing kwargs to docstrings


Fix missing args in docstring of load_dataset_builder

Currently, the docstring of `load_dataset_builder` only contains the first parameter `path` (no other):
- https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/loading_methods#datasets.load_dataset_builder.path


Fix kwargs in docstrings

To fix the rendering of `**kwargs` in docstrings, a parentheses must be added afterwards.

See:
- huggingface/doc-builder/issues/235


Dataset Viewer issue for openclimatefix/nimrod-uk-1km

If I understand correctly, this is due to the key `split` missing in the line https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/blob/main/nimrod-uk-1km.py#L41 of the script.
Maybe @albertvillanova could confirm.

Indeed there are several issues in this dataset loading script.

The one pointed out by @severo: for the default configuration "crops": https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/blob/main/nimrod-uk-1km.py#L244
- The download manager downloads `_URL`
- But `_URL` is not defined:  https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/blob/main/nimrod-uk-1km.py#L41
  ```python
  _URL = {'train': []}
  ```
- Afterwards, for each split, a different key in `_ULR` is used, but it only contains one key: "train"
  - "valid" key: https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/blob/main/nimrod-uk-1km.py#L260
  - "test key: https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/blob/main/nimrod-uk-1km.py#L269

These keys do not exist inside `_URL`, thus the error message reported in the viewer:
```
Exception:     KeyError
Message:       'valid'
```

Would anyone want to submit a Hub PR (or open a Discussion for the authors to be aware) to this dataset? https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km

Hi, I'm the main author for that dataset, so I'll work on updating it! I was working on debugging some stuff awhile ago, which is what broke it.

I've opened a Discussion page, so that we can ask/answer and propose fixes until the script works properly: https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/discussions/1

CC: @julien-c @jacobbieker


Dataset Viewer issue for amazon_polarity

Not sure what happened 😬, but it's fixed

### Link

https://huggingface.co/datasets/amazon_polarity/viewer/amazon_polarity/test

### Description

For some reason the train split is OK but the test split is not for this dataset:

```
Server error
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/cache/modules/datasets_modules/datasets/amazon_polarity/__init__.py'
```

### Owner

No


Dataset Viewer issue for aeslc

### Link

https://huggingface.co/datasets/aeslc

### Description

The dataset viewer can't find `dataset_infos.json` in it's cache:

```
Server error
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/cache/modules/datasets_modules/datasets/aeslc/eb8e30234cf984a58ebe9f205674597ac1db2ec91e7321cd7f36864f7e3671b8/dataset_infos.json'
```

### Owner

No


Update docs around audio and vision

documentation

Improvements or additions to documentation

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4440). All of your documentation changes will be reflected on that endpoint.

> Let me know what you think, especially if we should include some code samples for training a model in the audio/vision sections. I left this out since we already showed it in the NLP section. I want to keep the focus on using Datasets to load and process a dataset, and not so much the training part. Maybe we can add links to the Transformers docs instead?

We plan to address this with end-to-end examples (for each modality) more focused on preprocessing than the ones in the Transformers docs.

As part of the strategy to center the docs around the different modalities, this PR updates the quickstart to include audio and vision examples. This improves the developer experience by making audio and vision content more discoverable, enabling users working in these modalities to also quickly get started without digging too deeply into the docs.

Other changes include:

- Moved the installation guide to the Get Started section because it should be part of a user's onboarding to the library before exploring tutorials or how-to's.
- Updated the native TF code at creating a `tf.data.Dataset` because it was throwing an error. The `to_tensor()` bit was redundant and removing it fixed the error (please double-check me here!).
- Added some UI components to the quickstart so it's easier for users to navigate directly to the relevant section with context about what to expect.
- Reverted to the code tabs for content that don't have any framework-specific text. I think this saves space compared to the code blocks. We'll still use the code blocks if the `torch` text is different from the `tf` text.

Let me know what you think, especially if we should include some code samples for training a model in the audio/vision sections. I left this out since we already showed it in the NLP section. I want to keep the focus on using Datasets to load and process a dataset, and not so much the training part. Maybe we can add links to the Transformers docs instead?


TIMIT won't load after manual download: Errors about files that don't exist

bug

Something isn't working

To have some context, please see:
- #4145

Please, also note that we have recently made some fixes to the script, which are in our GitHub master branch but not yet released:
- #4422
- #4425
- #4436

Thanks Albert! I'll try pulling `datasets` from the git repo instead of PyPI, and/or just wait for the next release.

I'm closing this issue then. Please, feel free to reopen it again if the problem persists.

## Describe the bug

I get the message from HuggingFace that it must be downloaded manually. From the URL provided in the message, I got to UPenn page for manual download. (UPenn apparently want $250? for the dataset??)   ...So, ok, I obtained a copy from a friend and also a smaller version from Kaggle. But in both cases the HF dataloader fails; it is looking for files that don't exist anywhere in the dataset: it is looking for files with lower-case letters like "**test*" (all the filenames in both my copies are uppercase) and certain file extensions that exclude the .DOC which is provided in TIMIT:

## Steps to reproduce the bug
```python
data = load_dataset('timit_asr', 'clean')['train']
```

## Expected results
The dataset should load with no errors.

## Actual results
This error message:
```
  File "/home/ubuntu/envs/data2vec/lib/python3.9/site-packages/datasets/data_files.py", line 201, in resolve_patterns_locally_or_by_urls
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to resolve any data file that matches '['**test*', '**eval*']' at /home/ubuntu/datasets/timit with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

But this is a strange sort of error: why is it looking for lower-case file names when all the TIMIT dataset filenames are uppercase?  Why does it exclude .DOC files when the only parts of the TIMIT data set with "TEST" in them have ".DOC" extensions?   ...I wonder, how was anyone able to get this to work in the first place?

The files in the dataset look like the following:
```
³       PHONCODE.DOC
³       PROMPTS.TXT
³       SPKRINFO.TXT
³       SPKRSENT.TXT
³       TESTSET.DOC
```
...so why are these being excluded by the dataset loader?

## Environment info
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-1060-aws-x86_64-with-glibc2.27
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Fix docstring of inspect_dataset

_The documentation is not available anymore as the PR was closed or merged._

As pointed out by @sgugger:
- huggingface/doc-builder/issues/235


Add missing columns to `blended_skill_talk`

Adds the missing columns to `blended_skill_talk` to align the loading logic with [ParlAI](https://github.com/facebookresearch/ParlAI/blob/main/parlai/tasks/blended_skill_talk/build.py).

Fix #4426


Fix directory names for LDC data in timit_asr dataset

Related to:
- #4422


Load a local cached dataset that has been modified

Something isn't working

Hi! `datasets` caches every modification/loading, so you can either rerun the pipeline up to the `map` call or use `Dataset.from_file(modified_dataset)` to load the dataset directly from the cache file.

## Describe the bug
I have loaded a dataset as follows:
```
d = load_dataset("emotion", split="validation")
```
Afterwards I make some modifications to the dataset via a `map` call:
```
d.map(some_update_func, cache_file_name=modified_dataset)
```
This generates a cached version of the dataset on my local system in the same directory as the original download of the data (/path/to/cache). Running an `ls` returns:
```
modified_dataset
dataset_info.json
emotion-test.arrow
emotion-train.arrow
emotion-validation.arrow
```
as expected. However, when I try to load up the modified cached dataset via a call to
```
modified = load_dataset("emotion", split="validation", data_files="/path/to/cache/modified_dataset")
```
it simply redownloads a new version of the dataset and dumps to a new cache rather than loading up the original modified dataset:
```
Using custom data configuration validation-cdbf51685638421b
Downloading and preparing dataset emotion/validation to ...
```

How am I supposed to load the original modified local cache copy of the dataset?

## Environment info
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Fix dummy dataset generation script for handling nested types of _URLs

It seems that when user specify nested _URLs structures in their dataset script. An error will be raised when generating dummy dataset.

I think the types of all elements in `dummy_data_dict.values()` should be checked because they may have different types.

Linked to issue #4428

PS: I am not sure whether my code fix this issue in a proper way.


Fix script fetching and local path handling in `inspect_dataset` and `inspect_metric`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4433). All of your documentation changes will be reflected on that endpoint.

Fix #4348


Fix builder docstring

Currently, the args of `DatasetBuilder` do not appear in the docs: https://huggingface.co/docs/datasets/v2.1.0/en/package_reference/builder_classes#datasets.DatasetBuilder


Add personaldialog datasets

These test errors are related to issue #4428

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4431). All of your documentation changes will be reflected on that endpoint.

I only made a trivial modification in my commit https://github.com/huggingface/datasets/pull/4431/commits/402c893d35224d7828176717233909ac5f1e7b3e

I have submitted a PR #4434 for the about issue.

> Awesome thanks for adding this dataset :)
>
> I just have one comment about the licensing.
>
> Also it seems that you already have the dataset in https://huggingface.co/datasets/silver/personal_dialog, so it's unnecessary to add it here

Thank you very much for your comment.

So, should I close this PR?

Thanks for fixing the licensing section :)

> So, should I close this PR?

Yes you can close this PR, it's better if your dataset is under your namespace at https://huggingface.co/datasets/silver/personal_dialog :)

Don't forget to update the licensing section on https://huggingface.co/datasets/silver/personal_dialog as well

It seems that all tests are passed


Add ability to load newer, cleaner version of Multi-News

enhancement

New feature or request

Hi! Our versioning is based on Git revisions (the `revision` param in `load_dataset`), so you can just replace the old URL with the new one and open a PR :). I can also give you some pointers if needed.

@mariosasko Awesome thanks! I will do that. Looks like this new version of the data is not available as a zip but as three files (train/dev/test). How is this usually handled in HF Datasets, should `_URL` be a dict with keys `train`, `val`, `test` perhaps?

Yes! Let me help you with more detailed instructions.

In the first step, we need to update the URLs. One of the possible dictionary structures is as follows:
```python
_URLs = {
    "train": {"src": "https://drive.google.com/uc?export=download&id=1wHAWDOwOoQWSj7HYpyJ3Aeud8WhhaJ7P", "tgt": "https://drive.google.com/uc?export=download&id=1QVgswwhVTkd3VLCzajK6eVkcrSWEK6kq"}
    "val": ...
    "test": ...
}
```

(You can use this page to generate direct download links: https://sites.google.com/site/gdocs2direct/)

Then we move to the `split_generators` method:
```python
def _split_generators(self, dl_manager):
    """Returns SplitGenerators."""
    files = dl_manager.download(_URLs)
    return [
        datasets.SplitGenerator(
            name=datasets.Split.TRAIN,
            gen_kwargs={"src_file": files["train"]["src"], "tgt_file": files["train"]["tgt"]},
        ),
        ... # same for val and test
    ]
```
Finally, we adjust the signature of `_generate_examples`:
```python
def _generate_examples(self, src_file, tgt_file):
     """Yields examples."""
     with open(src_file, encoding="utf-8") as src_f, open(
         tgt_file, encoding="utf-8"
     ) as tgt_f:
         ... # the rest is the same
```

And that's it!

PS: Let me know if you need help updating the dummy data and regenerating the metadata file.

Awesome! Thanks for the detailed help, that was straightforward with your instruction. However, I think I am being blocked by this issue: https://github.com/huggingface/datasets/issues/4428

Feel free to open a PR, and I can fix this manually.

**Is your feature request related to a problem? Please describe.**

The [Multi-News dataloader points to the original version of the Multi-News dataset](https://github.com/huggingface/datasets/blob/12540dd75015678ec6019f258d811ee107439a73/datasets/multi_news/multi_news.py#L47), but this has [known errors in it](https://github.com/Alex-Fabbri/Multi-News/issues/11). There exists a [newer version which fixes some of these issues](https://drive.google.com/open?id=1jwBzXBVv8sfnFrlzPnSUBHEEAbpIUnFq).

Unfortunately I don't think you can just replace this old URL with the new one, otherwise this could lead to issues with reproducibility.

**Describe the solution you'd like**

Add a new version to the Multi-News dataloader that points to the updated dataset which has fixes for some known issues.

**Describe alternatives you've considered**

Replace the current URL to the original version to the dataset with the URL to the version with fixes.

**Additional context**

Would be happy to make a PR for this, could someone maybe point me to another dataloader that has multiple versions so I can see how this is handled in `datasets`?


Update builder docstring for deprecated/added arguments

@mishig25 is investigating why deprecated/added do not affect the enclosed text format when used in args docstring: no special formatting appears:
- https://moon-ci-docs.huggingface.co/docs/datasets/pr_4429/en/package_reference/builder_classes#datasets.DatasetBuilder

@albertvillanova please check now 👍
https://moon-ci-docs.huggingface.co/docs/datasets/pr_4429/en/package_reference/builder_classes#datasets.DatasetBuilder

<img width="500" alt="Screenshot 2022-06-06 at 10 20 34" src="https://user-images.githubusercontent.com/11827707/172123471-fab97138-c903-4a71-ab7f-c90e5e43c58f.png">

Thanks @mishig25.

Just one question: is it expected to have the deprecated box right edge not filling all the page width (contrary to the added box)?

> Just one question: is it expected to have the deprecated box right edge not filling all the page width (contrary to the added box)?

Yes, that is expected 😊 because the depreacted box is being bounded by its parent box (the box for `name` argument in the screenshot above)

This PR updates the builder docstring with deprecated/added directives for arguments name/config_name.

Follow up of:
- #4414
- huggingface/doc-builder#233

First merge:
- #4432


Errors when building dummy data if you use nested _URLS

bug

Something isn't working

## Describe the bug
When making dummy data with the `datasets-cli dummy_data` tool,
an error will be raised if you use a nested _URLS in your dataset script.

Traceback (most recent call last):
  File "/home/name/LCCC/datasets/src/datasets/commands/datasets_cli.py", line 43, in <module>
    main()
  File "/home/name/LCCC/datasets/src/datasets/commands/datasets_cli.py", line 39, in main
    service.run()
  File "/home/name/LCCC/datasets/src/datasets/commands/dummy_data.py", line 311, in run
    self._autogenerate_dummy_data(
  File "/home/name/LCCC/datasets/src/datasets/commands/dummy_data.py", line 337, in _autogenerate_dummy_data
    dataset_builder._split_generators(dl_manager)
  File "/home/name/.cache/huggingface/modules/datasets_modules/datasets/personal_dialog/559332bced5eeafa7f7efc2a7c10ce02cee2a8116bbab4611c35a50ba2715b77/personal_dialog.py", line 108, in _split_generators
    data_dir = dl_manager.download_and_extract(urls)
  File "/home/name/LCCC/datasets/src/datasets/commands/dummy_data.py", line 56, in download_and_extract
    dummy_output = self.mock_download_manager.download(url_or_urls)
  File "/home/name/LCCC/datasets/src/datasets/download/mock_download_manager.py", line 130, in download
    return self.download_and_extract(data_url)
  File "/home/name/LCCC/datasets/src/datasets/download/mock_download_manager.py", line 122, in download_and_extract
    return self.create_dummy_data_dict(dummy_file, data_url)
  File "/home/name/LCCC/datasets/src/datasets/download/mock_download_manager.py", line 165, in create_dummy_data_dict
    if isinstance(first_value, str) and len(set(dummy_data_dict.values())) < len(dummy_data_dict.values()):
TypeError: unhashable type: 'list'

## Steps to reproduce the bug

You can use my dataset script implemented here:
https://github.com/silverriver/datasets/blob/2ecd36760c40b8e29b1137cd19b5bad0e19c76fd/datasets/personal_dialog/personal_dialog.py

```python
datasets_cli dummy_data datasets/personal_dialog --auto_generate
```

You can change https://github.com/silverriver/datasets/blob/2ecd36760c40b8e29b1137cd19b5bad0e19c76fd/datasets/personal_dialog/personal_dialog.py#L54
to

```
"train": "https://huggingface.co/datasets/silver/personal_dialog/resolve/main/dev_random.jsonl.gz"
```

before runing the above script to avoid downloading a large training data.

## Expected results
The dummy data should be generated

## Actual results
An error is raised.

It seems that in https://github.com/huggingface/datasets/blob/12540dd75015678ec6019f258d811ee107439a73/src/datasets/download/mock_download_manager.py#L165
We only check if the first item of dummy_data_dict.values() is str.
However, dummy_data_dict.values() may have the type of [str, list, list].
A simple fix would be changing https://github.com/huggingface/datasets/blob/12540dd75015678ec6019f258d811ee107439a73/src/datasets/download/mock_download_manager.py#L165 to

```python
if all([isinstance(value, str) for value in dummy_data_dict.values()]) and len(set(dummy_data_dict.values())) < len(dummy_data_dict.values()):
```

But I don't know if this kinds of change may bring any side effect since I am not sure about the detail logic here.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Linux
- Python version: Python 3.9.10
- PyArrow version: 7.0.0


Add HF.co for PRs/Issues for specific datasets

_The documentation is not available anymore as the PR was closed or merged._

As in https://github.com/huggingface/transformers/pull/17485, issues and PR for datasets under a namespace have to be on the HF Hub


Add loading variable number of columns for different splits

enhancement

New feature or request

Hi! Indeed the column is missing, but you shouldn't get an error? Have you made some modifications (locally) to the loading script? I've opened a PR to add the missing columns to the script.

**Is your feature request related to a problem? Please describe.**
The original dataset `blended_skill_talk` consists of different sets of columns for the different splits: (test/valid) splits have additional data column `label_candidates` that the (train) doesn't have.
When loading such data, an exception occurs at table.py:cast_table_to_schema, because of mismatched columns.


Make extensions case-insensitive in timit_asr dataset

Related to #4422.


Fix DuplicatedKeysError in timit_asr dataset

Fix #4422.


Add new dataset MMChat

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4423). All of your documentation changes will be reflected on that endpoint.

Thanks ! As for https://github.com/huggingface/datasets/pull/4431 please also update the licensing section in https://huggingface.co/datasets/silver/mmchat ;)

Then if it's fine for you feel free to close this PR

Hi, I am adding a new dataset MMChat.
It seems that all tests are passed