Add Validation For README

Good start! Here are some proposed next steps:
- We want the Class structure to reflect the template - so the parser know what section titles to expect and when something has gone wrong
- As a result, we don't need to parse the table of contents, since it will always be the same
- For each section/subsection it would be cool to have a variable saying whether it's filled out or not (when it's either empty or has `[More Information Needed]`)
- `attributes` should probably be `text`

@yjernite @lhoestq

I have added basic validation checking in the class. It works based on a YAML string. The YAML string determines the expected structure and which text is to be checked. The `text` can be true or false showing whether the text has to be checked or not for emptiness. Similarly, each subsection is parsed recursively. I have used print statement currently so that all issues are shown.

Please let me know your thoughts.

I haven't added a variable that keeps a track of whether the text is empty or not but it can be done easliy if required.

This looks like a good start !
Maybe we can use a field named `allow_empty` instead of `text` ?
Also +1 for keeping track of empty texts

Do you think you can have a way to collect all the validation fails of a readme and then raise an error showing all the failures instead of using print ?

Then we can create a `tests/test_dataset_cards.py` test file to make sure all the readmes of the repo are valid !

Hi @lhoestq

I have added changes accordingly. I prepared a list which stores all the errors and raises them at the end. I'm not sure if there is a better way.

Hi @lhoestq @yjernite

Please find the output for the existing READMEs here: http://p.ip.fi/2vYU

Thanks,
Gunjan

Hi @lhoestq

I have added some basic tests, also have restructured `ReadMe` class slightly.

There is one print statement currently, I'm not sure how to remove it. Basically, I want to warn but not stop further validation. I can't append to a list because the `error_list` and `warning_list` are both only present in `validate` method, and this print is present in the `parse` method. This is done when someone has repeated a section multiple times.  For e.g.:

```markdown
---
---

# Dataset Card for FashionMNIST
## Dataset Description
## Dataset Description
```

In this case, I check for validation only in the latest entry.

I can also raise an error (ideal case scenario), but still, it is in the `parse`. Should I add `error_lines` and `warning_lines` as instance variables? That would probably solve the issue.

In tests, I'm using a dummy YAML string for structure, we can also make it into a file but I feel that is not a hard requirement. Let me know your thoughts.

I will add tests for `from_readme` as well.

However, I would love to be able to check the exact message in the test when an error is raised. I checked a couple of methods but couldn't get it working. Let me know if you're aware of a way to do that.

Hi @lhoestq

Thanks for merging. :)
Thanks a lot to you and @yjernite  for guiding me and helping me out.

Yes, I'll also use the next PR for combining the readme and tags validation. ^_^

Hi @lhoestq, @yjernite

This is a simple Readme parser. All classes specific to different sections can inherit `Section` class, and we can define more attributes in each.

Let me know if this is going in the right direction :)

Currently the output looks like this, for `to_dict()` on `FashionMNIST` `README.md`:

```json
{
    "name": "./datasets/fashion_mnist/README.md",
    "attributes": "",
    "subsections": [
        {
            "name": "Dataset Card for FashionMNIST",
            "attributes": "",
            "subsections": [
                {
                    "name": "Table of Contents",
                    "attributes": "- [Dataset Description](#dataset-description)\n  - [Dataset Summary](#dataset-summary)\n  - [Supported Tasks](#supported-tasks-and-leaderboards)\n  - [Languages](#languages)\n- [Dataset Structure](#dataset-structure)\n  - [Data Instances](#data-instances)\n  - [Data Fields](#data-instances)\n  - [Data Splits](#data-instances)\n- [Dataset Creation](#dataset-creation)\n  - [Curation Rationale](#curation-rationale)\n  - [Source Data](#source-data)\n  - [Annotations](#annotations)\n  - [Personal and Sensitive Information](#personal-and-sensitive-information)\n- [Considerations for Using the Data](#considerations-for-using-the-data)\n  - [Social Impact of Dataset](#social-impact-of-dataset)\n  - [Discussion of Biases](#discussion-of-biases)\n  - [Other Known Limitations](#other-known-limitations)\n- [Additional Information](#additional-information)\n  - [Dataset Curators](#dataset-curators)\n  - [Licensing Information](#licensing-information)\n  - [Citation Information](#citation-information)\n  - [Contributions](#contributions)",
                    "subsections": []
                },
                {
                    "name": "Dataset Description",
                    "attributes": "- **Homepage:** [GitHub](https://github.com/zalandoresearch/fashion-mnist)\n- **Repository:** [GitHub](https://github.com/zalandoresearch/fashion-mnist)\n- **Paper:** [arXiv](https://arxiv.org/pdf/1708.07747.pdf)\n- **Leaderboard:**\n- **Point of Contact:**",
                    "subsections": [
                        {
                            "name": "Dataset Summary",
                            "attributes": "Fashion-MNIST is a dataset of Zalando's article images\u2014consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. We intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits.",
                            "subsections": []
                        },
                        {
                            "name": "Supported Tasks and Leaderboards",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        },
                        {
                            "name": "Languages",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        }
                    ]
                },
                {
                    "name": "Dataset Structure",
                    "attributes": "",
                    "subsections": [
                        {
                            "name": "Data Instances",
                            "attributes": "A data point comprises an image and its label.",
                            "subsections": []
                        },
                        {
                            "name": "Data Fields",
                            "attributes": "- `image`: a 2d array of integers representing the 28x28 image.\n- `label`: an integer between 0 and 9 representing the classes with the following mapping:\n  | Label | Description |\n  | --- | --- |\n  | 0 | T-shirt/top |\n  | 1 | Trouser |\n  | 2 | Pullover |\n  | 3 | Dress |\n  | 4 | Coat |\n  | 5 | Sandal |\n  | 6 | Shirt |\n  | 7 | Sneaker |\n  | 8 | Bag |\n  | 9 | Ankle boot |",
                            "subsections": []
                        },
                        {
                            "name": "Data Splits",
                            "attributes": "The data is split into training and test set. The training set contains 60,000 images and the test set 10,000 images.",
                            "subsections": []
                        }
                    ]
                },
                {
                    "name": "Dataset Creation",
                    "attributes": "",
                    "subsections": [
                        {
                            "name": "Curation Rationale",
                            "attributes": "**From the arXiv paper:**\nThe original MNIST dataset contains a lot of handwritten digits. Members of the AI/ML/Data Science community love this dataset and use it as a benchmark to validate their algorithms. In fact, MNIST is often the first dataset researchers try. \"If it doesn't work on MNIST, it won't work at all\", they said. \"Well, if it does work on MNIST, it may still fail on others.\"\nHere are some good reasons:\n- MNIST is too easy. Convolutional nets can achieve 99.7% on MNIST. Classic machine learning algorithms can also achieve 97% easily. Check out our side-by-side benchmark for Fashion-MNIST vs. MNIST, and read \"Most pairs of MNIST digits can be distinguished pretty well by just one pixel.\"\n- MNIST is overused. In this April 2017 Twitter thread, Google Brain research scientist and deep learning expert Ian Goodfellow calls for people to move away from MNIST.\n- MNIST can not represent modern CV tasks, as noted in this April 2017 Twitter thread, deep learning expert/Keras author Fran\u00e7ois Chollet.",
                            "subsections": []
                        },
                        {
                            "name": "Source Data",
                            "attributes": "",
                            "subsections": [
                                {
                                    "name": "Initial Data Collection and Normalization",
                                    "attributes": "**From the arXiv paper:**\nFashion-MNIST is based on the assortment on Zalando\u2019s website. Every fashion product on Zalando has a set of pictures shot by professional photographers, demonstrating different aspects of the product, i.e. front and back looks, details, looks with model and in an outfit. The original picture has a light-gray background (hexadecimal color: #fdfdfd) and stored in 762 \u00d7 1000 JPEG format. For efficiently serving different frontend components, the original picture is resampled with multiple resolutions, e.g. large, medium, small, thumbnail and tiny.\nWe use the front look thumbnail images of 70,000 unique products to build Fashion-MNIST. Those products come from different gender groups: men, women, kids and neutral. In particular, whitecolor products are not included in the dataset as they have low contrast to the background. The thumbnails (51 \u00d7 73) are then fed into the following conversion pipeline:\n1. Converting the input to a PNG image.\n2. Trimming any edges that are close to the color of the corner pixels. The \u201ccloseness\u201d is defined by the distance within 5% of the maximum possible intensity in RGB space.\n3. Resizing the longest edge of the image to 28 by subsampling the pixels, i.e. some rows and columns are skipped over.\n4. Sharpening pixels using a Gaussian operator of the radius and standard deviation of 1.0, with increasing effect near outlines.\n5. Extending the shortest edge to 28 and put the image to the center of the canvas.\n6. Negating the intensities of the image.\n7. Converting the image to 8-bit grayscale pixels.",
                                    "subsections": []
                                },
                                {
                                    "name": "Who are the source image producers?",
                                    "attributes": "**From the arXiv paper:**\nEvery fashion product on Zalando has a set of pictures shot by professional photographers, demonstrating different aspects of the product, i.e. front and back looks, details, looks with model and in an outfit.",
                                    "subsections": []
                                }
                            ]
                        },
                        {
                            "name": "Annotations",
                            "attributes": "",
                            "subsections": [
                                {
                                    "name": "Annotation process",
                                    "attributes": "**From the arXiv paper:**\nFor the class labels, they use the silhouette code of the product. The silhouette code is manually labeled by the in-house fashion experts and reviewed by a separate team at Zalando. Each product Zalando is the Europe\u2019s largest online fashion platform. Each product contains only one silhouette code.",
                                    "subsections": []
                                },
                                {
                                    "name": "Who are the annotators?",
                                    "attributes": "**From the arXiv paper:**\nThe silhouette code is manually labeled by the in-house fashion experts and reviewed by a separate team at Zalando.",
                                    "subsections": []
                                }
                            ]
                        },
                        {
                            "name": "Personal and Sensitive Information",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        }
                    ]
                },
                {
                    "name": "Considerations for Using the Data",
                    "attributes": "",
                    "subsections": [
                        {
                            "name": "Social Impact of Dataset",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        },
                        {
                            "name": "Discussion of Biases",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        },
                        {
                            "name": "Other Known Limitations",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        }
                    ]
                },
                {
                    "name": "Additional Information",
                    "attributes": "",
                    "subsections": [
                        {
                            "name": "Dataset Curators",
                            "attributes": "Han Xiao and Kashif Rasul and Roland Vollgraf",
                            "subsections": []
                        },
                        {
                            "name": "Licensing Information",
                            "attributes": "MIT Licence",
                            "subsections": []
                        },
                        {
                            "name": "Citation Information",
                            "attributes": "@article{DBLP:journals/corr/abs-1708-07747,\n  author    = {Han Xiao and\n               Kashif Rasul and\n               Roland Vollgraf},\n  title     = {Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning\n               Algorithms},\n  journal   = {CoRR},\n  volume    = {abs/1708.07747},\n  year      = {2017},\n  url       = {http://arxiv.org/abs/1708.07747},\n  archivePrefix = {arXiv},\n  eprint    = {1708.07747},\n  timestamp = {Mon, 13 Aug 2018 16:47:27 +0200},\n  biburl    = {https://dblp.org/rec/bib/journals/corr/abs-1708-07747},\n  bibsource = {dblp computer science bibliography, https://dblp.org}\n}",
                            "subsections": []
                        },
                        {
                            "name": "Contributions",
                            "attributes": "Thanks to [@gchhablani](https://github.com/gchablani) for adding this dataset.",
                            "subsections": []
                        }
                    ]
                }
            ]
        }
    ]
}
```

Thanks,
Gunjan


dataset viewer does not work anymore

Thanks for reporting :) We're looking into it

Hi
I normally use this link to see all datasets and how I can load them

https://huggingface.co/datasets/viewer/

Now I am getting

502 Bad Gateway
nginx/1.18.0 (Ubuntu)

could you bring this webpage back ? this was very helpful @lhoestq
thanks for your help


copy.deepcopy os.environ instead of copy

Fixes: https://github.com/huggingface/datasets/issues/2115

- bug fix: using envrion.copy() returns a dict.
- using deepcopy(environ) returns an `_environ` object
- Changing the datatype of the _environ object can break code, if subsequent libraries perform operations using apis exclusive to the environ object, like `environ.getenv()` for example.

Testing:

Tested the change on my terminal:

```
>>> import os
>>> x = deepcopy(os.environ)
>>> y = os.environ
>>> x is y
False
>>> isinstance(x, type(os.environ))
True
>>> z = os.environ.copy()
>>> isinstance(z, type(os.environ))
False
>>> isinstance(z, dict)
True
```


Remove os.environ.copy in Dataset.map

I thought deepcopy on `os.environ` is unsafe (see [this](https://stackoverflow.com/questions/13142972/using-copy-deepcopy-on-os-environ-in-python-appears-broken)), but I can't replicate the behavior described in the linked SO thread.

Closing this one because #2119 has a much cleaner approach.

Replace `os.environ.copy` with in-place modification
Fixes #2115


load_metric from local "glue.py" meet error 'NoneType' object is not callable

> @Frankie123421 what was the resolution to this?

use glue_metric.py instead of glue.py in load_metric

actual_task = "mnli" if task == "mnli-mm" else task
dataset = load_dataset(path='/home/glue.py', name=actual_task)
metric = load_metric(path='/home/glue.py', name=actual_task)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-7ab77a465d81> in <module>
      1 actual_task = "mnli" if task == "mnli-mm" else task
      2 dataset = load_dataset(path='/home/jcli/glue.py', name=actual_task)
----> 3 metric = load_metric(path='/home/jcli/glue.py', name=actual_task)

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/datasets/load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, script_version, **metric_init_kwargs)
    508         keep_in_memory=keep_in_memory,
    509         experiment_id=experiment_id,
--> 510         **metric_init_kwargs,
    511     )
    512

TypeError: 'NoneType' object is not callable

Please help


Creating custom dataset results in error while calling the map() function

Hi,

the `_data` attribute is missing due to `MyDataset.__init__` not calling the parent `__init__`. However, I don't think it's a good idea to subclass the `datasets.Dataset` class (e.g. it's kind of dangerous to override `datasets.Dataset.__getitem__`). Instead, it's better to follow the "association over inheritance" approach with a simple wrapper class that delegates calls to a wrapped `Dataset` (map, etc.). Btw, the library offers the `datasets.Dataset.from_pandas` class method to directly create a `datasets.Dataset` from the dataframe.

calling `map()` of `datasets` library results into an error while defining a Custom dataset.
Reproducible example:
```
import datasets
class MyDataset(datasets.Dataset):

    def __init__(self, sentences):
        "Initialization"
        self.samples = sentences

    def __len__(self):
        "Denotes the total number of samples"
        return len(self.samples)

    def __getitem__(self, index):
        "Generates one sample of data"
        # Select sample
        # Load data and get label
        samples = self.samples[index]

        return samples

def preprocess_function_train(examples):
        inputs = examples
        labels = [example+tokenizer.eos_token for example in examples ]
        inputs = tokenizer(inputs, max_length=30, padding=True, truncation=True)
        labels = tokenizer(labels, max_length=30, padding=True, truncation=True)
        model_inputs = inputs
        model_inputs["labels"] = labels["input_ids"]
        print("about to return")
        return model_inputs

##train["sentence"] is dataframe column
train_dataset = MyDataset(train['sentence'].values.tolist())
train_dataset = train_dataset.map(
            preprocess_function,
            batched = True,
            batch_size=32
        )
```

Stack trace of error:
```
Traceback (most recent call last):
  File "dir/train_generate.py", line 362, in <module>
    main()
  File "dir/train_generate.py", line 245, in main
    train_dataset = train_dataset.map(
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1244, in map
    return self._map_single(
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 149, in wrapper
    unformatted_columns = set(self.column_names) - set(self._format_columns or [])
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 526, in column_names
    return self._data.column_names
AttributeError: 'MyDataset' object has no attribute '_data'
```


The datasets.map() implementation modifies the datatype of os.environ object

In our testing, we noticed that the datasets.map() implementation is modifying the datatype of python os.environ object from '_Environ' to 'dict'.

This causes following function calls to fail as follows:

`
     x = os.environ.get("TEST_ENV_VARIABLE_AFTER_dataset_map", default=None)
    TypeError: get() takes no keyword arguments
`
It looks like the following line in datasets.map implementation introduced this functionality.

https://github.com/huggingface/datasets/blob/0cb1ac06acb0df44a1cf4128d03a01865faa2504/src/datasets/arrow_dataset.py#L1421

Here is the test script to reproduce this error.

```
from datasets import load_dataset
from transformers import AutoTokenizer
import os

def test_train():
    model_checkpoint = "distilgpt2"
    datasets = load_dataset('wikitext', 'wikitext-2-raw-v1')
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    tokenizer.pad_token = tokenizer.eos_token

    def tokenize_function(examples):
        y = tokenizer(examples['text'], truncation=True, max_length=64)
        return y

    x = os.environ.get("TEST_ENV_VARIABLE_BEFORE_dataset_map", default=None)
    print(f"Testing environment variable: TEST_ENV_VARIABLE_BEFORE_dataset_map {x}")
    print(f"Data type of os.environ before datasets.map = {os.environ.__class__.__name__}")
    datasets.map(tokenize_function, batched=True, num_proc=2, remove_columns=["text"])
    print(f"Data type of os.environ after datasets.map = {os.environ.__class__.__name__}")
    x = os.environ.get("TEST_ENV_VARIABLE_AFTER_dataset_map", default=None)
    print(f"Testing environment variable: TEST_ENV_VARIABLE_AFTER_dataset_map {x}")

if __name__ == "__main__":
    test_train()

```


Support for legal NLP datasets (EURLEX, ECtHR cases and EU-REG-IR)

> Awesome thank you :)
> This is really cool
>
> I left a few comments.
>
> Also it looks like the dummy data are quite big (100-200KB each). Can you try to reduce their sizes please ? For example I noticed that all the jsonl files inside the `dummy_data.zip` files have 20 lines. Can you only keep 2 lines instead ?

Hi @lhoestq, I did my best to improve the README files, while I also decreased dummy data examples. I included one more legal dataset.

@lhoestq thanks for your review.

 I shortened the examples in README files and removed `DEFAULT_CONFIG_BUILDER` from `eu_regulatory_ir.py`.

Add support for two legal NLP datasets:

- EURLEX (https://www.aclweb.org/anthology/P19-1636/)
- ECtHR cases (https://arxiv.org/abs/2103.13084)
- EU-REG-IR (https://arxiv.org/abs/2101.10726)


Implement Dataset as context manager

When used as context manager, it would be safely deleted if some exception is raised.

This will avoid
> During handling of the above exception, another exception occurred:


Support for legal NLP datasets (EURLEX and ECtHR cases)

Add support for two legal NLP datasets:
- EURLEX (https://www.aclweb.org/anthology/P19-1636/)
- ECtHR cases (https://arxiv.org/abs/2103.13084)


Compute WER metric iteratively

I discussed with Patrick and I think we could have a nice addition: have a parameter `concatenate_texts` that, if `True`, uses the old implementation.

By default `concatenate_texts` would be `False`, so that sentences are evaluated independently, and to save resources (the WER computation has a quadratic complexity).

Some users might still want to use the old implementation.

@lhoestq @patrickvonplaten are you sure of the parameter name `concatenate_texts`? I was thinking about something like `iter`...

Not sure about the name, if you can improve it feel free to do so ^^'
The old implementation computes the WER on the concatenation of all the input texts, while the new one makes WER measures computation independent for each reference/prediction pair.
That's why I thought of `concatenate_texts`

@lhoestq yes, but the end user does not necessarily know the details of the implementation of the WER computation.

From the end user perspective I think it might make more sense: how do you want to compute the metric?
- all in once, more RAM memory needed?
- iteratively, less RAM requirements?

Because of that I was thinking of something like `iter` or `iterative`...

Personally like `concatenate_texts` better since I feel like `iter` or `iterate` are quite vague

Compute WER metric iteratively to avoid MemoryError.

Fix #2078.


Fix incorrect assertion in builder.py

Hi ! The SplitInfo is not always available. By default you would get `split_info.num_examples == 0`
So unfortunately we can't use this assertion you suggested

> Hi ! The SplitInfo is not always available. By default you would get `split_info.num_examples == 0`
> So unfortunately we can't use this assertion you suggested

Then it would be better to just remove the assertion, because the existing assertion does nothing.

Fix incorrect num_examples comparison assertion in builder.py


Add more issue templates and customize issue template chooser

If you agree, I could also add a link to [Discussions](https://github.com/huggingface/datasets/discussions) in order to reinforce the use of Discussion to make Questions (instead of Issues).

I could also add some other templates: Bug, Feature Request,...

@theo-m we wrote our same comments at the same time... 😉

When opening an issue, it is not evident for the users how to choose a blank issue template. There is a link at the bottom of all the other issue templates (`Don’t see your issue here? Open a blank issue.`), but this is not very visible for users. This is the reason why many users finally chose the `add-dataset` template instead (this is more visible) for issues that indeed are not requesting the addition of a new dataset.

~~With this PR, the default blank issue template would be as visible as the other templates (as the `add-dataset` template), thus making easier for the users to choose it.~~

With this PR:
- more issue templates, besides `add-dataset`, are added: `bug-report` and `feature-request`
- the issue template chooser is customized, so that it now includes a link to `Discussions` for questions


Is there a way to use a GPU only when training an Index  in the process of add_faisis_index?

question

Further information is requested

Motivation - Some FAISS indexes like IVF consist of the training step that clusters the dataset into a given number of indexes. It would be nice if we can use a GPU to do the training step and covert the index back to CPU as mention in [this faiss example](https://gist.github.com/mdouze/46d6bbbaabca0b9778fca37ed2bcccf6).


Metadata validation

> Also I was wondering this is really needed to have `utils.metadata` as a submodule of `datasets` ? This is only used by the CI so I'm not sure we should have this in the actual `datasets` package.

I'm unclear on the suggestion, would you rather have a root-level `./metadata.py` file? I think it's well where it is, if anything we could move it out of utils and into `datasets` as it could be used by e.g. `DatasetDict` so that users can pull the metadata easily rather than have to reparse the readme.

Ok that makes sense if we want to have functions that parse the metadata for users

Hi @theo-m @lhoestq

This seems very interesting. Should I add the descriptions to the PR on `datasets-tagging`? Alternatively, I can also create a google-sheet/markdown table :)

Sorry for the delay in responding.

Thanks,
Gunjan

> Hi @theo-m @lhoestq
>
> This seems very interesting. Should I add the descriptions to the PR on `datasets-tagging`? Alternatively, I can also create a google-sheet/markdown table :)
>
> Sorry for the delay in responding.
>
> Thanks,
> Gunjan

Hi @gchhablani, yes I think at the moment the best solution is for you to write in `datasets-tagging`, as the PR will allow us to discuss and review, even though the work will be ported to this repo in the end.
Or we wait for this to be merged and you reopen the PR here, your call :)

- `pydantic` metadata schema with dedicated validators against our taxonomy
- ci script to validate new changes against this schema and start a vertuous loop
- soft validation on tasks ids since we expect the taxonomy to undergo some changes in the near future

for reference with the current validation we have ~365~ 378 datasets with invalid metadata! full error report [_here_.](https://gist.github.com/theo-m/61b3c0c47fc6121d08d3174bd4c2a26b)


WMT19 Dataset for Kazakh-English is not formatted correctly

dataset bug

A bug in a dataset script provided in the library

Hi ! Thanks for reporting

By looking at the raw `news-commentary-v14.en-kk.tsv` file, it looks like there are at least 17 lines with this issue.
Moreover these issues are not always the same:
- L97 is only `kk` text and must be appended at the end of the `kk` text of the **next** line
- L2897 is only `kk` text and must be appended at the end of the `kk` text of the **previous** line
- L1247 and L1248 are only `kk` texts and must be inserted at the **beginning** of the `kk` text of the next line
- (and there are many others)

It would be nice to have a corrected version of this file ! The file is available in the `wmt/news-commentary` repository on the Datasets Hub here:
https://huggingface.co/datasets/wmt/news-commentary/tree/main/v14/training

Then maybe we can notify the WMT authors and host the corrected version somewhere

In addition to the bug of languages being switched from Issue @415, there are incorrect translations in the dataset because the English-Kazakh translations have a one off formatting error.

The News Commentary v14 parallel data set for kk-en from http://www.statmt.org/wmt19/translation-task.html has a bug here:

> Line 94. The Swiss National Bank, for its part, has been battling with the deflationary effects of the franc’s dramatic appreciation over the past few years.	Швейцарияның Ұлттық банкі өз тарапынан, соңғы бірнеше жыл ішінде франк құнының қатты өсуінің дефляциялық әсерімен күресіп келеді.
>
> Line 95. Дефляциялық күштер 2008 жылы терең және ұзаққа созылған жаһандық дағдарысқа байланысты орын алған ірі экономикалық және қаржылық орын алмасулардың арқасында босатылды.  Жеке қарыз қаражаты үлесінің қысқаруы орталық банктің рефляцияға жұмсалған күш-жігеріне тұрақты соққан қарсы желдей болды.
>
> Line 96. The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008. Private deleveraging became a steady headwind to central bank efforts to reflate.	2009 жылы, алдыңғы қатарлы экономикалардың шамамен үштен бірі бағаның төмендеуін көрсетті, бұл соғыстан кейінгі жоғары деңгей болды.

As you can see, line 95 has only the Kazakh translation which should be part of line 96. This causes all of the following English-Kazakh translation pairs to be one off rendering ALL of those translations incorrect. This issue was not fixed when the dataset was imported to Huggingface. By running this code

```
import datasets
from datasets import load_dataset
dataset = load_dataset('wmt19', 'kk-en')
for key in dataset['train']['translation']:
    if 'The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008.' in key['kk']:
        print(key['en'])
        print(key['kk'])
        break
```
we get:
> 2009 жылы, алдыңғы қатарлы экономикалардың шамамен үштен бірі бағаның төмендеуін көрсетті, бұл соғыстан кейінгі жоғары деңгей болды.
> The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008. Private deleveraging became a steady headwind to central bank efforts to reflate.

which shows that the issue still persists in the Huggingface dataset. The Kazakh sentence matches up to the next English sentence in the dataset instead of the current one.

Please let me know if there's you have any ideas to fix this one-off error from the dataset or if this can be fixed by Huggingface.


Request to remove S2ORC dataset

Hello @kyleclo! Currently, we are getting the data from your bucket, so if you remove it the HF script won't work anymore :)

Until you solve things on your end, @lhoestq suggested we just return a warning message when people try to load that dataset from HF. What would you like it to say?

Hi @kyleclo, as of today, you have not removed your bucket data yet, and therefore HuggingFace can download it from there.

Is it OK? Are you planning to eventually delete it? Thank you.

Hi!  Sorry I missed @yjernite 's previous message, thanks for responding!

Is there an option where we can keep our data in our bucket, but the HF script no longer pulls data from it?

Hi!  I was wondering if it's possible to remove [S2ORC](https://huggingface.co/datasets/s2orc) from hosting on Huggingface's platform?  Unfortunately, there are some legal considerations about how we make this data available.  Happy to add back to Huggingface's platform once we work out those hurdles!  Thanks!


Trouble loading wiki_movies

Hi ! `wiki_movies` was added in `datasets==1.2.0`. However it looks like you have `datasets==1.1.2`.

To use `wiki_movies`, please update `datasets` with
```
pip install --upgrade datasets
```

Thanks a lot! That solved it and I was able to upload a model trained on it as well :)

Hello,
I am trying to load_dataset("wiki_movies") and it gives me this error -

`FileNotFoundError: Couldn't find file locally at wiki_movies/wiki_movies.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/wiki_movies/wiki_movies.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/wiki_movies/wiki_movies.py`

Trying to do `python run_mlm.py \
    --model_name_or_path roberta-base \
    --dataset_name wiki_movies \` also gives the same error.

Is this something on my end? From what I can tell, this dataset was re-added by @lhoestq a few months ago.
Thank you!


citation, homepage, and license fields of `dataset_info.json` are duplicated many times

enhancement

New feature or request

good first issue

Good for newcomers

Thanks for reporting :)
Maybe we can concatenate fields only if they are different.

Currently this is done here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/src/datasets/info.py#L180-L196

This can be a good first contribution to the library.
Please comment if you'd like to improve this and open a PR :)

This happens after a `map` operation when `num_proc` is set to `>1`. I tested this by cleaning up the json before running the `map` op on the dataset so it's unlikely it's coming from an earlier concatenation.

Example result:
```
"citation": "@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n
```

@lhoestq and I believe this is happening due to the fields being concatenated `num_proc` times.


Move Dataset.to_csv to csv module

refactoring

Restructuring existing code without changing its external behavior

Move the implementation of `Dataset.to_csv` to module `datasets.io.csv`.


MIAM dataset - new citation details

Hi !
Looks like there's a unicode error in the new citation in the miam.py file.
Could you try to fix it ? Not sure from which character it comes from though

You can test if it works on your side with
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_miam
```

Hi @lhoestq, I have updated the citations to reference an OpenReview preprint.


Fix deprecated warning message and docstring

documentation

Improvements or additions to documentation

I have a question: what about `dictionary_encode_column_`?
- It is deprecated in Dataset, but it recommends using a non-existing method instead: `Dataset.dictionary_encode_column` does not exist.
- It is NOT deprecated in DatasetDict.

`dictionary_encode_column_ ` should be deprecated since it never worked correctly. It will be removed in a major release.
This has to be deprecated in `DatasetDict` as well.
And `Dataset.dictionary_encode_column` doesn't exist indeed.

Thanks @lhoestq. I have fixed deprecated for `dictionary_encode_column_`.

Fix deprecated warnings:
- Use deprecated Sphinx directive in docstring
- Fix format of deprecated message
- Raise FutureWarning


load_from_disk takes a long time to load local dataset

Hi !
Can you share more information about the features of your dataset ? You can get them by printing `my_dataset.features`
Can you also share the code of your `map` function ?

It is actually just the tokenized `wikipedia` dataset with `input_ids`, `attention_mask`, etc, with one extra column which is a list of integers. The `text` column is removed during tokenization.

```
def add_len_and_seq(example):
    end_idx = example['input_ids'].index(SEP)
    example['actual_len'] = end_idx-1
    seq_len = len(example['input_ids'])

    example['seq'] = [PAD_ID] + [np.uint8(example['some_integer'])]*(end_idx-1) + [PAD_ID]*(seq_len-end_idx)

    return example
```

Is `PAD_ID` a python integer ? You need all the integers in `example['seq']` to have the same type.
Does this work if you remove the `np.uint8` and use python integers instead ?

yup I casted it to `np.uint8` outside the function where it was defined. It was originally using python integers.

Strangely, even when I manually created `np.arrays` of specific `dtypes`, the types in the final `dataset_info.json` that gets written are still `int64`.

Update: I tried creating lists of `int8`s and got the same result.

Yes this is a known issue: #625
We're working on making the precision kept for numpy :)
To specify the precision of the integers, currently one needs to specify the output features with `.map(..., features=output_features)`

Do you know what step is taking forever in the code ?
What happens if you interrupt the execution of the dataset loading ?

After a synchronous discussion, we found that the cache file sizes have an enormous effect on the loading speed: smaller cache files result in faster load times. `num_proc` controls the number of cache files that are being written and is inversely proportional to the individual file size. In other words, increase `num_proc` for smaller cache files :)

Maybe this can be highlighted somewhere in the docs.

I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)


SQuAD version

Hi ! This is 1.1 as specified by the download urls here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/datasets/squad/squad.py#L50-L55

Hi~
I want train on squad dataset. What's the version of the squad? Is it 1.1 or 1.0? I'm new in QA, I don't find some descriptions about it.


fixes issue #1110 by descending further if `obj["_type"]` is a dict

Check metrics


CoNLL 2003 dataset not including German

dataset request

Requesting to add a new dataset

Hello, thanks for all the work on developing and maintaining this amazing platform, which I am enjoying working with!

I was wondering if there is a reason why the German CoNLL 2003 dataset is not included in the [repository](https://github.com/huggingface/datasets/tree/master/datasets/conll2003), since a copy of it could be found in some places on the internet such as GitHub? I could help adding the German data to the hub, unless there are some copyright issues that I am unaware of...

This is considering that many work use the union of CoNLL 2002 and 2003 datasets for comparing cross-lingual NER transfer performance in `en`, `de`, `es`, and `nl`. E.g., [XLM-R](https://www.aclweb.org/anthology/2020.acl-main.747.pdf).

## Adding a Dataset
- **Name:** CoNLL 2003 German
- **Paper:** https://www.aclweb.org/anthology/W03-0419/
- **Data:** https://github.com/huggingface/datasets/tree/master/datasets/conll2003


Fix: Allows a feature to be named "_type"

Nice thank you !
This looks like a pretty simple yet effective fix ;)
Could you just add a test in `test_features.py` to make sure that you can create `features` with a `_type` field and that it is possible to convert it as a dict and reload it ?
```python
from datasets import Features, Value

# We usually use `asdict` on a `DatasetInfo` object which is a dataclass instance that contains the features.
# So we need the conversion of features to dict to work.
# You can test that using `dataclasses._asdict_inner`.
# This is the function used by `dataclasses.asdict` to convert a dataclass instance attribute to a dict
from dataclasses import _asdict_inner

f = Features({"_type": Value("string")})
reloaded_f = Features.from_dict(_asdict_inner(f, dict))
assert reloaded_f == f
```

Sure, i will add a test.
One question: are the posted benchmarks reliable? The extra type check seems to add quite some overhead judging by the relative differences. Do you think this is an issue?

The benchmark has a bit of noise, the values are fine ;)
especially in the change you did since the overhead added is negligible.

Ok, i added the test you described above.

I avoided importing the private `_asdict_inner` method and directly used the `DatasetInfo` class, if this is ok with you. Thanks a lot for your support during this PR!

This PR tries to fix issue #1110. Sorry for taking so long to come back to this.

It's a simple fix, but i am not sure if it works for all possible types of `obj`. Let me know what you think @lhoestq


How to disable making arrow tables in load_dataset ?

Hi ! We plan to add streaming features in the future.

This should allow to load a dataset instantaneously without generating the arrow table. The trade-off is that accessing examples from a streaming dataset must be done in an iterative way, and with an additional (but hopefully minor) overhead.
What do you think about this ?

If you have ideas or suggestions of what you expect from such features as a user, feel free to share them, this is really valuable to us !

People mainly want this feature either because it takes too much time too make arrow tables, or they occupy too much memory on the disk. I think both the problem can be solved if we provide arrow tables themselves on datasets hub. Can we do this currently @lhoestq ?

@lhoestq I think the ```try_from_hf_gcs``` provide the same functionality. What all datasets are available on HF GCS? Are all the datasets on huggingFace datasets hub are made available on GCS, automatically?

Only datasets like wikipedia, wiki40b, wiki_dpr and natural questions are available already processed on the HF google storage. This is used to download directly the arrow file instead of building it from the original data files.

@lhoestq How can we make sure that the data we upload on HuggingFace hub is available in form of preprocessed arrow files ?

We're still working on this :) This will be available soon
Users will be able to put their processed arrow files on the Hub

Hi! You can now use `Dataset.push_to_hub` to store preprocessed files on the Hub.

And to avoid downloading preprocessed files, you can use streaming by setting `streaming=True` in `load_dataset`.

Is there a way to disable the construction of arrow tables, or to make them on the fly as the dataset is being used ?


Fix copy snippet in docs

With this change the lines starting with `...` in the code blocks can be properly copied to clipboard.


Add machine translated multilingual STS benchmark dataset

Hello dear maintainer, are there any comments or questions about this PR?

@iamollas thanks for the feedback. I did not see the template.
I improved it...

also see here https://github.com/PhilipMay/stsb-multi-mt


Add documentaton for dataset README.md files

Hi ! We are using the [datasets-tagging app](https://github.com/huggingface/datasets-tagging) to select the tags to add.

We are also adding the full list of tags in #2107
This covers multilinguality, language_creators, licenses, size_categories and task_categories.

In general if you want to add a tag that doesn't exist (for example for a custom license) you must make it start with `other-` and then a custom tag name.

edit (@theo-m) if you ever find yourself resorting to adding an `other-*` tag, please do ping us somewhere so we can think about adding it to the "official" list :)

@lhoestq hmm - ok thanks for the answer.
To be honest I am not sure if this issue can be closed now.
I just wanted to point out that this should either be documented or linked in the documentation.
If you feel like it is (will be) please just close this.

We're still working on the validation+documentation in this.
Feel free to keep this issue open till we've added them

@lhoestq what is the status on this? Did you add documentation?

Hi ! There's the tagging app at https://huggingface.co/datasets/tagging/ that you can use.
It shows the list of all the tags you can use.

It is based on all the tag sets defined in this folder:
https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources

@lhoestq is there something like this form Models?

I don't think so. Feel free to take a look at the tags of other models (example [here](https://huggingface.co/bert-base-uncased/blob/main/README.md)). But we should definitely have some docs or an app to write the tags. Feel free to open an issue in the `transformers` repo or in the `huggingface_hub` repo so we can discuss this

Hi,
the dataset README files have special headers.
Somehow a documenation of the allowed values and tags is missing.
Could you add that?

Just to give some concrete questions that should be answered imo:
- which values can be passted to multilinguality?
- what should be passed to language_creators?
- which values should licenses have? What do I say when it is a custom license? Should I add a link?
- how should I choose size_categories ? What are valid ranges?
- what are valid task_categories?

Thanks
Philip


change bibtex template to author instead of authors

Trailing whitespace was removed. So more changes in diff than just this fix.

Hi,
IMO when using BibTex Author should be used instead of Authors.
See here: http://www.bibtex.org/Using/de/

Thanks
Philip


Update metadata if dataset features are modified

@lhoestq I'll try to add a test later if you think this approach with the wrapper is good.

Awesome thank you !
Yes this approach with a wrapper is good :)

@lhoestq Added a test. To verify that this change fixes the problem, replace:
```
!pip install datasets==1.5
```
with:
```
!pip install git+https://github.com/mariosasko/datasets-1.git@update-metadata
```
in the first cell of the notebook that is attached to the linked issue.

The CI failure is unrelated I think (building the docs locally doesn't throw an error).

The CI fail for the docs has been fixed on master.
Merging :)

This PR adds a decorator that updates the dataset metadata if a previously executed transform modifies its features.
Fixes #2083


change user permissions to -rw-r--r--

I tried this with `ade_corpus_v2` dataset. `ade_corpus_v2-train.arrow` (downloaded dataset) and `cache-25d41a4d3c2d8a25.arrow` (ran a mapping function on the dataset) both had file permission with octal value of `0644`.

Fix for #2065


Fix max_wait_time in requests

it was handled as a min time, not max cc @SBrandeis


Requesting to add a new dataset

## Adding a Dataset
- **Name:** CUAD - Contract Understanding Atticus Dataset
- **Description:** As one of the only large, specialized NLP benchmarks annotated by experts, CUAD can serve as a challenging research benchmark for the broader NLP community.
- **Paper:** https://arxiv.org/abs/2103.06268
- **Data:** https://github.com/TheAtticusProject/cuad/
- **Motivation:** good domain specific datasets are valuable

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


`concatenate_datasets` throws error when changing the order of datasets to concatenate

Hi,

this bug is related to `Dataset.{remove_columns, rename_column, flatten}` not propagating the change to the schema metadata when the info features are updated, so this line is the culprit:
```python
common_voice_train = common_voice_train.remove_columns(['client_id', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment'])

```
The order is important because the resulting dataset inherits the schema metadata of the first dataset passed to the `concatenate_datasets(...)` function (`pa.concat_tables` [docs](https://arrow.apache.org/docs/python/generated/pyarrow.concat_tables.html)). I'll try to fix this ASAP.

Hey,

I played around with the `concatenate_datasets(...)` function: https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate_datasets#datasets.concatenate_datasets

and noticed that when the order in which the datasets are concatenated changes an error is thrown where it should not IMO.

Here is a google colab to reproduce the error: https://colab.research.google.com/drive/17VTFU4KQ735-waWZJjeOHS6yDTfV5ekK?usp=sharing


Updated card using information from data statement and datasheet

I updated and clarified the REFreSD [data card](https://github.com/mcmillanmajora/datasets/blob/refresd_card/datasets/refresd/README.md) with information from the Eleftheria's [website](https://elbria.github.io/post/refresd/). I added brief descriptions where the initial card referred to the paper, and I also recreated some of the tables in the paper to show relevant dataset statistics.

I'll email Eleftheria to see if she has any comments on the card.


Fix docstrings issues

Fix docstring issues.


Multidimensional arrays in a Dataset

Hi !

This is actually supported ! but not yet in `from_pandas`.
You can use `from_dict` for now instead:
```python
from datasets import Dataset, Array2D, Features, Value
import pandas as pd
import numpy as np

dataset = {
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
}
dataset = Dataset.from_dict(dataset)
```

This will work but to use it with the torch formatter you must specify the `Array2D` feature type in order to tell the shape:
```python
from datasets import Dataset, Array2D, Features, Value
import pandas as pd
import numpy as np

dataset = {
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
}
dataset = Dataset.from_dict(dataset, features=Features({
    "bbox": Array2D(shape=(3, 4), dtype="int64"),
    "input_ids": Value("int64")
}))
dataset.set_format("torch")
print(dataset[0]['bbox'])
# tensor([[1, 2, 3, 4],
#         [1, 2, 3, 4],
#         [1, 2, 3, 4]])
```
If you don't specify the `Array2D` feature type, then the inferred type will be Sequence(Sequence(Value("int64"))) and therefore the torch formatter will return list of tensors

Thanks for the explanation.
With my original DataFrame, I did
```
dataset = dataset.to_dict("list")
```
and then the rest of the transformation from dictionary works just fine.

Hi,

I'm trying to put together a `datasets.Dataset` to be used with LayoutLM which is available in `transformers`. This model requires as input the bounding boxes of each of the token of a sequence. This is when I realized that `Dataset` does not support multi-dimensional arrays as a value for a column in a row.

The following code results in conversion error in pyarrow (`pyarrow.lib.ArrowInvalid: ('Can only convert 1-dimensional array values', 'Conversion failed for column bbox with type object')`)

```
from datasets import Dataset
import pandas as pd
import numpy as np

dataset = pd.DataFrame({
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
})
dataset = Dataset.from_pandas(dataset)
```

Since I wanted to use pytorch for the downstream training task, I also tried a few ways to directly put in a column of 2-D pytorch tensor in a formatted dataset, but I can only have a list of 1-D tensors, or a list of arrays, or a list of lists.

```
import torch
from datasets import Dataset
import pandas as pd

dataset = pd.DataFrame({
    'bbox': [
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]]
    ],
    'input_ids': [1, 2, 3, 4]
})
dataset = Dataset.from_pandas(dataset)

def test(examples):
    return {'bbbox': torch.Tensor(examples['bbox'])}
dataset = dataset.map(test)
print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])

dataset.set_format(type='torch', columns=['input_ids', 'bbox'], output_all_columns=True)
print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])

def test2(examples):
    return {'bbbox': torch.stack(examples['bbox'])}
dataset = dataset.map(test2)

print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])
```

Is is possible to support n-D arrays/tensors in datasets?
It seems that it can also be useful for this [feature request](https://github.com/huggingface/datasets/issues/263).


Refactorize Metric.compute signature to force keyword arguments only

Minor refactoring of Metric.compute signature to force the use of keyword arguments, by using the single star syntax.


MemoryError when computing WER metric

metric bug

A bug in a metric script

Hi ! Thanks for reporting.
We're indeed using `jiwer` to compute the WER.

Maybe instead of calling `jiwer.wer` once for all the preditions/references we can compute the WER iteratively to avoid memory issues ? I'm not too familial with `jiwer` but this must be possible.

Currently the code to compute the WER is defined here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/metrics/wer/wer.py#L93-L94

Hi,

I've just pushed a pull request that is related to this issue https://github.com/huggingface/datasets/pull/2169. It's not iterative, but it should avoid memory errors. It's based on the editdistance python library. An iterative implementation should be as easy as storing scores and words stepwise and dividing at the end.

I see, this was solved by other thread. Ok, let me know if you want to switch the implementation for any reason :)

Thanks for diving into this anyway ^^'
As you said this actually got solved a few days ago

Someone created an issue https://github.com/jitsi/jiwer/issues/40 at jiwer which shows that this is still a problem in the current version. Would be curious to figure out how this can be fixed by jiwer... :) I assume that it runs of out memory because it's trying to compute the WER over (too many) test samples?

Hi !

It's computed iteratively so not sure what could go wrong

https://github.com/huggingface/datasets/blob/8afd0ba8c27800a55ea69d9fcd702dc97d9c16d8/metrics/wer/wer.py#L100-L106

@NiklasHoltmeyer what version of `datasets` are you running ?

One possible explanation might be that it is the user who is passing all the sentences in a single element to `wer.compute`?

As current implementation iterates over the elements of `predictions` and `references`, this can be problematic if `predictions` and `references` contain a single huge element each.

This could be the case, for example, with a single string with all sentences:
```python
result["predicted"] = "One sentence. Other sentence."
```
or with a __double__ nested list of sentence lists
```python
result["predicted"] = [[ ["One sentence."], ["Other sentence"] ]]
```

The user should check the dimensions of the data structure passed to `predictions` and `references`.

Hi all,

in my case I was using and older version of datasets and, as @albertvillanova points out, passing the full list of sentences for the metric calculation. The problem was in the way jiwer implements WER, as it tries to compute WER for the full list at once instead of doing it element-wise. I think that with the latest implementation of datasets, or by using the alternative WER function that I've contributed on this [pull request](https://github.com/huggingface/datasets/pull/2169) there shouldn't be memory errors.

@lhoestq i was using Datasets==1.5.0 with 1.6.1 it worked (atleast the first run) but 1.5.0 is not compatible with my preprocessing. i cant save my dataset to a parquet file while using the latest datasets version

->
```
  File "../preprocess_dataset.py", line 132, in <module>
    pq.write_table(train_dataset.data, f'{resampled_data_dir}/{data_args.dataset_config_name}.train.parquet')
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 1674, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 588, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
TypeError: Argument 'table' has incorrect type (expected pyarrow.lib.Table, got ConcatenationTable)
```

if i do
```
import pyarrow.parquet as pq
...
...
pq.write_table(train_dataset.data, 'train.parquet')
pq.write_table(eval_dataset.data, 'eval.parquet')
```

while using 1.6.1. and its working with 1.5.0

Hi ! You can pass dataset.data.table instead of dataset.data to pq.write_table

This seems to be working so far! Thanks!

Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.


Bump huggingface_hub version

`0.0.2 => 0.0.6`


Issue: Dataset download error

Hi @XuhuiZhou, thanks for reporting this issue.

Indeed, the old links are no longer valid (404 Not Found error), and the script must be updated with the new links to Google Drive.

It would be nice to update the urls indeed !

To do this, you just need to replace the urls in `iwslt2017.py` and then update the dataset_infos.json file with
```
datasets-cli test ./datasets/iwslt2017 --all_configs --save_infos --ignore_verifications
```

Is this a command to update my local files or fix the file Github repo in general? (I am not so familiar with the datasets-cli command here)

I also took a brief look at the **Sharing your dataset** section, looks like I could fix that locally and push it to the repo? I guess we are "canonical" category?

This command will update your local file. Then you can open a Pull Request to push your fix to the github repo :)
And yes you are right, it is a "canonical" dataset, i.e. a dataset script defined in this github repo (as opposed to dataset repositories of users on the huggingface hub)

Hi, thanks for the answer.

I gave a try to the problem today. But I encountered an upload error:

```
git push -u origin fix_link_iwslt
Enter passphrase for key '/home2/xuhuizh/.ssh/id_rsa':
ERROR: Permission to huggingface/datasets.git denied to XuhuiZhou.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

Any insight here?

By the way, when I run the datasets-cli command, it shows the following error, but does not seem to be the error coming from `iwslt.py`

```
Traceback (most recent call last):
  File "/home2/xuhuizh/anaconda3/envs/UMT/bin/datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/home2/xuhuizh/projects/datasets/src/datasets/commands/datasets_cli.py", line 35, in main
    service.run()
  File "/home2/xuhuizh/projects/datasets/src/datasets/commands/test.py", line 141, in run
    try_from_hf_gcs=False,
  File "/home2/xuhuizh/projects/datasets/src/datasets/builder.py", line 579, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home2/xuhuizh/projects/datasets/src/datasets/builder.py", line 639, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/home2/xuhuizh/projects/datasets/src/datasets/utils/info_utils.py", line 32, in verify_checksums
    raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
datasets.utils.info_utils.ExpectedMoreDownloadedFiles: {'https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz'}
```

Hi ! To create a PR on this repo your must fork it and create a branch on your fork. See how to fork the repo [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#start-by-preparing-your-environment).
And to make the command work without the `ExpectedMoreDownloadedFiles` error, you just need to use the `--ignore_verifications` flag.

Hi @XuhuiZhou,

As @lhoestq has well explained, you need to fork HF's repository, create a feature branch in your fork, push your changes to it and then open a Pull Request to HF's upstream repository. This is so because at HuggingFace Datasets we follow a development model called "Fork and Pull Model". You can find more information here:
- [Understanding the GitHub flow](https://guides.github.com/introduction/flow/)
- [Forking Projects](https://guides.github.com/activities/forking/)

Alternatively, if you find all these steps too complicated, you can use the GitHub official command line tool: [GitHub CLI](https://cli.github.com/). Once installed, in order to create a Pull Request, you only need to use this command:
```shell
gh pr create --web
```
This utility will automatically create the fork, push your changes and open a Pull Request, under the hood.

The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?


ConnectionError: Couldn't reach common_voice.py

Hi @LifaSun, thanks for reporting this issue.

Sometimes, GitHub has some connectivity problems. Could you confirm that the problem persists?

@albertvillanova Thanks! It works well now.

When I run:
from datasets import load_dataset, load_metric

common_voice_train = load_dataset("common_voice", "zh-CN", split="train+validation")
common_voice_test = load_dataset("common_voice", "zh-CN", split="test")

Got:
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/master/datasets/common_voice/common_voice.py

Version:
1.4.1

Thanks!  @lhoestq @LysandreJik @thomwolf


Fix size categories in YAML Tags

> It would be great if there was a way to make the task categories consistent too. For this, the streamlit app can look into all the datasets and check for existing categories and show them in the list. This may add some consistency.

We can also update the task lists here: https://github.com/huggingface/datasets-tagging/blob/main/task_set.json

Hi @lhoestq,

Thanks for approving.
How do I add the new categories to the tagging app? What I have added is till `1T` and not `1M`.

I'll also check the task list :)

Thanks,
Gunjan

I think you can change it here: https://github.com/huggingface/datasets-tagging/blob/main/tagging_app.py#L412-L423

Hi @lhoestq,

I have made a PR for size categories on `datasets-tagging`

For tags, I have thought of adding more tags and categories, based on what I know about the existing datasets, any list will not be exhaustive because the contributors can be very specific or very general. Hence, there could be a continuous process of evaluating existing tags and adding more and more.

```json
{
    "image-classification": {
        "description": "image classification tasks",
        "options": [
            "multi-class-classification",
            "multi-label-classification",
            "other"
        ]
    },
    "conditional-text-generation": {
        "description": "data-to-text and text transduction tasks such as translation or summarization",
        "options": [
            "machine-translation",
            "sentence-splitting-fusion",
            "extractive-and-abstractive-summarization",
            "abstractive-summarization",
            "extractive-summarization",
            "multi-document-summarization",
            "table-to-text",
            "text-simplification",
            "explanation-generation",
            "stuctured-to-text",
            "other"
        ]
    },
    "conditional-speech-generation": {
        "description": "speech generation tasks",
        "options": [
            "text-to-speech",
            "speech-translation",
            "other"
        ]
    },

    "conditional-structure-generation":{
        "description": "text or speech to structured data",
        "options":[
            "knowlege-graph-mining",
            "code-generation",
        ]
    },
    "question-answering": {
        "description": "question answering tasks",
        "options": [
            "open-domain-qa",
            "closed-domain-qa",
            "multiple-choice-qa",
            "extractive-qa",
            "abstractive-qa",
            "conversational-qa",
            "multi-document-qa",
            "other"
        ]
    },
    "speech-classification": {
        "description": "speech to label tasks",
        "options": [
            "other"
        ]
    },
    "sequence-modeling": {
        "description": "such as language, speech or dialogue modeling",
        "options": [
            "dialogue-modeling",
            "language-modeling",
            "speech-modeling",
            "multi-turn",
            "slot-filling",
            "other"
        ]
    },
    "speech-recognition": {
        "description": "speech to text tasks",
        "options": [
            "automatic-speech-recognition",
            "other"
        ]
    },
    "structure-prediction": {
        "description": "predicting structural properties of the text, such as syntax",
        "options": [
            "coreference-resolution",
            "named-entity-recognition",
            "part-of-speech-tagging",
            "parsing",
            "sentence-segmentation",
            "single-span-prediction",
            "multi-span-prediction",
            "clause-or-phrase-segmentation",
            "dependency-parsing",
            "constituency-parsing",
            "other"
        ]
    },

    "text-classification": {
        "description": "predicting a class index or boolean value",
        "options": [
            "acceptability-classification",
            "entity-linking-classification",
            "relation-extraction",
            "common-sense-reasoning",
            "fact-checking",
            "intent-classification",
            "multi-class-classification",
            "multi-label-classification",
            "natural-language-inference",
            "semantic-similarity-classification",
            "sentiment-classification",
            "topic-classification",
            "emotion-classification",
            "token-classification",
            "word-sense-disambiguation",
            "offense-classification",
            "hate-speech-classification",
            "language-classification",
            "bias-classification",
            "other"
        ]
    },
    "text-retrieval": {
        "description": "information or text retrieval tasks",
        "options": [
            "document-retrieval",
            "utterance-retrieval",
            "entity-linking-retrieval",
            "fact-checking-retrieval",
            "other"
        ]
    },
    "text-scoring": {
        "description": "text scoring tasks, predicting a real valued score for some text",
        "options": [
            "semantic-similarity-scoring",
            "sentiment-scoring",
            "other"
        ]
    },
    "other": {
        "description": "raw data or other task families",
        "options": [
            "data-mining",
            "raw-text",
            "raw-speech",
            "raw-image",
            "other"
        ]
    }
}
```
I'll sort this when adding it to the .json. Also, I'll change categories according to this if this seems okay to you and commit it to this PR.

I'll also fix spelling others, and some categories which are partially correct, for e.g. `other-machine-translation` to the correct tag.
Lastly, with the options also we can add a description to make it easier for the users to understand what we mean by each option. Example, for "emotion-classification", we can explain what kinds of data we are talking about, or what we mean by "single-span-prediction", etc.

Good idea thank you ! Can you open a PR on datasets-tagging for the tasks as well ?
Also you can update the dataset card with the new tasks categories in another PR if you don't mind

Hi @lhoestq,

Thanks, what all do I need to add to merge this PR?

We can merge this one once the PR on dataset sizes is merged on `datasets-tagging` ;)

Hi @lhoestq,

One problem with this approach is that for datasets like `ccaligned_multilingual`, the infos won't be complete because we don't have all configs. In that case, people might face trouble finding the datatset using the tag. Although, they probably won't be checking the size tag for a dataset like that.

What do you think?

CC @theo-m

For datasets like `ccaligned_multilingual` it's important to have all the tags for users to search and find it. Currently is has the full list of tags (without the config names). So you can actually find the dataset, but you don't know what tag correspond to what configuration.

This PR fixes several `size_categories` in YAML tags and makes them consistent. Additionally, I have added a few more categories after `1M`, up to `1T`. I would like to add that to the streamlit app also.

This PR also adds a couple of infos that I found missing.

The code for generating this:
```python
for dataset in sorted(os.listdir('./datasets/')):
    if '.' not in dataset and dataset not in ['c4', 'csv', 'downloads', 'cc100', 'ccaligned_multilingual', 'celeb_a', 'chr_en', 'emea', 'glue']:
        infos = {}
        stats = {}
        st = ''
        with open(f'datasets/{dataset}/README.md') as f:
            d = f.read()
        start_dash = d.find('---') + 3
        end_dash = d[start_dash:].find('---') + 3
        rest_text = d[end_dash + 3:]
        try:
            full_yaml = OmegaConf.create(d[start_dash:end_dash])
            readme = OmegaConf.to_container(full_yaml['size_categories'], resolve=True)
        except Exception as e:
            print(e)
            continue
        try:
            with open(f'datasets/{dataset}/dataset_infos.json') as f:
                data = json.load(f)
        except Exception as e:
            print(e)
            continue  # Skip those without infos.
        done_set = set([])
        num_keys = len(data.keys())
        for keys in data:
            # dataset = load_dataset('opus100', f'{dirs}')
            total = 0
            for split in data[keys]['splits']:
                total = total + data[keys]['splits'][split]['num_examples']
            if total < 1000:
                st += "- n<1K" + '\n'
                infos[keys] = ["n<1K"]
            elif total >= 1000 and total < 10000:
                infos[keys] = ["1K<n<10K"]
            elif total >= 10000 and total < 100000:
                infos[keys] = ["10K<n<100K"]
            elif total >= 100000 and total < 1000000:
                infos[keys] = ["100K<n<1M"]
            elif total >= 1000000 and total < 10000000:
                infos[keys] = ["1M<n<10M"]
            elif total >= 10000000 and total < 100000000:
                infos[keys] = ["10M<n<100M"]
            elif total >= 100000000 and total < 1000000000:
                infos[keys] = ["100M<n<1B"]
            elif total >= 1000000000 and total < 10000000000:
                infos[keys] = ["1B<n<10B"]
            elif total >= 10000000000 and total < 100000000000:
                infos[keys] = ["10B<n<100B"]
            elif total >= 100000000000 and total < 1000000000000:
                infos[keys] = ["100B<n<1T"]
            else:
                infos[keys] = ["n>1T"]
            done_set = done_set.union(infos[keys])
        if (isinstance(readme, list) and list(infos.values())[0] != readme) or (isinstance(readme, dict) and readme != infos):

            print('-' * 30)
            print(done_set)
            print(f"Changing Full YAML for {dataset}")
            print(OmegaConf.to_yaml(full_yaml))

            if len(done_set) == 1:
                full_yaml['size_categories'] = list(done_set)
            else:
                full_yaml['size_categories'] = dict([(k, v) for k, v in sorted(infos.items(), key=lambda x: x[0])])

            full_yaml_string = OmegaConf.to_yaml(full_yaml)
            print('-' * 30)
            print(full_yaml_string)
            inp = input('Do you wish to continue?(Y/N)')
            if inp == 'Y':
                with open(f'./datasets/{dataset}/README.md', 'w') as f:
                    f.write('---\n')
                    f.write(full_yaml_string)
                    f.write('---')
                    f.write(rest_text)
            else:
                break
```

Note that the lower-bound is inclusive. I'm unsure if this is how it is done in the tagging app.

EDIT:
It would be great if there was a way to make the task categories consistent too. For this, the streamlit app can look into all the datasets and check for existing categories and show them in the list. This may add some consistency.

EDIT:
I understand this will not work for cases where only the infos for some of the configs are present, for example: `ccaligned_multingual` has only 5 out of several configs present, and infos has only information about them.  Hence, I have skipped a few datasets in the code, if there are more such datasets, then I'll ignore them too.


Fixes check of TF_AVAILABLE and TORCH_AVAILABLE

# What is this PR doing

This PR implements the checks if `Tensorflow` and `Pytorch` are available the same way as `transformers` does it. I added the additional checks for the different `Tensorflow` and `torch` versions.  #2068


Fix docstring issues

I think I will stop pushing to this PR, so that it can me merged for today release.

I will open another PR for further fixing docs.

Do you agree, @lhoestq ?

Fix docstring issues.


Multiprocessing is slower than single process

bug

Something isn't working

```python
# benchmark_filter.py
import logging
import sys
import time

from datasets import load_dataset, set_caching_enabled

if __name__ == "__main__":
    set_caching_enabled(False)
    logging.basicConfig(level=logging.DEBUG)

    bc = load_dataset("bookcorpus")

    now = time.time()
    try:
        bc["train"].filter(lambda x: len(x["text"]) < 64, num_proc=int(sys.argv[1]))
    except Exception as e:
        print(f"cancelled: {e}")
    elapsed = time.time() - now

    print(elapsed)
```

Running `python benchmark_filter.py 1` (20min+) is faster than `python benchmark_filter.py 2` (2hrs+)


ArrowInvalid issue for squad v2 dataset

Hi ! This error happens when you use `map` in batched mode and then your function doesn't return the same number of values per column.

Indeed since you're using `map` in batched mode, `prepare_validation_features` must take a batch as input (i.e. a dictionary of multiple rows of the dataset), and return a batch.

However it seems like `tokenized_examples` doesn't have the same number of elements in each field. One field seems to have `1180` elements while `candidate_attention_mask` only has `1178`.

Hello, I am using the huggingface official question answering example notebook (https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb).

In the prepare_validation_features function, I made some modifications to tokenize a new set of quesions with the original contexts and save them in three different list called candidate_input_dis, candidate_attetion_mask and candidate_token_type_ids. When I try to run the next cell for dataset.map, I got the following error:

`ArrowInvalid: Column 1 named candidate_attention_mask expected length 1180 but got length 1178`

My code is as follows:

```
def generate_candidate_questions(examples):
  val_questions = examples["question"]
  candididate_questions = random.sample(datasets["train"]["question"], len(val_questions))
  candididate_questions = [x[:max_length] for x in candididate_questions]
  return candididate_questions

def prepare_validation_features(examples, use_mixing=False):
  pad_on_right = tokenizer.padding_side == "right"
  tokenized_examples = tokenizer(
      examples["question" if pad_on_right else "context"],
      examples["context" if pad_on_right else "question"],
      truncation="only_second" if pad_on_right else "only_first",
      max_length=max_length,
      stride=doc_stride,
      return_overflowing_tokens=True,
      return_offsets_mapping=True,
      padding="max_length",
  )
  if use_mixing:
    candidate_questions = generate_candidate_questions(examples)
    tokenized_candidates = tokenizer(
        candidate_questions if pad_on_right else examples["context"],
        examples["context"] if pad_on_right else candidate_questions,
        truncation="only_second" if pad_on_right else "only_first",
        max_length=max_length,
        stride=doc_stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )

  sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")

  tokenized_examples["example_id"] = []

  if use_mixing:
    tokenized_examples["candidate_input_ids"] = tokenized_candidates["input_ids"]
    tokenized_examples["candidate_attention_mask"] = tokenized_candidates["attention_mask"]
    tokenized_examples["candidate_token_type_ids"] = tokenized_candidates["token_type_ids"]

  for i in range(len(tokenized_examples["input_ids"])):
      sequence_ids = tokenized_examples.sequence_ids(i)
      context_index = 1 if pad_on_right else 0

      sample_index = sample_mapping[i]
      tokenized_examples["example_id"].append(examples["id"][sample_index])

      tokenized_examples["offset_mapping"][i] = [
          (o if sequence_ids[k] == context_index else None)
          for k, o in enumerate(tokenized_examples["offset_mapping"][i])
      ]

  return tokenized_examples

validation_features = datasets["validation"].map(
    lambda xs: prepare_validation_features(xs, True),
    batched=True,
    remove_columns=datasets["validation"].column_names
)
```

I guess this might happen because of the batched=True. I see similar issues in this repo related to arrow table length mismatch error, but in their cases, the numbers vary a lot. In my case, this error always happens when the expected length and unexpected length are very close. Thanks for the help!


Add and fix docstring for NamedSplit

Maybe we should add some other split classes?

Add and fix docstring for `NamedSplit`, which was missing.


PyTorch not available error on SageMaker GPU docker though it is installed

Hey @sivakhno,

how does your `requirements.txt` look like to install the `datasets` library and which version of it are you running? Can you try to install `datasets>=1.4.0`

Hi @philschmid - thanks for suggestion. I am using `datasets==1.4.1`.
I have also tried using `torch=1.6.0` (docker `763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.6.0-gpu-py3 `), but the error is the same.

Could paste the code you use the start your training job and the fine-tuning script you run?

@sivakhno this should be now fixed in `datasets>=1.5.0`.

@philschmid Recently released tensorflow-macos seems to be missing.

I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!


Multiprocessing windows error

Hi ! Thanks for reporting.
This looks like a bug, could you try to provide a minimal code example that reproduces the issue ? This would be very helpful !

Otherwise I can try to run the wav2vec2 code above on my side but probably not this week..

```
from datasets import load_dataset

dataset = load_dataset('glue', 'mrpc', split='train')

updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)

```

I was able to copy some of the shell
This is repeating every half second
Win 10, Anaconda with python 3.8, datasets installed from main branche
```

  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    _check_not_importing_main()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 134, in _check_not_importing_main
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    exitcode = _main(fd, parent_sentinel)
    raise RuntimeError('''
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 125, in _main
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.    return _run_module_code(code, init_globals, run_name,
    prepare(preparation_data)

  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 236, in prepare
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\test.py", line 6, in <module>
    updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1370, in map
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 119, in Pool
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 212, in __init__
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\test.py", line 6, in <module>
    self._repopulate_pool()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 303, in _repopulate_pool
    updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1370, in map
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 326, in _repopulate_pool_static
    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 119, in Pool
    w.start()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\process.py", line 121, in start
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 212, in __init__
    self._popen = self._Popen(self)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 327, in _Popen
    self._repopulate_pool()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 303, in _repopulate_pool
    return Popen(process_obj)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\popen_spawn_win32.py", line 45, in __init__
    return self._repopulate_pool_static(self._ctx, self.Process,
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 326, in _repopulate_pool_static
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 134, in _check_not_importing_main
    w.start()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\process.py", line 121, in start
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...
```

Thanks this is really helpful !
I'll try to reproduce on my side and come back to you

if __name__ == '__main__':

This line before calling the map function stops the error but the script still repeats endless

Indeed you needed `if __name__ == '__main__'` since accoding to [this stackoverflow post](https://stackoverflow.com/a/18205006):

> On Windows the subprocesses will import (i.e. execute) the main module at start. You need to insert an if __name__ == '__main__': guard in the main module to avoid creating subprocesses recursively.

Regarding the hanging issue, can you try to update `dill` and `multiprocess` ?

```
Traceback (most recent call last):
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 791, in move
    os.rename(src, real_dst)
FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\tmpx9fl_jg8' -> 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\cache-9b4f203a63742dfc.arrow'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\cvtrain.py", line 243, in <module>
    common_voice_train = common_voice_train.map(remove_special_characters, remove_columns=["sentence"])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1339, in map
    return self._map_single(
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 203, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1646, in _map_single
    shutil.move(tmp_file.name, cache_file_name)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 805, in move
    copy_function(src, real_dst)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  0%|                                                                                | 0/27771 [00:00<?, ?ex/s]
 File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
OSError: [Errno 22] Invalid argument: 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\cache-9b4f203a63742dfc.arrow'
```

I was adding freeze support before calling the mapping function like this
if __name__ == '__main__':
    freeze_support()
   dataset.map(....)

Usually OSError of an arrow file on windows means that the file is currently opened as a dataset object, so you can't overwrite it until the dataset object falls out of scope.
Can you make sure that there's no dataset object that loaded the `cache-9b4f203a63742dfc.arrow` file ?

Now I understand
The error occures because the script got restarted in another thread, so the object is already loaded.
Still don't have an idea why a new thread starts the whole script again

As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop


Fix docstring rendering of Dataset/DatasetDict.from_csv args

Fix the docstring rendering of Dataset/DatasetDict.from_csv args.