can't load SNLI dataset

I just added the processed files of `snli` on our google storage, so that when you do `load_dataset` it can download the processed files from there :)

We are thinking about having available those processed files for more datasets in the future, because sometimes files aren't available (like for `snli`), or the download speed is too slow, or sometimes the files take time to be processed.

Closing this one. Feel free to re-open if you have other questions :)

`nlp` seems to load `snli` from some URL based on nlp.stanford.edu. This subdomain is frequently down -- including right now, when I'd like to load `snli` in a Colab notebook, but can't.

Is there a plan to move these datasets to huggingface servers for a more stable solution?

Btw, here's the stack trace:

```
File "/content/nlp/src/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/content/nlp/src/nlp/builder.py", line 466, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/content/nlp/src/nlp/datasets/snli/e417f6f2e16254938d977a17ed32f3998f5b23e4fcab0f6eb1d28784f23ea60d/snli.py", line 76, in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 217, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in download
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/py_utils.py", line 190, in map_nested
    return function(data_struct)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in <lambda>
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 198, in cached_path
    local_files_only=download_config.local_files_only,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://nlp.stanford.edu/projects/snli/snli_1.0.zip
```


More faiss control

> Ok, so we're getting rid of the `FaissGpuOptions`?

We support `device=...` because it's simple, but faiss GPU options can be used in so many ways (you can set different gpu options for the different parts of your index for example) that it's probably better to let the user create and configure its index and then use `custom_index=...`

Allow users to specify a faiss index they created themselves, as sometimes indexes can be composite for examples


[Dataset requests] New datasets for Text Classification

help wanted

Extra attention is needed

dataset request

Requesting to add a new dataset

- `nlp` has MR! It's called `rotten_tomatoes`
- SST is part of GLUE, or is that just SST-2?
- `nlp` also has `ag_news`, a popular news classification dataset

I'd also like to see:
- the Yahoo Answers topic classification dataset
- the Kaggle Fake News classification dataset

Thanks @jxmorris12 for pointing this out.

In glue we only have SST-2 maybe we can add separately SST-1.

This is the homepage for the Amazon dataset: https://www.kaggle.com/datafiniti/consumer-reviews-of-amazon-products

Is there an easy way to download kaggle datasets programmatically? If so, I can add this one!

Hi @jxmorris12 for now I think our `dl_manager` does not download from Kaggle.
@thomwolf , @lhoestq

Pretty sure the quora dataset is the same one I implemented here: https://github.com/huggingface/nlp/pull/366

Great list. Any idea if Amazon Reviews has been added?

- ~40 GB of text (sadly no emoji)
- popular MLM pre-training dataset before bigger datasets like WebText https://arxiv.org/abs/1808.01371
- turns out that binarizing the 1-5 star rating leads to great Pos/Neg/Neutral dataset, T5 paper claims to get very high accuracy (98%!) on this with small amount of finetuning https://arxiv.org/abs/2004.14546

Apologies if it's been included (great to see where) and if not, it's one of the better medium/large NLP dataset for semi-supervised learning, albeit a bit out of date.

Thanks!!

cc @sshleifer

On the Amazon Reviews dataset, the original UCSD website has noted these are now updated to include product reviews through 2018 -- actually quite recent compared to many other datasets. Almost certainly the largest NLP dataset out there with labels!
https://jmcauley.ucsd.edu/data/amazon/

Any chance someone has time to onboard this dataset in a HF way?

cc @sshleifer

We are missing a few datasets for Text Classification which is an important field.

Namely, it would be really nice to add:
- TREC-6 dataset (see here for instance: https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.datasets.html#torchnlp.datasets.trec_dataset)  **[done]**
- Yelp-5
- Movie review (Movie Review (MR) dataset [156]) **[done (same as rotten_tomatoes)]**
- SST (Stanford Sentiment Treebank) **[include in glue]**
- Multi-Perspective Question Answering (MPQA) dataset **[require authentication (indeed manual download)]**
- Amazon. This is a popular corpus of product reviews collected from the Amazon website [159]. It contains labels for both binary classification and multi-class (5-class) classification
- 20 Newsgroups. The 20 Newsgroups dataset  **[done]**
- Sogou News dataset **[done]**
- Reuters news. The Reuters-21578 dataset [165] **[done]**
- DBpedia. The DBpedia dataset [170]
- Ohsumed. The Ohsumed collection [171] is a subset of the MEDLINE database
- EUR-Lex. The EUR-Lex dataset
- WOS. The Web Of Science (WOS) dataset **[done]**
- PubMed. PubMed [173]
- TREC-QA. TREC-QA
- Quora. The Quora dataset [180]

All these datasets are cited in https://arxiv.org/abs/2004.03705


🐛[BugFix]fix seqeval

I think this is good but can you detail a bit the behavior before and after your fix?

examples:

input: `['B', 'I', 'I', 'O', 'B', 'I']`
before: `[('B', 0, 0), ('I', 1, 2), ('B', 4, 4), ('I', 5, 5)]`
after: `[('_', 0, 2), ('_', 4, 5)]`

input: `['B-ARGM-LOC', 'I-ARGM-LOC', 'I-ARGM-LOC', 'O', 'B-ARGM-TIME', 'I-ARGM-TIME']`
before: `[('LOC', 0, 2), ('TIME', 4, 5)]`
after: `[('ARGM-LOC', 0, 2), ('ARGM-TIME', 4, 5)]`

This is my test code:

```python
from metrics.seqeval.seqeval import end_of_chunk, start_of_chunk

def before_get_entities(seq, suffix=False):
    """Gets entities from sequence.
    Args:
        seq (list): sequence of labels.
    Returns:
        list: list of (chunk_type, chunk_start, chunk_end).
    """
    if any(isinstance(s, list) for s in seq):
        seq = [item for sublist in seq for item in sublist + ['O']]

    prev_tag = 'O'
    prev_type = ''
    begin_offset = 0
    chunks = []
    for i, chunk in enumerate(seq + ['O']):
        if suffix:
            tag = chunk[-1]
            type_ = chunk.split('-')[0]
        else:
            tag = chunk[0]
            type_ = chunk.split('-')[-1]

        if end_of_chunk(prev_tag, tag, prev_type, type_):
            chunks.append((prev_type, begin_offset, i - 1))
        if start_of_chunk(prev_tag, tag, prev_type, type_):
            begin_offset = i
        prev_tag = tag
        prev_type = type_

    return chunks

def after_get_entities(seq, suffix=False):
    """Gets entities from sequence.
    Args:
        seq (list): sequence of labels.
    Returns:
        list: list of (chunk_type, chunk_start, chunk_end).
    """
    if any(isinstance(s, list) for s in seq):
        seq = [item for sublist in seq for item in sublist + ['O']]

    prev_tag = 'O'
    prev_type = ''
    begin_offset = 0
    chunks = []
    for i, chunk in enumerate(seq + ['O']):
        if suffix:
            tag = chunk[-1]
            type_ = chunk[:-1].rsplit('-', maxsplit=1)[0] or '_'
        else:
            tag = chunk[0]
            type_ = chunk[1:].split('-', maxsplit=1)[-1] or '_'

        if end_of_chunk(prev_tag, tag, prev_type, type_):
            chunks.append((prev_type, begin_offset, i - 1))
        if start_of_chunk(prev_tag, tag, prev_type, type_):
            begin_offset = i
        prev_tag = tag
        prev_type = type_

    return chunks

def main():
    examples_1 = ['B', 'I', 'I', 'O', 'B', 'I']
    print(before_get_entities(examples_1))
    print(after_get_entities(examples_1))
    examples_2 = ['B-ARGM-LOC', 'I-ARGM-LOC', 'I-ARGM-LOC', 'O', 'B-ARGM-TIME', 'I-ARGM-TIME']
    print(before_get_entities(examples_2))
    print(after_get_entities(examples_2))

if __name__ == '__main__':
    main()
```

And we can get more examples not correct, such as:

input: `['B', 'I', 'I-I']`
before: `[('B', 0, 0), ('I', 1, 2)]`
after: `[('_', 0, 1), ('I', 2, 2)]`

input: `['B-ARGM-TIME', 'I-ARGM-TIME', 'I-TIME']`
before: `[('TIME', 0, 2)]`
after: `[('ARGM-TIME', 0, 1), ('TIME', 2, 2)]`

I think i didn't break any thing. Maybe the checks should be restart?

Could you please rebase from master @AlongWY ? This should fix the CI stuff

Indeed the official repo is quite stale. Let's merge it here, thanks @AlongWY

Fix seqeval process labels such as 'B', 'B-ARGM-LOC'


add pandas dataset

Create a dataset from serialized pandas dataframes.
Usage:
```python
from nlp import load_dataset
dset = load_dataset("pandas", data_files="df.pkl")["train"]
```


add from_pandas and from_dict

I added two new methods to the `Dataset` class:
- `from_pandas()` to create a dataset from a pandas dataframe
- `from_dict()` to create a dataset from a dictionary (keys = columns)

It uses the `pa.Table.from_pandas` and `pa.Table.from_pydict` funcitons to do so.
It is also possible to specify the features types via `features=...` if there are ambiguities (null/nan values), otherwise the arrow schema is infered from the data automatically by pyarrow.

One question that I have right now:
+ Should we also add a `save()` method that would write the dataset on the disk ? Right now if we create a `Dataset` using those two new methods, the data are kept in RAM. Then to reload it we can call the `from_file()` method.


Hyperpartisan news detection

Thank you so much for working on this! This is awesome!

How much would it help you if we would remove the manual request?

We are naturally interested in getting some broad idea of how many people and who are using our dataset. But if you consider hosting the dataset yourself, I would rather remove this small barrier on our side (so that we then still get the download count from your library).

This is an interesting aspect indeed!
Do you want to send me an email (see my homepage) and I'll invite you on our slack channel to talk about that?
@ghomasHudson wanna reach out to me as well? I tried to find your email to invite you without success.

Adding the hyperpartisan news detection dataset from PAN. This contains news article text, labelled with whether they're hyper-partisan and why kinds of biases they display.

Implementation notes:
- As with many PAN tasks, the data is hosted on [Zenodo](https://zenodo.org/record/1489920) and must be requested before use. I've used the manual download stuff for this, although the dataset is provided under a Creative Commons Attribution 4.0 International License, so we could host a version if we wanted to?
- The 'bias' attribute doesn't exist for the 'byarticle' configuration. I've added an empty string to the class labels to deal with this. Is there a more standard value for empty data?
- Should we always subclass `nlp.BuilderConfig`?


Add OSCAR dataset

> @pjox I think the tests don't pass because you haven't provided any dummy data (`dummy_data.zip`).

But can I do the dummy data without running `python nlp-cli test datasets/<your-dataset-folder> --save_infos --all_configs` first? 🤔

You make a good point! Do you know how big is it uncompressed?

Hi ! I've been busy but I plan to compute the missing metadata soon !
Looking forward to be able to load a memory mapped version of OSCAR :)

> Hi ! I've been busy but I plan to compute the missing metadata soon !
> Looking forward to be able to load a memory mapped version of OSCAR :)

Amazing! Thanks! 😄

Hi there, are there any plans to complete this issue soon? I'm planning to use this dataset on a project. Let me know if there's anything I can do to help to finish this 🤗

Yes it will be added soon :)
Recently the OSCAR data files were moved to another host. We just need to update the script and compute the dataset_infos.json (it will probably take a few days).

@lhoestq I've seen in oscar.py that it isn't a dataset script with manual download way. Is that correct?
Some time ago, @pjox had some troubles with his servers providing that dataset 'cause it's really huge. Providing it on an automatic download way seems to be a little bit dangerous for me 😄

Now thanks to @pjox 's help OSCAR is hosted on HF's S3, which is probably more robust that the previous servers :)

Also small update on my side:
I launched the computation of the dataset_infos.json file, it will take a few days.

But is there a plan to provide the OSCAR's unshuffled version too?

The one we have on S3 is currently the unshuffled version

I've thought that you won't provide the unshuffled version 'cause this comment on oscar.py:

`# TODO(oscar): Implement unshuffled OSCAR`

That TODO is normal, I haven't touched the python script in months (I haven't had the time, sorry), but I guess @lhoestq fixed the paths if he's already working on the metadata. In any case from now on, only the unshuffled versions of OSCAR will be distributed through the hf/datasets library as in any case it is the version most people use to train language models.

If for any reason, you need the shuffled version it will always be available on the [OSCAR website](https://oscar-corpus.com).

Also future versions of OSCAR will be unshuffled only.

Should we close this PR now that the other one was merged?

@lhoestq just a little detail, is the Oscar version that HF offers the same one that was available on INRIA? By that I mean, have you done any further filtering or removing of data inside it? Thanks a lot!

Hello @jchwenger, this is exactly the same (unshuffled) version that's available at Inria. Sadly no further filtering is provided, but after the latest OSCAR audit (https://arxiv.org/abs/2103.12028) we're already working on future versions of OSCAR that will be "filtered" and that will be available on the OSCAR website and hopefully here as well.

@pjox brilliant, in my case I was hoping it would be unfiltered, good news!

I don't know if tests pass, when I run them it tries to download the whole corpus which is around 3.5TB compressed and I don't have that kind of space. I'll really need some help with it 😅

Thanks!


'cp950' codec error from load_dataset('xtreme', 'tydiqa')

dataset bug

A bug in a dataset script provided in the library

This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
See issues #242 and #307

It should be in `xtreme.py:L755`:
```python
        if self.config.name == "tydiqa" or self.config.name.startswith("MLQA") or self.config.name == "SQuAD":
            with open(filepath) as f:
                data = json.load(f)
```

Could you try to add the encoding parameter:
```python
open(filepath, encoding='utf-8')
```

Hello @jerryIsHere :) Did it work ?
If so we may change the dataset script to force the utf-8 encoding

@lhoestq sorry for being that late, I found 4 copy of xtreme.py. I did the changes as what has been told to all of them.
The problem is not solved

Could you provide a better error message so that we can make sure it comes from the opening of the `tydiqa`'s json files ?

@lhoestq
The error message is same as before:
Exception has occurred: UnicodeDecodeError
'cp950' codec can't decode byte 0xe2 in position 111: illegal multibyte sequence
  File "D:\python\test\test.py", line 3, in <module>
    dataset = load_dataset('xtreme', 'tydiqa')

![image](https://user-images.githubusercontent.com/50871412/87748794-7c216880-c829-11ea-94f0-7caeacb4d865.png)

I said that I found 4 copy of xtreme.py and add the 「, encoding='utf-8'」 parameter to the open() function
these python script was found under this directory
C:\Users\USER\AppData\Local\Programs\Python\Python37\Lib\site-packages\nlp\datasets\xtreme

Hi there !
I encountered the same issue with the IMDB dataset on windows. It threw an error about charmap not being able to decode a symbol during the first time I tried to download it. I checked on a remote linux machine I have, and it can't be reproduced.
I added ```encoding='UTF-8'``` to both lines that have ```open``` in ```imdb.py```  (108 and 114) and it worked for me.
Thank you !

> Hi there !
> I encountered the same issue with the IMDB dataset on windows. It threw an error about charmap not being able to decode a symbol during the first time I tried to download it. I checked on a remote linux machine I have, and it can't be reproduced.
> I added `encoding='UTF-8'` to both lines that have `open` in `imdb.py` (108 and 114) and it worked for me.
> Thank you !

Hello !
Glad you managed to fix this issue on your side.
Do you mind opening a PR for IMDB ?

> This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
> Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
> See issues #242 and #307

Sorry for not responding for about a month.
I have just found that it is necessary to change / add the environment variable as what was told in #242.
Everything works after I add the new environment variable and restart my PC.

I think the encoding issue for windows isn't limited to the open() function call specific to few dataset, but actually in the entire library, depends on the machine / os you use.

Since #481 we shouldn't have other issues with encodings as they need to be set to "utf-8" be default.

Closing this one, but feel free to re-open if you gave other questions

![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly


Add emotion dataset

I've tried it and am getting the same error as you.

You could use the text files rather than the pickle:
```
https://www.dropbox.com/s/ikkqxfdbdec3fuj/test.txt
https://www.dropbox.com/s/1pzkadrvffbqw6o/train.txt
https://www.dropbox.com/s/2mzialpsgf9k5l3/val.txt
```

Then you would get all 3 splits rather than just the train split.

Thanks a lot @ghomasHudson - silly me for not spotting that!

I'll keep the PR open for now since I'm quite close to wrapping it up.

Hi @ghomasHudson your suggestion worked like a charm - the PR is now ready for review 😎

Hello, I probably have a silly question but the labels of the emotion dataset are in the form of numbers and not string, so I can not use the function classification_report because it mixes numbers and string (prediction). How can I access the label in the form of a string and not a number?
Thank you in advance.

Hi @juliette-sch! Yes, I believe that having the labels as integers is now the default for many classification datasets. You can access the string label via the `ClassLabel.int2str` function ([docs](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=int2str#datasets.ClassLabel.int2str)), so you could add a new column to the dataset as follows:

```python
from datasets import load_dataset

emotions = load_dataset("emotion")

def label_int2str(row):
    return {"label_name": emotions["train"].features["label"].int2str(row["label"])}

# adds a new column called `label_name`
emotions = emotions.map(label_int2str)
```

Hi, @lewtun
When I load "emotion"
```
from datasets import load_dataset

emotions = load_dataset("emotion")
```

There is an error:

```
ConnectionError: Couldn't reach https://www.dropbox.com/s/1pzkadrvffbqw6o/train.txt?dl=1 (SSLError(MaxRetryError("HTTPSConnectionPool(host='www.dropbox.com', port=443): Max retries exceeded with url: /s/1pzkadrvffbqw6o/train.txt?dl=1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)')))")))
```
Can you please tell me what is wrong?

Thanks a lot.
Dan

Hi ! I could't reproduce the error, can you try again ? You can also try updating the `datasets` library and see if it fixes the issue

Hi @lhoestq

It seems my company's internet blocked dropbox, I am sorry.
Thanks a lot.

Hello 🤗  team!

I am trying to add an emotion classification dataset ([link](https://github.com/dair-ai/emotion_dataset)) to `nlp` but I am a bit stuck about what I should do when the URL for the dataset is not a ZIP file, but just a pickled `pandas.DataFrame` (see [here](https://www.dropbox.com/s/607ptdakxuh5i4s/merged_training.pkl)).

With the current implementation, running

```bash
python nlp-cli test datasets/emotion --save_infos --all_configs
```

throws a `_pickle.UnpicklingError: invalid load key, '<'.` error (full stack trace below). The strange thing is that the path to the file does not carry the `.pkl` extension and instead appears to be some md5 hash (see the `FILE PATH` print statement in the stack trace).

Note: I have checked that the `merged_training.pkl` file is not corrupted when I download it with `wget`.

Any pointers on what I'm doing wrong would be greatly appreciated!

**Stack trace**

```
INFO:nlp.load:Checking datasets/emotion/emotion.py for additional imports.
INFO:filelock:Lock 140330435928512 acquired on datasets/emotion/emotion.py.lock
INFO:nlp.load:Found main folder for dataset datasets/emotion/emotion.py at /Users/lewtun/git/nlp/src/nlp/datasets/emotion
INFO:nlp.load:Creating specific version folder for dataset datasets/emotion/emotion.py at /Users/lewtun/git/nlp/src/nlp/datasets/emotion/59666994754d1b369228a749b695e377643d141fa98c6972be00407659788c7b
INFO:nlp.load:Copying script file from datasets/emotion/emotion.py to /Users/lewtun/git/nlp/src/nlp/datasets/emotion/59666994754d1b369228a749b695e377643d141fa98c6972be00407659788c7b/emotion.py
INFO:nlp.load:Couldn't find dataset infos file at datasets/emotion/dataset_infos.json
INFO:nlp.load:Creating metadata file for dataset datasets/emotion/emotion.py at /Users/lewtun/git/nlp/src/nlp/datasets/emotion/59666994754d1b369228a749b695e377643d141fa98c6972be00407659788c7b/emotion.json
INFO:filelock:Lock 140330435928512 released on datasets/emotion/emotion.py.lock
INFO:nlp.builder:Generating dataset emotion (/Users/lewtun/.cache/huggingface/datasets/emotion/emotion/1.0.0)
INFO:nlp.builder:Dataset not on Hf google storage. Downloading and preparing it from source
Downloading and preparing dataset emotion/emotion (download: Unknown size, generated: Unknown size, total: Unknown size) to /Users/lewtun/.cache/huggingface/datasets/emotion/emotion/1.0.0...
INFO:nlp.builder:Generating split train
0 examples [00:00, ? examples/s]FILE PATH /Users/lewtun/.cache/huggingface/datasets/3615dcb52b7ba052ef63e1571894c4b67e8e12a6ab1ef2f756ec3c380bf48490
Traceback (most recent call last):
  File "nlp-cli", line 37, in <module>
    service.run()
  File "/Users/lewtun/git/nlp/src/nlp/commands/test.py", line 83, in run
    builder.download_and_prepare(
  File "/Users/lewtun/git/nlp/src/nlp/builder.py", line 431, in download_and_prepare
    self._download_and_prepare(
  File "/Users/lewtun/git/nlp/src/nlp/builder.py", line 483, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/lewtun/git/nlp/src/nlp/builder.py", line 664, in _prepare_split
    for key, record in utils.tqdm(generator, unit=" examples", total=split_info.num_examples, leave=False):
  File "/Users/lewtun/miniconda3/envs/nlp/lib/python3.8/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/Users/lewtun/git/nlp/src/nlp/datasets/emotion/59666994754d1b369228a749b695e377643d141fa98c6972be00407659788c7b/emotion.py", line 87, in _generate_examples
    data = pickle.load(f)
_pickle.UnpicklingError: invalid load key, '<'.
```


Supporting documents in ELI5

Hi @saverymax ! For licensing reasons, the original team was unable to release pre-processed CommonCrawl documents. Instead, they provided a script to re-create them from a CommonCrawl dump, but it unfortunately requires access to a medium-large size cluster:
https://github.com/facebookresearch/ELI5#downloading-support-documents-from-the-commoncrawl

In order to make the task accessible to people who may not have access to this kind of infrastructure, we suggest to use Wikipedia as a knowledge source rather than the full CommonCrawl. The following blog post shows how you can create Wikipedia support documents and get a performance that is on par with a system that uses CommonCrawl pages.
https://yjernite.github.io/lfqa.html#task_description

Hope that helps, using ElasticSearch to index Wiki40b and create the documents should take about 4 hours. Let us know if you have any trouble with the blog post though!

Hi, thanks for the quick response. The blog post is quite an interesting working example, thanks for sharing it.
Two follow-up points/questions about my original question:

1. Yes, I read that the facebook team could not share the CommonCrawl b/c of licensing reasons. They state "No, we are not allowed to host processed Reddit or CommonCrawl data," which indicates they could also not share the Reddit data for licensing reasons. But it seems that HuggingFace is able to share the Reddit data, so why not a subset of CommonCrawl?

2. Thanks for the suggestion about ElasticSearch and Wiki40b. This is good to know about performance. I definitely could do the indexing and querying myself. What I like about the ELI5 dataset though, at least what is suggested by the paper, is that to create the dataset they had already selected the top 100 web sources and made a single support document from those. Though it doesn't appear to be too sophisticated an approach, having a single support document pre-computed (without having to run the facebook code or a replacement with another dataset) is super useful for my work, especially since I'm not working on developing the latest and greatest retrieval model. Of course, I don't expect HF NLP datasets to be perfectly tailored to my use-case. I know there is overhead to any project, I'm just illustrating a use-case of ELI5 which is not possible with the data provided as-is. If it's for licensing reasons, that is perfectly acceptable a reason, and I appreciate your response.

I was attempting to use the ELI5 dataset, when I realized that huggingface does not provide the supporting documents (the source documents from the common crawl). Without the supporting documents, this makes the dataset about as useful for my project as a block of cheese, or some other more apt metaphor.  According to facebook, the entire document collection is quite large. However, it would still be helpful to at least include a subset of the supporting documents i.e., having some data is better than having a block of cheese, in my case at least.

If you choose not to include them, it would be helpful to have documentation mentioning this specifically. It is especially confusing because the hf nlp ELI5 dataset has the key `'document'` but there are no documents to be found :(


Search qa

Could you rebase from master just to make sure we won't break anything for `fever` pls @mariamabarham ?

This PR adds the Search QA dataset used in **SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine**. The dataset has the following config name:

- raw_jeopardy: raw data

- train_test_val:  which is the splitted version

#336


Fix nested tensorflow format

In #339 and #337 we are thinking about adding a way to export datasets to tfrecords.

However I noticed that it was not possible to do `dset.set_format("tensorflow")` on datasets with nested features like `squad`. I fixed that using a nested map operations to convert features to `tf.ragged.constant`.

I also added tests on the `set_format` function.


Features should be updated when `map()` changes schema

`dataset.column_names` are being updated but `dataset.features` aren't indeed...

`dataset.map()` can change the schema and column names.

We should update the features in this case (with what is possible to infer).


add fever dataset

This PR add the FEVER dataset https://fever.ai/ used in with the paper: FEVER: a large-scale dataset for Fact Extraction and VERification (https://arxiv.org/pdf/1803.05355.pdf).
#336


Update cfq.py

Make the dataset name consistent with in the paper:  Compositional Freebase Question => Compositional Freebase Questions.


Add dataset.export() to TFRecords

Really cool @jarednielsen !
Do you think we can make it work with dataset with nested features like `squad` ?

I just did a PR to fix `.set_format` for datasets with nested features, but as soon as it's merged we could try to make the conversion work on a dataset like `squad`.

For datasets with nested features we have two aspects to take into account:
1) There can be nested dict of features. What is done in tensorflow_datasets to make things work is to flatten the dictionaries to end up with one single dictionary. A dict like `{"column1": {"subfeature": ...}}` is converted to `{"column1/subfeature":...}`
2) There can be ragged tensors, i.e. lists of objects with non-fixed shapes. For example in squad there are often multiple possible answers per question. What is done in tensorflow_datasets to make things work is to concatenate everything and add ragged attributes (cf serialization code [here](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/example_serializer.py))

Note that we have `flatten` method in `ArrowDataset`

I added support for nested dictionaries. A few more design decisions popped up:

_Should we serialize from NumPy arrays or from tf.Tensors?_
- The [tfds example serializer](url) works from NumPy arrays.
- Calling `dset.set_format("tensorflow")` makes `__getitem__` return a tf.Tensor. So serializing from NumPy arrays would mean calling `dset.export()` before setting the format, which is confusing.
- NumPy arrays can be serialized as their underlying datatype (int, float), while tf.Tensors must be converted to strings before serialization. This adds another step when serializing and deserializing, and removes the static-typing advantages of the TFRecord format.

I think we should export directly from the underlying NumPy arrays into TFRecords, rather than using an intermediate step of tf.Tensor.

_Should we serialize lists of dictionaries?_
- The test_format_nested() test creates a list of dictionaries: https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/tests/test_arrow_dataset.py#L278-L288
- This is difficult to serialize effectively, and I'm not aware of any dataset that has this format. SQuAD has a dictionary of lists, such as the `answers` key. Is this necessary?

Thanks @thomwolf, used dset.flatten() to simplify. That handles the case of nested dictionaries, and then lists can be read into a tf.io.RaggedFeature in the case of something like squad answers.

@jarednielsen I just checked and indeed we don't have lists of dicts, we can just focus on the squad format as a reference then :) I'll change the test to remove this format that's not supposed to happen

Actually I realised that `flatten` also handles nested things like pyarrow's list<struct> so it's fine :D
This is so cool !

Could you also add a test with a squad-like dataset ? As soon as we have that I think we'll be good to merge @jarednielsen  :)
Good job !

Great, done! I think this could be a great canonical way to generate a dataset.

I tried to match the format of Dataset.sort() and Dataset.shuffle() with the docstring. What difference are you referring to specifically?

Oh my bad they're fine actually (I was thinking of the backticks that we don't use in the docstrings of the transformers repo for argument names)

One final thing: now that we have a brand new documentation, could you just add `export` to the list of documented methods in [docs/source/package_reference/main_classes.rst](https://github.com/huggingface/nlp/blob/master/docs/source/package_reference/main_classes.rst) (so that it will appear in the docs [here](https://huggingface.co/nlp/package_reference/main_classes.html)) ?

Since #403 (it just got merged), we return python objects and not numpy arrays anymore (unless format="numpy" is specified).
Do you think it can break the export method ? Could you try to rebase from master to run the CI to make sure it's fine ?

Good catch. I fixed it up so it works with the new format. By the way, when dset.format == "numpy", it now returns single items (like `0`) as a 0-dimensional NumPy array. Not sure if that is desired.

I played a little bit with the code and it works quite well :)

I found two cases for which it doesn't work though:
- if the features dict depth is > 2 (ex: wikisql), because `flatten` only flattens the first level of nesting (it can be fixed by calling `flatten` several times in a row, see [here](https://issues.apache.org/jira/browse/ARROW-4090))
- Or if there are 2d features (ex: wikisql, `table.rows` is a sequence of sequences of strings), because tf.train.Features only support 1-d lists. That's why tensorflow-datasets flattens these 2-d features to 1-d and adds ragged features that are the shapes of the arrays, so that they can be reconstructed.

I think we can ignore the 2d stuff right now (some work is being done in #363 ), but I'd like to see the `flatten` issue fixed soon

That seems like a bug in `pyarrow`, or at least in `flatten()`. Looks like it should be a separate PR.

I made `.flatten` work on our side (it calls pyarrow's flatten several times until it's really flat).

The only datasets that won't work are those with lists of lists of features, which is a rare case. Hopefully we can make this work with the multi-dimensional arrays changes we're also doing.

I think we can merge now :) cc @thomwolf

Fixes https://github.com/huggingface/nlp/issues/337

Some design decisions:

- Simplified the function API to not handle sharding. It writes the entire dataset as a single TFRecord file. This simplifies the function logic and users can use other functions (`select`, `shard`, etc) to handle custom sharding or splitting.
- Use `from_generator()` instead of `from_tensor_slices()` to address the memory issues discussed in https://github.com/huggingface/nlp/issues/315 and https://github.com/huggingface/nlp/issues/193.
- Performs introspection using the values from `dataset.set_format()` to identify the TF datatypes. Currently it supports string, float, and int. If this should be extended for other datatypes, let me know.
- There are quite a few helper functions required within the `export()` method. If these are better placed in a utils file somewhere, let me know.

Also, I noticed that
```python
dataset = dataset.select(indices)
dataset.set_format("tensorflow")
# dataset._format_type is "tensorflow"
```
gives a different output than
```python
dataset.set_format("tensorflow")
dataset = dataset.select(indices)
# dataset._format_type is None
```
The latter loses the format of its parent dataset. Is there interest in making `set_format` a functional method that returns itself (can be chained), and that derived datasets maintain the format of their parent?


Run `make style`

These files get changed when I run `make style` on an unrelated PR. Upstreaming these changes so development on a different branch can be easier.


[Feature request] Export Arrow dataset to TFRecords

The TFRecord generation process is error-prone and requires complex separate Python scripts to download and preprocess the data. I propose to combine the user-friendly features of `nlp` with the speed and efficiency of TFRecords. Sample API:

```python
# use these existing methods
ds = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
ds = ds.map(lambda ex: tokenizer(ex))
ds.set_format("tensorflow", columns=["input_ids", "token_type_ids", "attention_mask"])
# then add this method
ds.export(folder="/my/tfrecords", prefix="myrecord", num_shards=8, format="tfrecord")
```
which would create files like so:
```bash
/my/tfrecords/myrecord_1.tfrecord
/my/tfrecords/myrecord_2.tfrecord
...
```

I would be happy to contribute this method. We could use a similar approach for PyTorch. Thoughts?


[Dataset requests] New datasets for Open Question Answering

help wanted

Extra attention is needed

dataset request

Requesting to add a new dataset

We are still a few datasets missing for Open-Question Answering which is currently a field in strong development.

Namely, it would be really nice to add:
- WebQuestions (Berant et al., 2013) [done]
- CuratedTrec (Baudis et al. 2015) [not open-source]
- MS-MARCO (NGuyen et al. 2016) [done]
- SearchQA (Dunn et al. 2017) [done]
- FEVER (Thorne et al. 2018) - [ done]

All these datasets are cited in http://arxiv.org/abs/2005.11401


BioMRC Dataset presented in BioNLP 2020 ACL Workshop

I fixed the issues that you pointed out, re-run all the test and pushed the fixed code :-)

```
=================================== FAILURES ===================================
___________________ AWSDatasetTest.test_load_dataset_pandas ____________________

self = <tests.test_dataset_common.AWSDatasetTest testMethod=test_load_dataset_pandas>
dataset_name = 'pandas'

    def test_load_dataset(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name)[:1]
>       self.dataset_tester.check_load_dataset(dataset_name, configs)

tests/test_dataset_common.py:231:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:125: in check_load_dataset
    dl_manager=mock_dl_manager, download_mode=GenerateMode.FORCE_REDOWNLOAD, ignore_verifications=True
../.local/lib/python3.6/site-packages/nlp/builder.py:432: in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
../.local/lib/python3.6/site-packages/nlp/builder.py:466: in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nlp.datasets.pandas.91271af5018cf7184c27d5cd64802a1b234b3cf0e37dbca0d60f03179b13e926.pandas.Pandas object at 0x7f3b84f655c0>
dl_manager = <nlp.utils.mock_download_manager.MockDownloadManager object at 0x7f3b84f3d320>

    def _split_generators(self, dl_manager):
        """ We handle string, list and dicts in datafiles
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"files": files})]
        splits = []
        for split_name in [nlp.Split.TRAIN, nlp.Split.VALIDATION, nlp.Split.TEST]:
>           if split_name in self.config.data_files:
E           TypeError: argument of type 'NoneType' is not iterable

../.local/lib/python3.6/site-packages/nlp/datasets/pandas/91271af5018cf7184c27d5cd64802a1b234b3cf0e37dbca0d60f03179b13e926/pandas.py:23: TypeError
------------------------------ Captured log call -------------------------------
INFO     filelock:filelock.py:274 Lock 139893169180856 acquired on /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py.lock
INFO     nlp.utils.file_utils:file_utils.py:386 https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py not found in cache or force_download set to True, downloading to /home/circleci/.cache/huggingface/datasets/tmpwmbk8e8d
INFO     nlp.utils.file_utils:file_utils.py:391 storing https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py in cache at /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py
INFO     nlp.utils.file_utils:file_utils.py:394 creating metadata file for /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py
INFO     filelock:filelock.py:318 Lock 139893169180856 released on /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py.lock
INFO     nlp.load:load.py:157 Checking /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py for additional imports.
INFO     filelock:filelock.py:274 Lock 139893610536912 acquired on /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py.lock
INFO     nlp.load:load.py:320 Found main folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/pandas
INFO     nlp.load:load.py:333 Found specific version folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/pandas/91271af5018cf7184c27d5cd64802a1b234b3cf0e37dbca0d60f03179b13e926
INFO     nlp.load:load.py:346 Found script file from https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py to /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/pandas/91271af5018cf7184c27d5cd64802a1b234b3cf0e37dbca0d60f03179b13e926/pandas.py
INFO     nlp.load:load.py:354 Couldn't find dataset infos file at https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/dataset_infos.json
INFO     nlp.load:load.py:371 Found metadata file for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/pandas/91271af5018cf7184c27d5cd64802a1b234b3cf0e37dbca0d60f03179b13e926/pandas.json
INFO     filelock:filelock.py:318 Lock 139893610536912 released on /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py.lock
INFO     filelock:filelock.py:274 Lock 139893610533608 acquired on /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py.lock
INFO     nlp.utils.file_utils:file_utils.py:386 https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py not found in cache or force_download set to True, downloading to /home/circleci/.cache/huggingface/datasets/tmp00hpyxrs
INFO     nlp.utils.file_utils:file_utils.py:391 storing https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py in cache at /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py
INFO     nlp.utils.file_utils:file_utils.py:394 creating metadata file for /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py
INFO     filelock:filelock.py:318 Lock 139893610533608 released on /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py.lock
INFO     nlp.load:load.py:157 Checking /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py for additional imports.
INFO     filelock:filelock.py:274 Lock 139893610371224 acquired on /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py.lock
INFO     nlp.load:load.py:320 Found main folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/pandas
INFO     nlp.load:load.py:333 Found specific version folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/pandas/91271af5018cf7184c27d5cd64802a1b234b3cf0e37dbca0d60f03179b13e926
INFO     nlp.load:load.py:346 Found script file from https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py to /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/pandas/91271af5018cf7184c27d5cd64802a1b234b3cf0e37dbca0d60f03179b13e926/pandas.py
INFO     nlp.load:load.py:354 Couldn't find dataset infos file at https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/dataset_infos.json
INFO     nlp.load:load.py:371 Found metadata file for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/pandas/pandas.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/pandas/91271af5018cf7184c27d5cd64802a1b234b3cf0e37dbca0d60f03179b13e926/pandas.json
INFO     filelock:filelock.py:318 Lock 139893610371224 released on /home/circleci/.cache/huggingface/datasets/e5827d40e7a41d66bc5a2eded8dbc90694265f47d9e7cb0273ff6ff11ba426d9.aa556094028e27447a02bb38655ff97b3f4e06db1ac04c1bcdcf5b283b0f75b6.py.lock
WARNING  nlp.builder:builder.py:215 Using custom data configuration default
INFO     nlp.builder:builder.py:349 Generating dataset pandas (/tmp/tmp296h8eeg/pandas/default/0.0.0)
INFO     nlp.builder:builder.py:397 Dataset not on Hf google storage. Downloading and preparing it from source
____________________ AWSDatasetTest.test_load_dataset_text _____________________

self = <tests.test_dataset_common.AWSDatasetTest testMethod=test_load_dataset_text>
dataset_name = 'text'

    def test_load_dataset(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name)[:1]
>       self.dataset_tester.check_load_dataset(dataset_name, configs)

tests/test_dataset_common.py:231:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:125: in check_load_dataset
    dl_manager=mock_dl_manager, download_mode=GenerateMode.FORCE_REDOWNLOAD, ignore_verifications=True
../.local/lib/python3.6/site-packages/nlp/builder.py:432: in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
../.local/lib/python3.6/site-packages/nlp/builder.py:466: in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nlp.datasets.text.bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b.text.Text object at 0x7f3b6a111550>
dl_manager = <nlp.utils.mock_download_manager.MockDownloadManager object at 0x7f3b85582908>

    def _split_generators(self, dl_manager):
        """ The `datafiles` kwarg in load_dataset() can be a str, List[str], Dict[str,str], or Dict[str,List[str]].

            If str or List[str], then the dataset returns only the 'train' split.
            If dict, then keys should be from the `nlp.Split` enum.
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            # Handle case with only one split
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"files": files})]
        else:
            # Handle case with several splits and a dict mapping
            splits = []
            for split_name in [nlp.Split.TRAIN, nlp.Split.VALIDATION, nlp.Split.TEST]:
>               if split_name in self.config.data_files:
E               TypeError: argument of type 'NoneType' is not iterable

../.local/lib/python3.6/site-packages/nlp/datasets/text/bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b/text.py:24: TypeError
------------------------------ Captured log call -------------------------------
INFO     filelock:filelock.py:274 Lock 139893159303656 acquired on /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py.lock
INFO     nlp.utils.file_utils:file_utils.py:386 https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py not found in cache or force_download set to True, downloading to /home/circleci/.cache/huggingface/datasets/tmpk63omy4v
INFO     nlp.utils.file_utils:file_utils.py:391 storing https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py in cache at /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py
INFO     nlp.utils.file_utils:file_utils.py:394 creating metadata file for /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py
INFO     filelock:filelock.py:318 Lock 139893159303656 released on /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py.lock
INFO     nlp.load:load.py:157 Checking /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py for additional imports.
INFO     filelock:filelock.py:274 Lock 139893159171352 acquired on /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py.lock
INFO     nlp.load:load.py:320 Found main folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/text
INFO     nlp.load:load.py:333 Found specific version folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/text/bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b
INFO     nlp.load:load.py:346 Found script file from https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py to /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/text/bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b/text.py
INFO     nlp.load:load.py:354 Couldn't find dataset infos file at https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/dataset_infos.json
INFO     nlp.load:load.py:371 Found metadata file for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/text/bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b/text.json
INFO     filelock:filelock.py:318 Lock 139893159171352 released on /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py.lock
INFO     filelock:filelock.py:274 Lock 139893618479176 acquired on /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py.lock
INFO     nlp.utils.file_utils:file_utils.py:386 https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py not found in cache or force_download set to True, downloading to /home/circleci/.cache/huggingface/datasets/tmpkeykru_f
INFO     nlp.utils.file_utils:file_utils.py:391 storing https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py in cache at /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py
INFO     nlp.utils.file_utils:file_utils.py:394 creating metadata file for /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py
INFO     filelock:filelock.py:318 Lock 139893618479176 released on /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py.lock
INFO     nlp.load:load.py:157 Checking /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py for additional imports.
INFO     filelock:filelock.py:274 Lock 139893618423848 acquired on /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py.lock
INFO     nlp.load:load.py:320 Found main folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/text
INFO     nlp.load:load.py:333 Found specific version folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/text/bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b
INFO     nlp.load:load.py:346 Found script file from https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py to /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/text/bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b/text.py
INFO     nlp.load:load.py:354 Couldn't find dataset infos file at https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/dataset_infos.json
INFO     nlp.load:load.py:371 Found metadata file for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/text/text.py at /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/text/bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b/text.json
INFO     filelock:filelock.py:318 Lock 139893618423848 released on /home/circleci/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.88f858fae8ed77fdff99fe23b726fce01f73388251e0a09a226e6f82cd4ffe6c.py.lock
WARNING  nlp.builder:builder.py:215 Using custom data configuration default
INFO     nlp.builder:builder.py:349 Generating dataset text (/tmp/tmpbu67mvue/text/default/0.0.0)
INFO     nlp.builder:builder.py:397 Dataset not on Hf google storage. Downloading and preparing it from source
=============================== warnings summary ===============================
/home/circleci/.local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py:15
  /home/circleci/.local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

tests/test_dataset_common.py::LocalDatasetTest::test_builder_class_tydiqa
  /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/tydiqa/42d88245bde7c0db6c0d48c822dcaa26c7299e0b40cace7e8d6a9e3628135125/tydiqa.py:85: DeprecationWarning: invalid escape sequence \G
    """

tests/test_dataset_common.py::AWSDatasetTest::test_builder_class_mwsc
  /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/mwsc/53c0daac11b6794ff62b52a3a46c4f9da1bef68fd664a2f97b8918917aead715/mwsc.py:70: DeprecationWarning: invalid escape sequence \[
    pattern = "\[.*\]"

tests/test_dataset_common.py::AWSDatasetTest::test_builder_class_squadshifts
  /home/circleci/.local/lib/python3.6/site-packages/nlp/datasets/squadshifts/15536d7296a785325b99f6d84dfdceafa427419dd6caad110eabb5e5b4156cc2/squadshifts.py:47: DeprecationWarning: invalid escape sequence \
    """

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::AWSDatasetTest::test_load_dataset_pandas
FAILED tests/test_dataset_common.py::AWSDatasetTest::test_load_dataset_text
===== 2 failed, 934 passed, 516 skipped, 4 warnings in 1562.46s (0:26:02) ======

Exited with code exit status 1
CircleCI received exit code 1
```
I get this failed test on CircleCI , but all the tests that I run locally where successful. The error also seems not to have any, obvious at least, connection with my code.

Any suggestions? Thanks! :-)


Add dataset.shard() method

Fixes https://github.com/huggingface/nlp/issues/312


fix variable name typo

Good catch :)
I think there is another occurence that needs to be fixed in the second gist (line 4924 of the notebook file):
```python
bleu = nlp.load_metric(...)
```

Was fixed in e16f79b5f7fc12a6a30c777722be46897a272e6f
Closing it.


Add wiki_dpr

The two configurations don't have the same sizes, I may change that so that they both have 21015300 examples for convenience, even though it's supposed to have 21015324 examples in total.

One configuration only has 21015300 examples because it seems that the embeddings of the last 24 examples are missing.

It's ok to merge now imo. I'll make another PR if we find a way to have the missing embeddings

Presented in the [Dense Passage Retrieval paper](https://arxiv.org/pdf/2004.04906.pdf), this dataset consists in 21M passages from the english wikipedia along with their 768-dim embeddings computed using DPR's context encoder.

Note on the implementation:
- There are two configs: with and without the embeddings (73GB vs 14GB)
- I used a non-fixed-size sequence of floats to describe the feature format of the embeddings. I wanted to use fixed-size sequences but I had issues with reading the arrow file afterwards (for example `dataset[0]` was crashing)
- I added the case for lists of urls as input of the download_manager


Loading CNN/Daily Mail dataset produces `nlp.utils.info_utils.NonMatchingSplitsSizesError`

I couldn't reproduce on my side.
It looks like you were not able to generate all the examples, and you have the problem for each split train-test-validation.
Could you try to enable logging, try again and send the logs ?
```python
import logging
logging.basicConfig(level=logging.INFO)
```

here's the log
```
>>> import nlp
import logging
logging.basicConfig(level=logging.INFO)
nlp.load_dataset('cnn_dailymail', '3.0.0')
>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> nlp.load_dataset('cnn_dailymail', '3.0.0')
INFO:nlp.load:Checking /u/jm8wx/.cache/huggingface/datasets/720d2e20d8dc6d98f21195a39cc934bb41dd0a40b57ea3d323661a7c5d70522c.d44c2417f4e0fe938ede0a684dcbb1fa9b4789de22e8a99c43103d4b4c374b3b.py for additional imports.
INFO:filelock:Lock 140443095301136 acquired on /u/jm8wx/.cache/huggingface/datasets/720d2e20d8dc6d98f21195a39cc934bb41dd0a40b57ea3d323661a7c5d70522c.d44c2417f4e0fe938ede0a684dcbb1fa9b4789de22e8a99c43103d4b4c374b3b.py.lock
INFO:nlp.load:Found main folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py at /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail
INFO:nlp.load:Found specific version folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py at /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail/9645e0bc96f647decf46541f6f4bef6936ee82ace653ac362bab03309a46d4ad
INFO:nlp.load:Found script file from https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py to /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail/9645e0bc96f647decf46541f6f4bef6936ee82ace653ac362bab03309a46d4ad/cnn_dailymail.py
INFO:nlp.load:Updating dataset infos file from https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/dataset_infos.json to /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail/9645e0bc96f647decf46541f6f4bef6936ee82ace653ac362bab03309a46d4ad/dataset_infos.json
INFO:nlp.load:Found metadata file for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py at /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail/9645e0bc96f647decf46541f6f4bef6936ee82ace653ac362bab03309a46d4ad/cnn_dailymail.json
INFO:filelock:Lock 140443095301136 released on /u/jm8wx/.cache/huggingface/datasets/720d2e20d8dc6d98f21195a39cc934bb41dd0a40b57ea3d323661a7c5d70522c.d44c2417f4e0fe938ede0a684dcbb1fa9b4789de22e8a99c43103d4b4c374b3b.py.lock
INFO:nlp.info:Loading Dataset Infos from /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail/9645e0bc96f647decf46541f6f4bef6936ee82ace653ac362bab03309a46d4ad
INFO:nlp.builder:Generating dataset cnn_dailymail (/u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0)
INFO:nlp.builder:Dataset not on Hf google storage. Downloading and preparing it from source
Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.26 GiB, total: 1.81 GiB) to /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0...
INFO:nlp.utils.info_utils:All the checksums matched successfully.
INFO:nlp.builder:Generating split train
INFO:nlp.arrow_writer:Done writing 285161 examples in 1240618482 bytes /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0.incomplete/cnn_dailymail-train.arrow.
INFO:nlp.builder:Generating split validation
INFO:nlp.arrow_writer:Done writing 13255 examples in 56637485 bytes /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0.incomplete/cnn_dailymail-validation.arrow.
INFO:nlp.builder:Generating split test
INFO:nlp.arrow_writer:Done writing 11379 examples in 48931393 bytes /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0.incomplete/cnn_dailymail-test.arrow.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/load.py", line 520, in load_dataset
    builder_instance.download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 431, in download_and_prepare
    self._download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=49424491, num_examples=11490, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='test', num_bytes=48931393, num_examples=11379, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='train', num_bytes=1249178681, num_examples=287113, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='train', num_bytes=1240618482, num_examples=285161, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='validation', num_bytes=57149241, num_examples=13368, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='validation', num_bytes=56637485, num_examples=13255, dataset_name='cnn_dailymail')}]
```

> here's the log
>
> ```
> >>> import nlp
> import logging
> logging.basicConfig(level=logging.INFO)
> nlp.load_dataset('cnn_dailymail', '3.0.0')
> >>> import logging
> >>> logging.basicConfig(level=logging.INFO)
> >>> nlp.load_dataset('cnn_dailymail', '3.0.0')
> INFO:nlp.load:Checking /u/jm8wx/.cache/huggingface/datasets/720d2e20d8dc6d98f21195a39cc934bb41dd0a40b57ea3d323661a7c5d70522c.d44c2417f4e0fe938ede0a684dcbb1fa9b4789de22e8a99c43103d4b4c374b3b.py for additional imports.
> INFO:filelock:Lock 140443095301136 acquired on /u/jm8wx/.cache/huggingface/datasets/720d2e20d8dc6d98f21195a39cc934bb41dd0a40b57ea3d323661a7c5d70522c.d44c2417f4e0fe938ede0a684dcbb1fa9b4789de22e8a99c43103d4b4c374b3b.py.lock
> INFO:nlp.load:Found main folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py at /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail
> INFO:nlp.load:Found specific version folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py at /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail/9645e0bc96f647decf46541f6f4bef6936ee82ace653ac362bab03309a46d4ad
> INFO:nlp.load:Found script file from https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py to /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail/9645e0bc96f647decf46541f6f4bef6936ee82ace653ac362bab03309a46d4ad/cnn_dailymail.py
> INFO:nlp.load:Updating dataset infos file from https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/dataset_infos.json to /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail/9645e0bc96f647decf46541f6f4bef6936ee82ace653ac362bab03309a46d4ad/dataset_infos.json
> INFO:nlp.load:Found metadata file for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py at /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail/9645e0bc96f647decf46541f6f4bef6936ee82ace653ac362bab03309a46d4ad/cnn_dailymail.json
> INFO:filelock:Lock 140443095301136 released on /u/jm8wx/.cache/huggingface/datasets/720d2e20d8dc6d98f21195a39cc934bb41dd0a40b57ea3d323661a7c5d70522c.d44c2417f4e0fe938ede0a684dcbb1fa9b4789de22e8a99c43103d4b4c374b3b.py.lock
> INFO:nlp.info:Loading Dataset Infos from /p/qdata/jm8wx/datasets/nlp/src/nlp/datasets/cnn_dailymail/9645e0bc96f647decf46541f6f4bef6936ee82ace653ac362bab03309a46d4ad
> INFO:nlp.builder:Generating dataset cnn_dailymail (/u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0)
> INFO:nlp.builder:Dataset not on Hf google storage. Downloading and preparing it from source
> Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.26 GiB, total: 1.81 GiB) to /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0...
> INFO:nlp.utils.info_utils:All the checksums matched successfully.
> INFO:nlp.builder:Generating split train
> INFO:nlp.arrow_writer:Done writing 285161 examples in 1240618482 bytes /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0.incomplete/cnn_dailymail-train.arrow.
> INFO:nlp.builder:Generating split validation
> INFO:nlp.arrow_writer:Done writing 13255 examples in 56637485 bytes /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0.incomplete/cnn_dailymail-validation.arrow.
> INFO:nlp.builder:Generating split test
> INFO:nlp.arrow_writer:Done writing 11379 examples in 48931393 bytes /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0.incomplete/cnn_dailymail-test.arrow.
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/p/qdata/jm8wx/datasets/nlp/src/nlp/load.py", line 520, in load_dataset
>     builder_instance.download_and_prepare(
>   File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 431, in download_and_prepare
>     self._download_and_prepare(
>   File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 488, in _download_and_prepare
>     verify_splits(self.info.splits, split_dict)
>   File "/p/qdata/jm8wx/datasets/nlp/src/nlp/utils/info_utils.py", line 70, in verify_splits
>     raise NonMatchingSplitsSizesError(str(bad_splits))
> nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=49424491, num_examples=11490, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='test', num_bytes=48931393, num_examples=11379, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='train', num_bytes=1249178681, num_examples=287113, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='train', num_bytes=1240618482, num_examples=285161, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='validation', num_bytes=57149241, num_examples=13368, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='validation', num_bytes=56637485, num_examples=13255, dataset_name='cnn_dailymail')}]
> ```

With `nlp == 0.3.0` version, I'm not able to reproduce this error on my side.
Which version are you using for reproducing your bug?

```
>> nlp.load_dataset('cnn_dailymail', '3.0.0')

8.90k/8.90k [00:18<00:00, 486B/s]

Downloading: 100%
9.37k/9.37k [00:00<00:00, 234kB/s]

Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.26 GiB, total: 1.81 GiB) to /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0...
Downloading:
159M/? [00:09<00:00, 16.7MB/s]

Downloading:
376M/? [00:06<00:00, 62.6MB/s]

Downloading:
2.11M/? [00:06<00:00, 333kB/s]

Downloading:
46.4M/? [00:02<00:00, 18.4MB/s]

Downloading:
2.43M/? [00:00<00:00, 2.62MB/s]

Dataset cnn_dailymail downloaded and prepared to /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0. Subsequent calls will reuse this data.
{'test': Dataset(schema: {'article': 'string', 'highlights': 'string'}, num_rows: 11490),
 'train': Dataset(schema: {'article': 'string', 'highlights': 'string'}, num_rows: 287113),
 'validation': Dataset(schema: {'article': 'string', 'highlights': 'string'}, num_rows: 13368)}

>> ...

```

In general if some examples are missing after processing (hence causing the `NonMatchingSplitsSizesError `), it is often due to either
1) corrupted cached files
2) decoding errors

I just checked the dataset script for code that could lead to decoding errors but I couldn't find any. Before we try to dive more into the processing of the dataset, could you try to clear your cache ? Just to make sure that it isn't 1)

Yes thanks for the support! I cleared out my cache folder and everything works fine now

```
>>> import nlp
>>> nlp.load_dataset('cnn_dailymail', '3.0.0')
Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.26 GiB, total: 1.81 GiB) to /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0...

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/load.py", line 520, in load_dataset
    builder_instance.download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 431, in download_and_prepare
    self._download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=49424491, num_examples=11490, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='test', num_bytes=48931393, num_examples=11379, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='train', num_bytes=1249178681, num_examples=287113, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='train', num_bytes=1240618482, num_examples=285161, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='validation', num_bytes=57149241, num_examples=13368, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='validation', num_bytes=56637485, num_examples=13255, dataset_name='cnn_dailymail')}]
```


Doc red

Adding [DocRED](https://github.com/thunlp/DocRED) - a relation extraction dataset which tests document-level RE. A few implementation notes:

- There are 2 separate versions of the training set - *annotated* and *distant*. Instead of `nlp.Split.Train` I've used the splits `"train_annotated"` and `"train_distant"` to reflect this.
- As well as the relation id, the full relation name is mapped from `rel_info.json`
- I renamed the 'h', 'r', 't' keys to 'head', 'relation' and 'tail' to make them more readable.
- Used the fix from #319 to allow nested sequences of dicts.


[Bug] FileLock dependency incompatible with filesystem

Hi, can you give details on your environment/os/packages versions/etc?

Environment is Ubuntu 18.04, Python 3.7.5, nlp==0.3.0, filelock=3.0.12.

The external volume is Amazon FSx for Lustre, and it by default creates files with limited permissions. My working theory is that FileLock creates a lockfile that isn't writable, and thus there's no way to acquire it by removing the .lock file. But Python is able to create new files and write to them outside of the FileLock package.

When I attempt to use FileLock within a Docker container by writing to `/root/.cache/hello.txt`, it succeeds. So there's some permissions issue. But it's not a Docker configuration issue; I've replicated it without Docker.
```bash
echo "hello world" >> hello.txt
ls -l

-rw-rw-r-- 1 ubuntu ubuntu 10 Jun 30 19:52 hello.txt
```

Looks like the `flock` syscall does not work on Lustre filesystems by default: https://github.com/benediktschmitt/py-filelock/issues/67.

I added the `-o flock` option when mounting the filesystem, as [described here](https://docs.aws.amazon.com/fsx/latest/LustreGuide/getting-started-step2.html), which fixed the issue.

Awesome, thanks a lot for sharing your fix!

I'm downloading a dataset successfully with
`load_dataset("wikitext", "wikitext-2-raw-v1")`

But when I attempt to cache it on an external volume, it hangs indefinitely:
`load_dataset("wikitext", "wikitext-2-raw-v1", cache_dir="/fsx") # /fsx is an external volume mount`

The filesystem when hanging looks like this:
```bash
/fsx
----downloads
       ----94be...73.lock
----wikitext
       ----wikitext-2-raw
             ----wikitext-2-raw-1.0.0.incomplete
```

It appears that on this filesystem, the FileLock object is forever stuck in its "acquire" stage. I have verified that the issue lies specifically with the `filelock` dependency:
```python
open("/fsx/hello.txt").write("hello") # succeeds

from filelock import FileLock
with FileLock("/fsx/hello.lock"):
    open("/fsx/hello.txt").write("hello") # hangs indefinitely
```

Has anyone else run into this issue? I'd raise it directly on the FileLock repo, but that project appears abandoned with the last update over a year ago. Or if there's a solution that would remove the FileLock dependency from the project, I would appreciate that.


Fork dataset

To be able to generate the Arrow dataset you need to either use our csv or json utilities `load_dataset("json", data_files=my_json_files)` OR write your own custom dataset script (you can find some inspiration from the [squad](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py) script for example). Custom dataset scripts can be called locally with `nlp.load_dataset(path_to_my_script_directory)`.

This should help you get what you call "Dataset1".

Then using some dataset transforms like `.map` for example you can get to "DatasetNER" and "DatasetREL".

Thanks for the helpful advice, @lhoestq  -- I wasn't quite able to get the json recipe working -

```
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/ipc.py in __init__(self, source)
     60
     61     def __init__(self, source):
---> 62         self._open(source)
     63
     64
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/ipc.pxi in pyarrow.lib._RecordBatchStreamReader._open()
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowInvalid: Tried reading schema message, was null or length 0
```

But I'm going to give the generator_dataset_builder a try.

1 more quick question -- can .map be used to output different length mappings -- could I skip one, or yield 2, can you map_batch

You can use `.map(my_func, batched=True)` and return less examples, or more examples if you want

Thanks this answers my question. I think the issue I was having using the json loader were due to using gzipped jsonl files.

The error I get now is :

```

Using custom data configuration test
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-38-29082a31e5b2> in <module>
      5 print(ner_datafiles)
      6
----> 7 ds = nlp.load_dataset("json", "test",  data_files=ner_datafiles[0])
      8

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    430                 verify_infos = not save_infos and not ignore_verifications
    431                 self._download_and_prepare(
--> 432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    433                 )
    434                 # Sync info

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    481             try:
    482                 # Prepare split will record examples associated to the split
--> 483                 self._prepare_split(split_generator, **prepare_split_kwargs)
    484             except OSError:
    485                 raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _prepare_split(self, split_generator)
    736                     schema_dict[field.name] = Value(str(field.type))
    737
--> 738         parse_schema(writer.schema, features)
    739         self.info.features = Features(features)
    740

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in parse_schema(schema, schema_dict)
    734                     parse_schema(field.type.value_type, schema_dict[field.name])
    735                 else:
--> 736                     schema_dict[field.name] = Value(str(field.type))
    737
    738         parse_schema(writer.schema, features)

<string> in __init__(self, dtype, id, _type)

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in __post_init__(self)
     55
     56     def __post_init__(self):
---> 57         self.pa_type = string_to_arrow(self.dtype)
     58
     59     def __call__(self):

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in string_to_arrow(type_str)
     32         if str(type_str + "_") not in pa.__dict__:
     33             raise ValueError(
---> 34                 f"Neither {type_str} nor {type_str + '_'} seems to be a pyarrow data type. "
     35                 f"Please make sure to use a correct data type, see: "
     36                 f"https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions"

ValueError: Neither list<item: int64> nor list<item: int64>_ seems to be a pyarrow data type. Please make sure to use a correct data type, see: https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions.
```

If I just create a pa- table manually like is done in the jsonloader -- it seems to work fine. Ths JSON I'm trying to load isn't overly complex -  1 integer field, the rest text fields with a nested list of objects with text fields .

I'll close this -- It's still unclear how to go about troubleshooting the json example as I mentioned above. If I decide it's worth the trouble, I'll create another issue, or wait for a better support for using nlp for making custom data-loaders.

We have a multi-task learning model training I'm trying to convert to using the Arrow-based nlp dataset.

We're currently training a custom TensorFlow model but the nlp paradigm should be a bridge for us to be able to use the wealth of pre-trained models in Transformers.

Our preprocessing flow parses raw text and json with Entity and Relations annotations and creates 2 datasets for training a NER and Relations prediction heads.

Is there some good way to "fork" dataset-

EG

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 -> DatasetREL

or

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 + DatasetNER -> DatasetREL


set seed for suffling tests

Some tests were randomly failing because of a missing seed in a test for `train_test_split(shuffle=True)`


Large dataset in Squad2-format

I'm pretty sure you can get some inspiration from the squad_v2 script. It looks like the dataset is quite big so it will take some time for the users to generate it, but it should be reasonable.

Also you are saying that you are still making the dataset grow in size right ?
It's probably good practice to let the users do their training/evaluations with the exact same version of the dataset.
We allow for each dataset to specify a version (ex: 1.0.0) and increment this number every time there are new samples in the dataset for example. Does it look like a good solution for you ? Or would you rather have one final version with the full dataset ?

It would also be good if there is any possibility for versioning, I think this way is much better than the dynamic way.
If you mean that part to put the tiles into one is the generation it would take up to 15-20 minutes on home computer hardware.
Are there any compression or optimization algorithms while generating the dataset ?
Otherwise the hardware limit is around 32 GB ram at the moment.
If everything works well we will add some more gigabytes of data in future what would make it pretty memory costly.

15-20 minutes is fine !
Also there's no RAM limitations as we save to disk every 1000 elements while generating the dataset by default.
After generation, the dataset is ready to use with (again) no RAM limitations as we do memory-mapping.

Wow, that sounds pretty cool.
Actually I have the problem of running out of memory while tokenization on our local machine.
That wouldn't happen again, would it ?

You can do the tokenization step using `my_tokenized_dataset = my_dataset.map(my_tokenize_function)` that writes the tokenized texts on disk as well. And then `my_tokenized_dataset` will be a memory-mapped dataset too, so you should be fine :)

Does it have an affect to the trainings speed ?

In your training loop, loading the tokenized texts is going to be fast and pretty much negligible compared to a forward pass. You shouldn't expect any slow down.

Closing this one. Feel free to re-open if you have other questions

At the moment we are building an large question answering dataset and think about sharing it with the huggingface community.
Caused the computing power we splitted it into multiple tiles, but they are all in the same format.
Right now the most important facts about are this:
- Contexts: 1.047.671
- questions: 1.677.732
- Answers: 6.742.406
- unanswerable: 377.398

It is already cleaned

train_data = [
    {
        'context': "this is the context",
        'qas': [
            {
                'id': "00002",
                'is_impossible': False,
                'question': "whats is this",
                'answers': [
                    {
                        'text': "answer",
                        'answer_start': 0
                    }
                ]
            },
            {
                'id': "00003",
                'is_impossible': False,
                'question': "question2",
                'answers': [
                    {
                        'text': "answer2",
                        'answer_start': 1
                    }
                ]
            }
        ]
    }
]

Cause it is growing every day we are thinking about an structure like this:
We host an Json file, containing all the download links and the script can load it dynamically.
At the moment it is around ~20GB

Any advice how to handle this, or an ready to use template ?


Add SQuADShifts dataset

Very cool to have this dataset, thank you for adding it :)

This PR adds the four new variants of the SQuAD dataset used in [The Effect of Natural Distribution Shift on Question Answering Models](https://arxiv.org/abs/2004.14444) to facilitate evaluating model robustness to distribution shift.


Error when calculating glue score

The glue metric for cola is a metric for classification. It expects label ids as integers as inputs.

I want to evaluate a sentence pair whether they are semantically equivalent, so I used MRPC and it gives the same error, does that mean we have to encode the sentences and parse as input?

using BertTokenizer;
```
encoded_reference=tokenizer.encode(reference, add_special_tokens=False)
encoded_prediction=tokenizer.encode(prediction, add_special_tokens=False)
```

`glue_score = glue_metric.compute(encoded_prediction, encoded_reference)`
```

ValueError                                Traceback (most recent call last)
<ipython-input-9-4c3a3ce7b583> in <module>()
----> 1 glue_score = glue_metric.compute(encoded_prediction, encoded_reference)

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/metric.py in compute(self, predictions, references, timeout, **metrics_kwargs)
    198         predictions = self.data["predictions"]
    199         references = self.data["references"]
--> 200         output = self._compute(predictions=predictions, references=references, **metrics_kwargs)
    201         return output
    202

/usr/local/lib/python3.6/dist-packages/nlp/metrics/glue/27b1bc63e520833054bd0d7a8d0bc7f6aab84cc9eed1b576e98c806f9466d302/glue.py in _compute(self, predictions, references)
    101             return pearson_and_spearman(predictions, references)
    102         elif self.config_name in ["mrpc", "qqp"]:
--> 103             return acc_and_f1(predictions, references)
    104         elif self.config_name in ["sst2", "mnli", "mnli_mismatched", "mnli_matched", "qnli", "rte", "wnli", "hans"]:
    105             return {"accuracy": simple_accuracy(predictions, references)}

/usr/local/lib/python3.6/dist-packages/nlp/metrics/glue/27b1bc63e520833054bd0d7a8d0bc7f6aab84cc9eed1b576e98c806f9466d302/glue.py in acc_and_f1(preds, labels)
     60 def acc_and_f1(preds, labels):
     61     acc = simple_accuracy(preds, labels)
---> 62     f1 = f1_score(y_true=labels, y_pred=preds)
     63     return {
     64         "accuracy": acc,

/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in f1_score(y_true, y_pred, labels, pos_label, average, sample_weight, zero_division)
   1097                        pos_label=pos_label, average=average,
   1098                        sample_weight=sample_weight,
-> 1099                        zero_division=zero_division)
   1100
   1101

/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in fbeta_score(y_true, y_pred, beta, labels, pos_label, average, sample_weight, zero_division)
   1224                                                  warn_for=('f-score',),
   1225                                                  sample_weight=sample_weight,
-> 1226                                                  zero_division=zero_division)
   1227     return f
   1228

/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in precision_recall_fscore_support(y_true, y_pred, beta, labels, pos_label, average, warn_for, sample_weight, zero_division)
   1482         raise ValueError("beta should be >=0 in the F-beta score")
   1483     labels = _check_set_wise_labels(y_true, y_pred, average, labels,
-> 1484                                     pos_label)
   1485
   1486     # Calculate tp_sum, pred_sum, true_sum ###

/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in _check_set_wise_labels(y_true, y_pred, average, labels, pos_label)
   1314             raise ValueError("Target is %s but average='binary'. Please "
   1315                              "choose another average setting, one of %r."
-> 1316                              % (y_type, average_options))
   1317     elif pos_label not in (None, 1):
   1318         warnings.warn("Note that pos_label (set to %r) is ignored when "

ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

```

MRPC is also a binary classification task, so its metric is a binary classification metric.

To evaluate if pairs of sentences are semantically equivalent, maybe you could take a look at models that compute if one sentence entails the other or not (typically the kinds of model that could work well on the MRPC task).

Closing this one. Feel free to re-open if you have other questions :)

I was trying glue score along with other metrics here. But glue gives me this error;

```
import nlp
glue_metric = nlp.load_metric('glue',name="cola")

glue_score = glue_metric.compute(predictions, references)
```

```
---------------------------------------------------------------------------
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-b9210a524504> in <module>()
----> 1 glue_score = glue_metric.compute(predictions, references)

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/metric.py in compute(self, predictions, references, timeout, **metrics_kwargs)
    191         """
    192         if predictions is not None:
--> 193             self.add_batch(predictions=predictions, references=references)
    194         self.finalize(timeout=timeout)
    195

/usr/local/lib/python3.6/dist-packages/nlp/metric.py in add_batch(self, predictions, references, **kwargs)
    207         if self.writer is None:
    208             self._init_writer()
--> 209         self.writer.write_batch(batch)
    210
    211     def add(self, prediction=None, reference=None, **kwargs):

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    155         if self.pa_writer is None:
    156             self._build_writer(pa_table=pa.Table.from_pydict(batch_examples))
--> 157         pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
    158         if writer_batch_size is None:
    159             writer_batch_size = self.writer_batch_size

/usr/local/lib/python3.6/dist-packages/pyarrow/types.pxi in __iter__()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib.asarray()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

TypeError: an integer is required (got type str)
```
I'm not sure whether I'm doing this wrong or whether it's an issue. I would like to know a workaround. Thank you.


Add package path to sys when downloading package as github archive

Sorry for the long diff, everything after the imports comes from `black` for code quality :/

 I think it's fine and I can't think of another way to make the import work anyways.

Maybe we can have the `sys.path` behavior inside `prepare_module` instead ? Currently it seems to come out of nowhere in the code ^^'
We could check if external imports have a `__init__.py` and if it is the case then we can add to directory to the `PYTHONPATH`

This fixes the `coval.py` metric so that imports within the downloaded module work correctly. We can use a similar trick to add the BLEURT metric (@ankparikh)

@thomwolf not sure how you feel about adding to the `PYTHONPATH` from the script. This is the only way I could make it work with my understanding of `importlib` but there might be a more elegant method.

This PR fixes https://github.com/huggingface/nlp/issues/305


output nested dict in get_nearest_examples

As we are using a columnar format like arrow as the backend for datasets, we expect to have a dictionary of columns when we slice a dataset like in this example:
```python
my_examples = dataset[0:10]
print(type(my_examples))
# >>> dict
print(my_examples["my_column"][0]
# >>> this is the first element of the column 'my_column'
```

Therefore I wanted to keep this logic when calling `get_nearest_examples` that returns the top 10 nearest examples:
```python
dataset.add_faiss_index(column="embeddings")
scores, examples = dataset.get_nearest_examples("embeddings", query=my_numpy_embedding)
print(type(examples))
# >>> dict
```

Previously it was returning a list[dict]. It was the only place that was using this output format.

To make it work I had to implement `__getitem__(key)` where `key` is a list.
This is different from `.select` because `.select` is a dataset transform (it returns a new dataset object) while `__getitem__` is an extraction method (it returns python dictionaries).