Add WIT Dataset

Hi! Coming in late with some context.

There are two versions of the WIT dataset:
1. The original source dataset managed by Wikimedia. It has more information, raw image representations, and each row corresponds to an image linked to all of its captions wherever it happens in Wikipedia (in multiple languages)
2. The Google version, corresponding to the data script in this PR, which duplicates image instances and requires the user to download the images themselves from the provided URL (note that a basic implementation will have them download the same picture several time. @thomasw21 using our download manager instead of `urllib` could help with that, but it wouldn't be required if people had access to the first version)

The Wikimedia folks were really interested in us hosting a ready-to-go streaming version of this dataset where users don't have to download the version themselves, which is why we have the pre-processed versions on an HF bucket, with the raw images and a pre-computed embedding (don't remember the model, we can keep it ). That's the data script currently in https://github.com/huggingface/datasets/pull/2981 . It's nearly ready to go, the one thing we should do is move the raw data from our HF google Cloud bucket to the Hub.

How do you want to move forward? IMO the best way would be to have a WIT dataset under the Wikimedia org with both configurations, but it depends on everyone's timelines

Okay after offline discussion. We'll improve this versions and push it to the hub under `google` namespace.

> which duplicates image instances and requires the user to download the images themselves from the provided URL (note that a basic implementation will have them download the same picture several time. @thomasw21 using our download manager instead of urllib could help with that, but it wouldn't be required if people had access to the first version)

Ah interesting wasn't aware of this duplication issue, concretely it'll just mean that our dataset in bigger than expected ... I think this should be handled after this loading script (though I have to figure our how to spawn a dl_manager).

> The Wikimedia folks were really interested in us hosting a ready-to-go streaming version of this dataset where users don't have to download the version themselves, which is why we have the pre-processed versions on an HF bucket, with the raw images and a pre-computed embedding (don't remember the model, we can keep it ). That's the data script currently in https://github.com/huggingface/datasets/pull/2981 . It's nearly ready to go, the one thing we should do is move the raw data from our HF google Cloud bucket to the Hub.

Similarly a script will be written and pushed to `wikimedia` organisation.

@mariosasko can you make one last review concerning the text description changes? Then I'll handle putting it under `google` namespace and close this PR.

Closing as this has been migrated to the hub under `google` namespace: https://huggingface.co/datasets/google/wit

closes #2981 #2810

@nateraw @hassiahk I've listed you guys as co-author as you've contributed previously to this dataset


Fix when map function modifies input in-place

_The documentation is not available anymore as the PR was closed or merged._

When `function` modifies input in-place, the guarantee that columns in `remove_columns` are contained in `input` doesn't hold true anymore. Therefore we need to relax way we pop elements by checking if that column exists.


Stream private zipped images

oops looks like some tests are failing sorry, will fix them tomorrow

EDIT: not today but asap hopefully

cc @mariosasko this is ready for review, let me know what you think !

As mentioned in https://github.com/huggingface/datasets/issues/4139 it's currently not possible to stream private/gated zipped images from the Hub.

This is because `Image.decode_example` does not handle authentication. Indeed decoding requires to access and download the file from the private repository.

In this PR I added authentication to `Image.decode_example` via a `token_per_repo_id` optional argument. I first wanted to just pass `use_auth_token` but a single `Image` instance can be responsible of decoding images from a combination of several datasets together (from `interleave_datasets` for example). Therefore I just used a dictionary `repo_id` -> `token` instead.

I'm getting the `repo_id` from the dataset builder (I replaced the `namepace` attribute with `repo_id`)

I did the same for `Audio.decode_example`

cc @SBrandeis @severo


Update assin2 dataset_infos.json

Following comments in https://github.com/huggingface/datasets/issues/4003 we found that it was outdated and casing an error when loading the dataset


to_tf_dataset rewrite

[Magic is now banned](https://www.youtube.com/watch?v=WIn58XoY728#t=36s) by decree of @sgugger. This is honestly much cleaner, and the functionality will make much more sense in `transformers` anyway!

@gante I renamed the default collator to `minimal_tf_collate_fn`!

@lhoestq @sgugger @gante

I think this should now be ready, it looks good in testing! I'll try a few more notebooks today and tomorrow to be sure before I merge. Key changes are:

- No column autodetection magic (will make a separate PR to add this as a `transformers` function)
- Drops non-numerical features automatically (this is more of a 'DataLoader' method, we'll have a separate method to expose 'raw' datasets to `tf.data`)
- Better autodetection of numerical features.
- Shouldn't randomly crash mid-function :skull:

We definitely have some questions still to resolve about how to handle making a 'DataLoader' dataset versus a 'raw' dataset - see [the Notion doc](https://www.notion.so/huggingface2/Splitting-to_tf_dataset-c2e0773c4bec484384064b30ed634383) if you're interested. Still, since this PR is just fixes/improvements to an existing method which never supported non-numerical features anyway, we can merge it before we've resolved those issues, and then think about how to name and split things afterwards.

P.S. I'll take out the region comments at the end before I merge, I promise! They're just helpful while I'm editing it

+1 for the tests

> Drops non-numerical features automatically

Can you give more details on how this work and the rationale as well ? This is not explained in the docs

Also why are you adding `error_on_missing` and `auto_fix_label_names ` ? The rationale is not clear to me. In particular I think it is sensible enough to expect users to not ask columns that don't exist, and to rename a label column when required.

@lhoestq I rewrote those parts - they were causing some other issues too! `error_on_missing` and `auto_fix_label_names` have been removed. The new logic is to simply drop (before batch collation) all columns the user doesn't ask for, but not to raise errors if the user asked for columns not in the dataset, as they may be added by the collator. Hopefully this cleans it up and matches the documentation better!

Seeing some other random tests failing that don't look to be associated with this PR.

@lhoestq I can't figure out these test failures! They don't seem related to this PR at all, but I rebased to the latest version and they keep happening, even though they're not visible on master.

Thanks for the ping, will take a look tomorrow :)

Maybe the rebase didn't go well for the code recently merged about label alignment from https://github.com/huggingface/datasets/pull/4277 ?

It's very strange! The rebase looks fine to me. I might try to move my changes to a new branch from `master` and see if I can figure out which change causes this problem to appear.

@lhoestq Got it! It was caused by a name collision - I was importing `typing.Sequence`, but the code also needed `features.Sequence`. The tests from that PR were expecting the latter but got the former, and then crashed.

@lhoestq Thanks! Also, when you're ready, don't merge it immediately! I'd like to do a quick round of manual testing with the very final build once you're happy to make sure it still works in our notebooks and examples.

@lhoestq Tests look good to me, merging now!

This PR rewrites almost all of `to_tf_dataset()`, which makes it kind of hard to list all the changes, but the most critical ones are:

- Much better stability and no more dropping unexpected column names (Sorry @NielsRogge)
- Doesn't clobber custom transforms on the data (Sorry @NielsRogge again)
- Much better handling of the situation when the `collate_fn` adds columns that aren't in the dataset.
- Better inference of shapes and data types
- Lots of hacky special-casing code removed
- Can return string columns (as `tf.String`)
- Most arguments have default values, calling the method should be much simpler
- ~~Can accept a `model` argument and only return columns that are valid inputs to that model~~
- Drops the `dummy_labels` argument - this was a workaround for Keras issues that have been resolved by changes in `transformers`. Also remove it from tests and the Overview notebook.

I still have a couple of TODOs remaining and some testing to do, so don't merge yet, but it should be mostly ready for review at this point!


Timit_asr dataset cannot be previewed recently

Thanks for reporting. The bug has already been detected, and we hope to fix it soon.

TIMIT is now a dataset that requires manual download, see #4145

Therefore it might take a bit more time to fix it

> TIMIT is now a dataset that requires manual download, see #4145
>
> Therefore it might take a bit more time to fix it

Thank you for your quickly response. Exactly, I also found the manual download issue in the morning. But when I used *list_datasets()* to check the available datasets, *'timit_asr'* is still in the list. So I am a little bit confused. If *'timit_asr'* need to be manually downloaded, does that mean we can **not** automatically download it **any more** in the future?

Yes exactly. If you try to load the dataset it will ask you to download it manually first, and to pass the downloaded and extracted data like `load_dataset("timir_asr", data_dir="path/to/extracted/data")`

The URL we were using was coming from a host that doesn't have the permission to redistribute the data, and the dataset owners (LDC) notified us about it.

## Dataset viewer issue for '*timit_asr*'

**Link:** *https://huggingface.co/datasets/timit_asr*

Issue: The timit-asr dataset cannot be previewed recently.

Am I the one who added this dataset ? Yes-No
No


Add code examples to API docs

documentation

Improvements or additions to documentation

_The documentation is not available anymore as the PR was closed or merged._

> Do you think it is clearer to make every code example fully reproducible so when users copy the code they can actually run it and get an output? This seems quite repetitive - maybe even unnecessary - but it is definitely clearer.

I think it's ok to be repetitive to get more clarity. Many users come from `transformers` and may have little experience with some processing methods (especially torch users).

> Should we showcase a function with more than one parameter to highlight different use-cases (it's pretty basic right now, but I'd be happy to add more)?

Maybe let's do it case by case, depending on whether there are parameters that are likely to be used often ?

> For the class_encode_column function, let me know if there is a simpler dataset with fewer columns (currently using winograd_wsc) so it is easier for users to see what changed.

You can try with `boolq`, it has a boolean column that can be converted to labels

> Where possible, I try to show the input before and the output after using a function like flatten for example. Do you think this is too much and just showing the usage (ie, >>> ds.flatten()) will be sufficient?

No I don't think it's too much, it's nice this way thanks :)

Updated each code example so they are fully reproducible (where applicable)! The next step will be to identify some functions where we can show off some parameters that are useful or commonly used. Some useful parameters can be:

- use `map(batched=True)` to process batches of examples.
- set a seed in `shuffle`.
- set `shuffle` and `seed` in `train_test_split`.

Let me know if you think of anything else related to the functions in `arrow_dataset.py`!

Cool thanks ! I think you can also do `num_proc` for `map`

This PR adds code examples for functions related to the base Datasets class to highlight usage. Most of the examples use the `rotten_tomatoes` dataset since it is nice and small. Several things I would appreciate feedback on:

- Do you think it is clearer to make every code example fully reproducible so when users copy the code they can actually run it and get an output? This seems quite repetitive - maybe even unnecessary - but it is definitely clearer. Personally, I think we might be able to get away with not including this since users probably want to try the function on their own dataset. For example:

   ```py
   >>> from datasets import load_dataset
   >>> ds = load_dataset("rotten_tomatoes", split="validation")
   >>> code example goes here
   ```

- Should we showcase a function with more than one parameter to highlight different use-cases (it's pretty basic right now, but I'd be happy to add more)?
- For the `class_encode_column` function, let me know if there is a simpler dataset with fewer columns (currently using `winograd_wsc`) so it is easier for users to see what changed.
- Where possible, I try to show the input before and the output after using a function like `flatten` for example. Do you think this is too much and just showing the usage (ie, `>>> ds.flatten()`) will be sufficient?

Thanks :)


Avoid rate limit in update hub repositories

I also set GIT_LFS_SKIP_SMUDGE=1 to speed up git clones

use http.extraHeader to avoid rate limit


Fix exact match

Clarify docs and add clarifying example to the exact_match metric


Fix google bleu typos, examples


Fix duplicate key in multi_news

To merge after this job succeeded: https://github.com/huggingface/datasets/runs/6012207928


Optional Content Warning for Datasets

enhancement

New feature or request

Hi! You can use the `extra_gated_prompt` YAML field in a dataset card for displaying custom messages/warnings that the user must accept before gaining access to the actual dataset. This option also keeps the viewer hidden until the user agrees to terms.

Hi @mariosasko, thanks for explaining how to add this feature.

If the current dataset yaml is:
```
---
annotations_creators:
- expert
language_creators:
- expert-generated
languages:
- en
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: HatemojiBuild
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- text-classification
task_ids:
- hate-speech-detection
---
```

Can you provide a minimal working example of how to added the gated prompt?

Thanks!

```
---
annotations_creators:
- expert
language_creators:
- expert-generated
languages:
- en
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: HatemojiBuild
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- text-classification
task_ids:
- hate-speech-detection
extra_gated_prompt: "This repository contains harmful content."
---
```
\+ enable `User Access requests` under the Settings pane.

There's a brief guide here https://discuss.huggingface.co/t/how-to-customize-the-user-access-requests-message/13953 , and you can see the field in action here, https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0/blob/main/README.md (you need to agree the terms in the Dataset Card pane to be able to access the files pane, so this comes up 403 at first).

And a working example here! https://huggingface.co/datasets/DDSC/dkhate :) Great to be able to mitigate harms in text.

-- is there a way to gate content anonymously, i.e. without registering which users access it?

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

We now have hate speech datasets on the hub, like this one: https://huggingface.co/datasets/HannahRoseKirk/HatemojiBuild

I'm wondering if there is an option to select a content warning message that appears before the dataset preview? Otherwise, people immediately see hate speech when clicking on this dataset.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

Implementation of a content warning message that separates users from the dataset preview until they click out of the warning.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

Possibly just a way to remove the dataset preview completely? I think I like the content warning option better, though.

**Additional context**
Add any other context about the feature request here.


Add Conceptual 12M

Looks like your dummy_data.zip file is not in the right location ;)
datasets/datasets/conceptual_12m/dummy/default/0.0.0/dummy_data.zip
->
datasets/conceptual_12m/dummy/default/0.0.0/dummy_data.zip


Add Visual Genome

Hum there seems to be some issues with tasks in test:
 - some tasks don't fit anything in `tasks.json`. Do I remove them in `task_categories`?
 - some tasks should exist, typically `visual-question-answering` (https://github.com/huggingface/datasets/blame/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/utils/resources/tasks.json#L195) yet the exception is failing on me. I'm guessing it's because my `master` is not up-to-date. However this means that the testing only tests my branch instead of the one merged with master?

 cc @mariosasko @lhoestq

> some tasks don't fit anything in tasks.json. Do I remove them in task_categories?

You can keep them, but add `other-` as a prefix to those tasks to make the CI ignore it

> some tasks should exist, typically visual-question-answering (https://github.com/huggingface/datasets/blame/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/utils/resources/tasks.json#L195) yet the exception is failing on me. I'm guessing it's because my master is not up-to-date. However this means that the testing only tests my branch instead of the one merged with master?

Feel free to merge upstream/master into your branch ;)

EDIT: actually I just noticed you've already done this, thanks !

After offline discussions: will keep that image essentially it's necessary as I have a mapping that creates a mapping between url and local path (images are downloaded via a zip file) and dummy data needs to store that dummy image. The issue is when I read an annotation, I get a url, compute the local path, and basically I assume the local path exists since I've extracted all the images ... This isn't true if dummy data doesn't have all the images, so instead I've added a script that "fixes" the dummy data after using the CLI, it essentially adds the dummy image in the zip corresponding to the url.


RGBA images not showing

dataset-viewer

Related to the dataset viewer on huggingface.co

Thanks for reporting. It's a known issue, and we hope to fix it soon.

## Dataset viewer issue for ceyda/smithsonian_butterflies_transparent

[**Link:** *link to the dataset viewer page*](https://huggingface.co/datasets/ceyda/smithsonian_butterflies_transparent)

![image](https://user-images.githubusercontent.com/15624271/163117683-e91edb28-41bf-43d9-b371-5c62e14f40c9.png)

Am I the one who added this dataset ? Yes

👉  More of a general issue of 'RGBA' png images not being supported
(the dataset itself is just for the huggan sprint and not that important, consider it just an example)


Add `TruthfulQA` dataset

Bump. (I'm not sure which reviewer to `@` but, previously, @lhoestq has been very helpful 🤗 )


Add AUC ROC Metric


Fix formatting in BLEU metric card

Fix #4148


Adding STSb-TR dataset

Semantic Textual Similarity benchmark Turkish (STSb-TR) dataset introduced in our paper [Semantic Similarity Based Evaluation for Abstractive News Summarization](https://aclanthology.org/2021.gem-1.3.pdf) added.


Make HANS dataset streamable

Fix #4133


Generate tasks.json taxonomy from `huggingface_hub`

Ok recomputed the json file, this should be ready to review now! @lhoestq

Note: the generated JSON from `hf/hub-docs` can be found in the output of a GitHub Action run on that repo, for instance in https://github.com/huggingface/hub-docs/runs/6006686983?check_suite_focus=true

(click on "Run export-tasks script")

Should we not add the tasks with hideInDatasets?

yes, probably true – i'll change that in a PR in `hub-docs`


Adding Text-based NP Enrichment (TNE) dataset

Hey @lhoestq, can you please have a look? 🙏

Great, thanks again @lhoestq! I think we're good to go now

Added the [TNE](https://github.com/yanaiela/TNE) dataset to the library


ArrayND error in pyarrow 5

Where do we bump the required pyarrow version? Any inputs on how I fix this issue?

We need to bump it in `setup.py` as well as update some CI job to use pyarrow 6 instead of 5 in `.circleci/config.yaml` and `.github/workflows/benchmarks.yaml`

As found in https://github.com/huggingface/datasets/pull/3903, The ArrayND features fail on pyarrow 5:
```python
import pyarrow as pa
from datasets import Array2D
from datasets.table import cast_array_to_feature

arr = pa.array([[[0]]])
feature_type = Array2D(shape=(1, 1), dtype="int64")
cast_array_to_feature(arr, feature_type)
```
raises
```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-04610f9fa78c> in <module>
----> 1 cast_array_to_feature(pa.array([[[0]]]), Array2D(shape=(1, 1), dtype="int32"))

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in cast_array_to_feature(array, feature, allow_number_to_str)
   1806         return array_cast(array, get_nested_type(feature), allow_number_to_str=allow_number_to_str)
   1807     elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1808         return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
   1809     raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
   1810

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in array_cast(array, pa_type, allow_number_to_str)
   1705         array = array.storage
   1706     if isinstance(pa_type, pa.ExtensionType):
-> 1707         return pa_type.wrap_array(array)
   1708     elif pa.types.is_struct(array.type):
   1709         if pa.types.is_struct(pa_type) and (

AttributeError: 'Array2DExtensionType' object has no attribute 'wrap_array'
```

The thing is that `cast_array_to_feature` is called when writing an Arrow file, so creating an Arrow dataset using any ArrayND type currently fails.

`wrap_array` has been added in pyarrow 6, so we can either bump the required pyarrow version or fix this for pyarrow 5


Add missing label for emotion description


Inconsistent splits generation for datasets without loading script (packaged dataset puts everything into a single split)

bug

Something isn't working

## Describe the bug
Splits for dataset loaders without scripts are prepared inconsistently. I think it might be confusing for users.

## Steps to reproduce the bug
* If you load a packaged datasets from Hub, it infers splits from directory structure / filenames (check out the data [here](https://huggingface.co/datasets/nateraw/test-imagefolder-dataset)):
```python
ds = load_dataset("nateraw/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 6
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 4
    })
})
```
* If you do the same from locally stored data specifying only directory path you'll get the same:
```python
ds = load_dataset("/path/to/local/data/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 6
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 4
    })
})
```
* However, if you explicitely specify package name (like `imagefolder`, `csv`, `json`), all the data is put into a single split:
```python
ds = load_dataset("imagefolder", data_dir="/path/to/local/data/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 10
    })
})
```

## Expected results
For `load_dataset("imagefolder", data_dir="/path/to/local/data/test-imagefolder-dataset")` I expect the same output as of the two first options.


load_dataset for winoground returning decoding error

Something isn't working

I thought I had fixed it with this after some helpful hints from @severo
```python
import datasets
token = 'hf_XXXXX'
dataset = datasets.load_dataset(
    'facebook/winoground',
    name='facebook--winoground',
    split='train',
    streaming=True,
    use_auth_token=token,
)
```
but I found out that wasn't the case
```python
[x for x in dataset]
...
ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/facebook/winoground/resolve/a86a60456fbbd242e9a744199071a6bd3e7fd9de/examples.jsonl')
```

Hi ! This dataset structure (image + labels in a JSON file) is not supported yet, though we're adding support for this in  in #4069

The following structure will be supported soon:
```
metadata.json
images/
     image0.png
     image1.png
    ...
```
Where `metadata.json` is a JSON Lines file with labels or other metadata, and each line must have a "file_name" field with the name of the image file.

For the moment are only supported:
- JSON files only
- image files only

Since this dataset is a mix of the two, at the moment it fails trying to read the images as JSON.

Therefore to be able to load this dataset we need to wait for the new structure to be supported (very soon ^^), or add a dataset script in the repository that reads both the JSON and the images cc @TristanThrush

We'll also investigate the issue with the streaming download manager in https://github.com/huggingface/datasets/issues/4139 ;) thanks for reporting

In the meantime, anyone can always download the images.zip and examples.jsonl files directly from huggingface.co - let me know if anyone has issues with that.

I mirrored the files at https://huggingface.co/datasets/facebook/winoground in a folder on my local machine `winground`
and when I tried
```python
import datasets
ds = datasets.load_from_disk('./winoground')
```
I get the following error
```python
--------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 ds = datasets.load_from_disk('./winoground')

File ~/.local/lib/python3.8/site-packages/datasets/load.py:1759, in load_from_disk(dataset_path, fs, keep_in_memory)
   1757     return DatasetDict.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
   1758 else:
-> 1759     raise FileNotFoundError(
   1760         f"Directory {dataset_path} is neither a dataset directory nor a dataset dict directory."
   1761     )

FileNotFoundError: Directory ./winoground is neither a dataset directory nor a dataset dict directory.
```
so still some work to be done on the backend imo.

Note that `load_from_disk` is the function that reloads an Arrow dataset saved with `my_dataset.save_to_disk`.

Once we do support images with metadata you'll be able to use `load_dataset("facebook/winoground")` directly (or `load_dataset("./winoground")` of you've cloned the winoground repository locally).

Apologies for the delay. I added a custom dataset loading script for winoground. It should work now, with an auth token:

`examples = load_dataset('facebook/winoground', use_auth_token=<your auth token>)`

Let me know if there are any issues

Adding the dataset loading script definitely didn't take as long as I thought it would 😅

## Describe the bug
I am trying to use datasets to load winoground and I'm getting a JSON decoding error.

## Steps to reproduce the bug
```python
from datasets import load_dataset
token = 'hf_XXXXX' # my HF access token
datasets = load_dataset('facebook/winoground', use_auth_token=token)
```

## Expected results
I downloaded images.zip and examples.jsonl manually. I was expecting to have some trouble decoding json so I didn't use jsonlines but instead was able to get a complete set of 400 examples by doing
```python
import json

with open('examples.jsonl', 'r') as f:
    examples = f.read().split('\n')

# Thinking this would error if the JSON is not utf-8 encoded
json_data = [json.loads(x) for x in examples]
print(json_data[-1])
```
and I see
```python
{'caption_0': 'someone is overdoing it',
 'caption_1': 'someone is doing it over',
 'collapsed_tag': 'Relation',
 'id': 399,
 'image_0': 'ex_399_img_0',
 'image_1': 'ex_399_img_1',
 'num_main_preds': 1,
 'secondary_tag': 'Morpheme-Level',
 'tag': 'Scope, Preposition'}

```
so I'm not sure what's going on here honestly. The file `examples.jsonl` doesn't have non-UTF-8 encoded text.

## Actual results
During the split operation after downloading, datasets encounters an error in the JSON ([trace](https://gist.github.com/odellus/e55d390ca203386bf551f38e0c63a46b) abbreviated for brevity).
```
datasets/packaged_modules/json/json.py:144 in Json._generate_tables(self, files)
...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4
- Platform: Linux-5.13.0-39-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0


fix confusing bleu metric example

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
I would like to see the example in "Metric Card for BLEU" changed.
The 0th element in the predictions list is not closed in square brackets, and the 1st list is missing a comma.
The BLEU score are calculated correctly, but it is difficult to understand, so it would be helpful if you could correct this.
```
>> predictions = [
...     ["hello", "there", "general", "kenobi",       # <- no closing square bracket.
...     ["foo", "bar" "foobar"]                              # <- no comma between "bar" and "foobar"
... ]
>>> references = [
...     [["hello", "there", "general", "kenobi"]],
...     [["foo", "bar", "foobar"]]
... ]
>>> bleu = datasets.load_metric("bleu")
>>> results = bleu.compute(predictions=predictions, references=references)
>>> print(results)
{'bleu': 0.6370964381207871, ...
```

**Describe the solution you'd like**
```
>> predictions = [
...     ["hello", "there", "general", "kenobi",       # <- no closing square bracket.
...     ["foo", "bar" "foobar"]                              # <- no comma between "bar" and "foobar"
... ]
# and
>>> print(results)
{'bleu':1.0, ...
```


Adjust path to datasets tutorial in How-To

_The documentation is not available anymore as the PR was closed or merged._

The link in the How-To overview page to the Datasets tutorials is currently broken. This is just a small adjustment to make it match the format used in https://github.com/huggingface/datasets/blob/master/docs/source/tutorial.md.

(Edit to add: The link in the PR deployment (https://moon-ci-docs.huggingface.co/docs/datasets/pr_4147/en/how_to) is also broken since it's actually hardcoded to `master` and not dynamic to the branch name, but other links seem to behave similarly.)


SAMSum dataset viewer not working

bug

Something isn't working

https://huggingface.co/datasets/samsum

```
Status code:   400
Exception:     ValueError
Message:       Cannot seek streaming HTTP file
```

Currently, only the datasets that can be streamed support the dataset viewer. Maybe @lhoestq @albertvillanova or @mariosasko could give more details about why the dataset cannot be streamed.

It looks like the host (https://arxiv.org) doesn't allow HTTP Range requests, which is what we use to stream data.

This can be fix if we host the data ourselves, which is ok since the dataset is under CC BY-NC-ND 4.0

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Redirect TIMIT download from LDC

CI is failing because some tags are outdated, but they're fixed in #4067

We may do a release pretty soon (today ?), let me know if it's fine to include it in the new release

LDC data is protected under US copyright laws and under various legal agreements between the Linguistic Data Consortium/the University of Pennsylvania and data providers which prohibit redistribution of that data by anyone other than LDC. Similarly, LDC's membership agreements, non-member user agreement and various corpus-specific license agreements specifically state that users cannot publish, retransmit, disclose, copy, reproduce or redistribute LDC databases to others outside their organizations.

LDC explicitly asked us to remove the download script for the TIMIT dataset. In this PR I remove all means to download the dataset, and redirect users to download the data from https://catalog.ldc.upenn.edu/LDC93S1


Fix splits in local packaged modules, local datasets without script and hub datasets without script

Thanks !
I'm in favor of this change, even though it's a breaking change:

if you had a dataset
```
data/
  train.csv
  test.csv
```

then running this code would now return both train and test splits:
```python
load_dataset("csv", data_dir="data/")
```
whereas right now it returns only a train split with the data from both CSV files.

In my opinion it's ok do do this breaking change because:
- it makes this behavior consistent with `load_dataset("path/to/data")` that also returns both splits: data_files resolution must be the same
- I don't expect too many affected users (unless people really wanted to group train and test images in the train split on purpose ?) compared to the many new users to come (especially with #4069 )
- this usage will become more and more common as we add packaged builder and imagefolder/audiofolder usage grows, so it may be better to do this change early

Let me know if you think this is acceptable @mariosasko @albertvillanova or not, and if you think we need to first have a warning for some time before switching to this new behavior

Also, if people really want to put train and test, say, images in a single train split they could do
`load_dataset("imagefolder", data_files={"train": "/path/to/data/**})`. Probably (arguably :)), if this is a more counterintuitive case, then it should require manual files specification, not a default one (in which we expect that users do want to infer splits from filenames / dir structure but currently they have to pass smth like `{"train": "/path/to/data/train*", "test": "/path/to/data/test*"}` explicitly as `data_files`)

I also like this change, and  I don't think we even need a warning during the transition period, considering I've been asked several times since the release of `imagefolder` why splits are not correctly inferred if the directory structure is as follows:
```
data_dir
    train
        label_a
            0.jpg
            ...
        label_b
            0.jpg
            ...
    test
        label_a
            0.jpg
            ...
        label_b
            0.jpg
            ...
```

Cool ! Feel free to add a test (maybe something similar to `test_PackagedDatasetModuleFactory_with_data_dir` but with a data_dir that contains several splits) and mark this PR as ready for review then @polinaeterna :)

@lhoestq @mariosasko do you think it's a good idea to do the same with `HubDatasetModuleFactoryWithoutScript` and `LocalDatasetModuleFactoryWithoutScript` (see the latest change). If we agree on the current change, doing
```python
ds = load_dataset("polinaeterna/jsonl_test", data_dir="data/")
```
on dataset with the following structure:
```
train.jsonl
test.jsonl
data/
   train.jsonl
   test.jsonl
```
will result in having two splits from files under `data/` dir in specified repo, while master version returns a single train split.
The same would be for local dataset without script if doing smth like:
```python
ds = load_dataset("/home/polina/workspace/repos/jsonl_test", data_dir="/home/polina/workspace/repos/jsonl_test/data")
```
(though I'm not sure I understand this use case :D)
Let me know if you think we should preserve the same logic for all factories or if I should roll back this change.

@lhoestq to test passing subdirectory (`base_path`) to data_files functions and methods, I extended the temporary test directory with data so that it contains subdirectory. Because of that the number of files in this directory increased, so I had to change some numbers and patterns to account for this change - [907ddf0](https://github.com/huggingface/datasets/pull/4144/commits/907ddf09d3afece5afbae18675c859d6e453f2bf)

Do you think it's ok? Another option is to create another tmp dir and do all the checks inside it.

fixes #4150

I suggest to infer splits structure from files when `data_dir` is passed with `get_patterns_locally`, analogous to what's done in `LocalDatasetModuleFactoryWithoutScript` with `self.path`, instead of generating files with `data_dir/**` patterns and putting them all into a single default (train) split.

I would also suggest to align `HubDatasetModuleFactoryWithoutScript` and `LocalDatasetModuleFactoryWithoutScript` with this logic (remove `data_files = os.path.join(data_dir, "**")`). It's not reflected in the current code now as I'd like to discuss it cause I might be unaware of some use cases. @lhoestq @mariosasko @albertvillanova  WDYT?


Unable to download `Wikepedia` 20220301.en version

Something isn't working

Hi! We've recently updated the Wikipedia script, so these changes are only available on master and can be fetched as follows:
```python
dataset_wikipedia = load_dataset("wikipedia", "20220301.en", revision="master")
```

Hi, how can I load the previous "20200501.en" version of wikipedia which had been downloaded to the default path? Thanks!

## Describe the bug

Unable to download `Wikepedia` dataset, 20220301.en version

## Steps to reproduce the bug
```python
!pip install apache_beam mwparserfromhell
dataset_wikipedia = load_dataset("wikipedia", "20220301.en")
```

## Actual results
```
ValueError: BuilderConfig 20220301.en not found.
Available: ['20200501.aa', '20200501.ab', '20200501.ace', '20200501.ady', '20200501.af', '20200501.ak', '20200501.als', '20200501.am', '20200501.an', '20200501.ang', '20200501.ar', '20200501.arc', '20200501.arz', '20200501.as', '20200501.ast', '20200501.atj', '20200501.av', '20200501.ay', '20200501.az', '20200501.azb', '20200501.ba', '20200501.bar', '20200501.bat-smg', '20200501.bcl', '20200501.be', '20200501.be-x-old', '20200501.bg', '20200501.bh', '20200501.bi', '20200501.bjn', '20200501.bm', '20200501.bn', '20200501.bo', '20200501.bpy', '20200501.br', '20200501.bs', '20200501.bug', '20200501.bxr', '20200501.ca', '20200501.cbk-zam', '20200501.cdo', '20200501.ce', '20200501.ceb', '20200501.ch', '20200501.cho', '20200501.chr', '20200501.chy', '20200501.ckb', '20200501.co', '20200501.cr', '20200501.crh', '20200501.cs', '20200501.csb', '20200501.cu', '20200501.cv', '20200501.cy', '20200501.da', '20200501.de', '20200501.din', '20200501.diq', '20200501.dsb', '20200501.dty', '20200501.dv', '20200501.dz', '20200501.ee', '20200501.el', '20200501.eml', '20200501.en', '20200501.eo', '20200501.es', '20200501.et', '20200501.eu', '20200501.ext', '20200501.fa', '20200501.ff', '20200501.fi', '20200501.fiu-vro', '20200501.fj', '20200501.fo', '20200501.fr', '20200501.frp', '20200501.frr', '20200501.fur', '20200501.fy', '20200501.ga', '20200501.gag', '20200501.gan', '20200501.gd', '20200501.gl', '20200501.glk', '20200501.gn', '20200501.gom', '20200501.gor', '20200501.got', '20200501.gu', '20200501.gv', '20200501.ha', '20200501.hak', '20200501.haw', '20200501.he', '20200501.hi', '20200501.hif', '20200501.ho', '20200501.hr', '20200501.hsb', '20200501.ht', '20200501.hu', '20200501.hy', '20200501.ia', '20200501.id', '20200501.ie', '20200501.ig', '20200501.ii', '20200501.ik', '20200501.ilo', '20200501.inh', '20200501.io', '20200501.is', '20200501.it', '20200501.iu', '20200501.ja', '20200501.jam', '20200501.jbo', '20200501.jv', '20200501.ka', '20200501.kaa', '20200501.kab', '20200501.kbd', '20200501.kbp', '20200501.kg', '20200501.ki', '20200501.kj', '20200501.kk', '20200501.kl', '20200501.km', '20200501.kn', '20200501.ko', '20200501.koi', '20200501.krc', '20200501.ks', '20200501.ksh', '20200501.ku', '20200501.kv', '20200501.kw', '20200501.ky', '20200501.la', '20200501.lad', '20200501.lb', '20200501.lbe', '20200501.lez', '20200501.lfn', '20200501.lg', '20200501.li', '20200501.lij', '20200501.lmo', '20200501.ln', '20200501.lo', '20200501.lrc', '20200501.lt', '20200501.ltg', '20200501.lv', '20200501.mai', '20200501.map-bms', '20200501.mdf', '20200501.mg', '20200501.mh', '20200501.mhr', '20200501.mi', '20200501.min', '20200501.mk', '20200501.ml', '20200501.mn', '20200501.mr', '20200501.mrj', '20200501.ms', '20200501.mt', '20200501.mus', '20200501.mwl', '20200501.my', '20200501.myv', '20200501.mzn', '20200501.na', '20200501.nah', '20200501.nap', '20200501.nds', '20200501.nds-nl', '20200501.ne', '20200501.new', '20200501.ng', '20200501.nl', '20200501.nn', '20200501.no', '20200501.nov', '20200501.nrm', '20200501.nso', '20200501.nv', '20200501.ny', '20200501.oc', '20200501.olo', '20200501.om', '20200501.or', '20200501.os', '20200501.pa', '20200501.pag', '20200501.pam', '20200501.pap', '20200501.pcd', '20200501.pdc', '20200501.pfl', '20200501.pi', '20200501.pih', '20200501.pl', '20200501.pms', '20200501.pnb', '20200501.pnt', '20200501.ps', '20200501.pt', '20200501.qu', '20200501.rm', '20200501.rmy', '20200501.rn', '20200501.ro', '20200501.roa-rup', '20200501.roa-tara', '20200501.ru', '20200501.rue', '20200501.rw', '20200501.sa', '20200501.sah', '20200501.sat', '20200501.sc', '20200501.scn', '20200501.sco', '20200501.sd', '20200501.se', '20200501.sg', '20200501.sh', '20200501.si', '20200501.simple', '20200501.sk', '20200501.sl', '20200501.sm', '20200501.sn', '20200501.so', '20200501.sq', '20200501.sr', '20200501.srn', '20200501.ss', '20200501.st', '20200501.stq', '20200501.su', '20200501.sv', '20200501.sw', '20200501.szl', '20200501.ta', '20200501.tcy', '20200501.te', '20200501.tet', '20200501.tg', '20200501.th', '20200501.ti', '20200501.tk', '20200501.tl', '20200501.tn', '20200501.to', '20200501.tpi', '20200501.tr', '20200501.ts', '20200501.tt', '20200501.tum', '20200501.tw', '20200501.ty', '20200501.tyv', '20200501.udm', '20200501.ug', '20200501.uk', '20200501.ur', '20200501.uz', '20200501.ve', '20200501.vec', '20200501.vep', '20200501.vi', '20200501.vls', '20200501.vo', '20200501.wa', '20200501.war', '20200501.wo', '20200501.wuu', '20200501.xal', '20200501.xh', '20200501.xmf', '20200501.yi', '20200501.yo', '20200501.za', '20200501.zea', '20200501.zh', '20200501.zh-classical', '20200501.zh-min-nan', '20200501.zh-yue', '20200501.zu']
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Ubuntu
- Python version: 3.6
- PyArrow version: 6.0.1


Add ObjectFolder 2.0 dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** ObjectFolder 2.0
- **Description:** ObjectFolder 2.0 is a dataset of 1,000 objects in the form of implicit representations. It contains 1,000 Object Files each containing the complete multisensory profile for an object instance.
- **Paper:** [*link to the dataset paper if available*](https://arxiv.org/abs/2204.02389)
- **Data:** https://github.com/rhgao/ObjectFolder

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Why is the dataset not visible under the dataset preview section?


Error loading arxiv data set

bug

Something isn't working

Hi! I think this error may be related to using an older version of the library. I was able to load the dataset without any issues using the latest version of `datasets`.  Can you upgrade to the latest version of `datasets` and try again? :)

Hi! As @stevhliu suggested, to fix the issue, update the lib to the newest version with:
```
pip install -U datasets
```
and download the dataset as follows:
```python
from datasets import load_dataset
dset =  load_dataset('scientific_papers', 'arxiv', download_mode="force_redownload")
```

Thanks for the quick response! It works now. The problem is that I used nlp. load_dataset instead of datasets. load_dataset.

## Describe the bug
A clear and concise description of what the bug is.

I met the error below when loading arxiv dataset via `nlp.load_dataset('scientific_papers', 'arxiv',)`.
```
Traceback (most recent call last):
  File "scripts/summarization.py", line 354, in <module>
    main(args)
  File "scripts/summarization.py", line 306, in main
    model.hf_datasets = nlp.load_dataset('scientific_papers', 'arxiv')
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/load.py", line 549, in load_dataset
    download_config=download_config, download_mode=download_mode, ignore_verifications=ignore_verifications,
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 463, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 522, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 38, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
nlp.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?id=1b3rmCSIoh6VhD4HKWjI4HOW-cSwcwbeC&export=download', 'https://drive.google.com/uc?id=1lvsqvsFi3W-pE1SqNZI0s8NR9rC1tsja&export=download']
```

I then tried to ignore verification steps by `ignore_verifications=True` and there is another error.

```
Traceback (most recent call last):
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 537, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 810, in _prepare_split
    for key, record in utils.tqdm(generator, unit=" examples", total=split_info.num_examples, leave=False):
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/datasets/scientific_papers/9e4f2cfe3d8494e9f34a84ce49c3214605b4b52a3d8eb199104430d04c52cc12/scientific_papers.py", line 108, in _generate_examples
    with open(path, encoding="utf-8") as f:
NotADirectoryError: [Errno 20] Not a directory: '/home/username/.cache/huggingface/datasets/downloads/c0deae7af7d9c87f25dfadf621f7126f708d7dcac6d353c7564883084a000076/arxiv-dataset/train.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/summarization.py", line 354, in <module>
    main(args)
  File "scripts/summarization.py", line 306, in main
    model.hf_datasets = nlp.load_dataset('scientific_papers', 'arxiv', ignore_verifications=True)
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/load.py", line 549, in load_dataset
    download_config=download_config, download_mode=download_mode, ignore_verifications=ignore_verifications,
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 463, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/opt/conda/envs/longformer/lib/python3.7/site-packages/nlp/builder.py", line 539, in _download_and_prepare
    raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))
OSError: Cannot find data file.
```

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Dataset viewer issue for Winoground

related (same dataset): https://github.com/huggingface/datasets/issues/4149. But the issue is different. Looking at it

I thought this issue was related to the error I was seeing, but upon consideration I'd think the dataset viewer would return a 500 (unable to create the split like me) or a 404 (unable to load split b/c it was never created) error if it was having the issue I was seeing in #4149. 401 message makes it look like dataset viewer isn't passing through the identity of the user who has signed the licensing agreement when making the request to GET [examples.jsonl](https://huggingface.co/datasets/facebook/winoground/resolve/a86a60456fbbd242e9a744199071a6bd3e7fd9de/examples.jsonl).

Pinging @SBrandeis, as it seems related to gated datasets and access tokens.

To replicate:

```python
>>> import datasets
>>> dataset= datasets.load_dataset('facebook/winoground', name='facebook--winoground', split='train', use_auth_token="hf_app_...", streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 439, in wrapper
    for key, table in generate_tables_fn(**kwargs):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/packaged_modules/json/json.py", line 85, in _generate_tables
    for file_idx, file in enumerate(files):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 679, in __iter__
    yield from self.generator(*self.args, **self.kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 731, in _iter_from_urlpaths
    for dirpath, _, filenames in xwalk(urlpath, use_auth_token=use_auth_token):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 623, in xwalk
    for dirpath, dirnames, filenames in fs.walk(main_hop):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 372, in walk
    listing = self.ls(path, detail=True, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 85, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 65, in sync
    raise return_result
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 196, in _ls
    out = await self._ls_real(url, detail=detail, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 150, in _ls_real
    self._raise_not_found_for_status(r, url)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 208, in _raise_not_found_for_status
    response.raise_for_status()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/facebook/winoground/resolve/a86a60456fbbd242e9a744199071a6bd3e7fd9de/examples.jsonl')
```

*edited to fix `use_token` -> `use_auth_token`, thx @odellus*

~~Using your command to replicate and changing `use_token` to `use_auth_token` fixes the problem I was seeing in #4149.~~
Nevermind it gave me an iterator to a method returning the same 401s. Changing `use_token` to `use_auth_token` does not fix the issue.

After investigation with @severo , we found a potential culprit: https://github.com/huggingface/datasets/blob/3cd0a009a43f9f174056d70bfa2ca32216181926/src/datasets/utils/streaming_download_manager.py#L610-L624

The streaming manager does not seem to pass `use_auth_token` to `fsspec` when streaming and not iterating content of a zip archive

cc @albertvillanova @lhoestq

I was able to reproduce it on a private dataset, let me work on a fix

Hey @lhoestq, Thanks for working on a fix! Any plans to merge #4173 into master?

Thanks for the heads up, I still need to fix some tests that are failing in the CI before merging ;)

The fix has been merged, we'll do a new release soon, and update the dataset viewer

## Dataset viewer issue for 'Winoground'

**Link:** [*link to the dataset viewer page*](https://huggingface.co/datasets/facebook/winoground/viewer/facebook--winoground/train)

*short description of the issue*
Getting 401, message='Unauthorized'
The dataset is subject to authorization, but I can access the files from the interface, so I assume I'm granted to access it. I'd assume the permission somehow doesn't propagate to the dataset viewer tool.

Am I the one who added this dataset ? No


Incorrect Russian filenames encoding after extraction by datasets.DownloadManager.download_and_extract()

To reproduce:

```python
>>> import datasets
>>> datasets.get_dataset_split_names('MalakhovIlya/RuREBus', config_name='raw_txt')
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 280, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/MalakhovIlya--RuREBus/21046f5f1a0cf91187d68c30918d78d934ec7113ec435e146776d4f28f12c4ed/RuREBus.py", line 101, in _split_generators
    decode_file_names(folder)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/MalakhovIlya--RuREBus/21046f5f1a0cf91187d68c30918d78d934ec7113ec435e146776d4f28f12c4ed/RuREBus.py", line 26, in decode_file_names
    for root, dirs, files in os.walk(folder, topdown=False):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/streaming.py", line 66, in wrapper
    return function(*args, use_auth_token=use_auth_token, **kwargs)
TypeError: xwalk() got an unexpected keyword argument 'topdown'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 323, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 285, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

It's not related to the dataset viewer. Maybe @albertvillanova or @lhoestq could help more on this issue.

Hi! This issue stems from the fact that `xwalk`, which is a streamable version of `os.walk`, doesn't support the `topdown` param due to `fsspec`'s `walk` also not supporting it, so fixing this issue could be tricky.

@MalakhovIlyaPavlovich You can avoid the error by tweaking your data processing and not using this param. (and `Path.rename`, which also cannot be streamed)

@mariosasko thank you for your reply. I couldn't reproduce error showed by @severo either on Ubuntu 20.04.3 LTS, Windows 10 and Google Colab environments. But trying to avoid using os.walk(topdown=False) and Path.rename(), In _split_generators I replaced
```
def decode_file_names(folder):
    for root, dirs, files in os.walk(folder, topdown=False):
        root = Path(root)
        for file in files:
            old_name = root / Path(file)
            new_name = root / Path(
                file.encode('cp437').decode('cp866'))
            old_name.rename(new_name)
        for dir in dirs:
            old_name = root / Path(dir)
            new_name = root / Path(dir.encode('cp437').decode('cp866'))
            old_name.rename(new_name)

folder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']
decode_file_names(folder)
```
by
```
def extract(zip_file_path):
    p = Path(zip_file_path)
    dest_dir = str(p.parent / 'extracted' / p.stem)
    os.makedirs(dest_dir, exist_ok=True)
    with zipfile.ZipFile(zip_file_path) as archive:
        for file_info in tqdm(archive.infolist(), desc='Extracting'):
            filename = file_info.filename.encode('cp437').decode('cp866')
            target = os.path.join(dest_dir, *filename.split('/'))
            os.makedirs(os.path.dirname(target), exist_ok=True)
            if not file_info.is_dir():
                with archive.open(file_info) as source, open(target, 'wb') as dest:
                    shutil.copyfileobj(source, dest)
    return dest_dir

zip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']
if not is_url(zip_file):
    folder = extract(zip_file)
else:
    folder = None
```
and now everything works well except data viewer for "raw_txt" subset: dataset preview on hub shows "No data.". As far as I understand dl_manager.download returns original URL when we are calling datasets.get_dataset_split_names and my suspicions are that dataset viewer can do smth similar. I couldn't find information about how it works. I would be very grateful, if you could tell me how to fix this)

This is what I get when I try to stream the `raw_txt` subset:
```python
>>> dset = load_dataset("MalakhovIlya/RuREBus", "raw_txt", split="raw_txt", streaming=True)
>>> next(iter(dset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
```
So there is a bug in your script.

streaming=True helped me to find solution. I fixed
```
def extract(zip_file_path):
    p = Path(zip_file_path)
    dest_dir = str(p.parent / 'extracted' / p.stem)
    os.makedirs(dest_dir, exist_ok=True)
    with zipfile.ZipFile(zip_file_path) as archive:
        for file_info in tqdm(archive.infolist(), desc='Extracting'):
            filename = file_info.filename.encode('cp437').decode('cp866')
            target = os.path.join(dest_dir, *filename.split('/'))
            os.makedirs(os.path.dirname(target), exist_ok=True)
            if not file_info.is_dir():
                with archive.open(file_info) as source, open(target, 'wb') as dest:
                    shutil.copyfileobj(source, dest)
    return dest_dir

zip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']
folder = extract(zip_file)
```
by
```
folder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']
path = os.path.join(folder, 'MED_txt/unparsed_txt')
for root, dirs, files in os.walk(path):
    decoded_root_name = Path(root).name.encode('cp437').decode('cp866')
```
@mariosasko thank you for your help :)

## Dataset viewer issue for 'MalakhovIlya/RuREBus'

**Link:** https://huggingface.co/datasets/MalakhovIlya/RuREBus

**Description**
Using os.walk(topdown=False) in DatasetBuilder causes following error:
Status code:   400
Exception:     TypeError
Message:       xwalk() got an unexpected keyword argument 'topdown'
Couldn't find where "xwalk" come from. How can I fix this?

Am I the one who added this dataset ? Yes


Add single dataset citations for TweetEval

_The documentation is not available anymore as the PR was closed or merged._

The `test_dataset_cards` method is failing with the error:

```
if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           The following typing errors are found: {'annotations_creators': "(Expected `typing.List` with length > 0. Found value of type: `<class 'list'>`, with length: 0.\n)\nOR\n(Expected `typing.Dict` with length > 0. Found value of type: `<class 'list'>`, with length: 0.\n)"}
```

Adding `found` as annotation creators.

This PR adds single data citations as per request of the original creators of the TweetEval dataset.

This is a recent email from the creator:

> Could I ask you a favor? Would you be able to add at the end of the README the citations of the single datasets as well? You can just copy our readme maybe? https://github.com/cardiffnlp/tweeteval#citing-tweeteval
(just to be sure that the creator of the single datasets also get credits when tweeteval is used)

Please let me know if this looks okay or if any changes are needed.

Thanks,
Gunjan


Support streaming xtreme dataset for PAN-X config.


ELI5 supporting documents

question

Further information is requested

Hi ! Please post your question on the [forum](https://discuss.huggingface.co/), more people will be able to help you there ;)

if i am using dense search to create supporting documents for eli5 how much time it will take bcz i read somewhere that it takes about 18 hrs??


HANS dataset preview broken

The dataset cannot be loaded, be it in normal or streaming mode.

```python
>>> import datasets
>>> dataset=datasets.load_dataset("hans", split="train", streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/hans/1bbcb735c482acd54f2e118074b59cfd2bf5f7a5a285d4d540d1e632216672ac/hans.py", line 121, in _generate_examples
    for idx, line in enumerate(open(filepath, "rb")):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 1595, in __next__
    out = self.readline()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 1592, in readline
    return self.readuntil(b"\n")
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 1581, in readuntil
    self.seek(start + found + len(char))
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 676, in seek
    raise ValueError("Cannot seek streaming HTTP file")
ValueError: Cannot seek streaming HTTP file
>>> dataset=datasets.load_dataset("hans", split="train", streaming=False)
Downloading and preparing dataset hans/plain_text (download: 29.51 MiB, generated: 30.34 MiB, post-processed: Unknown size, total: 59.85 MiB) to /home/slesage/.cache/huggingface/datasets/hans/plain_text/1.0.0/1bbcb735c482acd54f2e118074b59cfd2bf5f7a5a285d4d540d1e632216672ac...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/load.py", line 1687, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 1104, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 694, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 1087, in _prepare_split
    for key, record in logging.tqdm(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/hans/1bbcb735c482acd54f2e118074b59cfd2bf5f7a5a285d4d540d1e632216672ac/hans.py", line 121, in _generate_examples
    for idx, line in enumerate(open(filepath, "rb")):
ValueError: readline of closed file
```

Hi! I've opened a PR that should make this dataset stremable. You can test it as follows:
```python
from datasets import load_dataset
dset = load_dataset("hans", split="train", streaming=True, revision="49decd29839c792ecc24ac88f861cbdec30c1c40")
```

@severo The current script doesn't throw an error in normal mode (only in streaming mode) on my local machine or in Colab. Can you update your installation of `datasets` and see if that fixes the issue?

Thanks for this. It works well, thanks! The dataset viewer is using https://github.com/huggingface/datasets/releases/tag/2.0.0, I'm eager to upgrade to 2.0.1 😉

## Dataset viewer issue for '*hans*'

**Link:** [https://huggingface.co/datasets/hans](https://huggingface.co/datasets/hans)

HANS dataset preview is broken with error 400

Am I the one who added this dataset ? No


Support streaming xtreme dataset for PAWS-X config.


Support streaming xtreme dataset for udpos config.


Add SBU Captions Photo Dataset


dataset metadata for reproducibility

enhancement

New feature or request

When pulling a dataset from the hub, it would be useful to have some metadata about the specific dataset and version that is used. The metadata could then be passed to the `Trainer` which could then be saved to a model card. This is useful for people who run many experiments on different versions (commits/branches) of the same dataset.

The dataset could have a list of “source datasets” metadata and ignore what happens to them before arriving in the Trainer (i.e. ignore mapping, filtering, etc.).

Here is a basic representation (made by @lhoestq )
```python
>>> from datasets import load_dataset
>>>
>>> my_dataset = load_dataset(...)["train"]
>>> my_dataset = my_dataset.map(...)
>>>
>>> my_dataset.sources
[HFHubDataset(repo_id=..., revision=..., arguments={...})]
```


More robust `cast_to_python_objects` in `TypedSequence`

Adds a fallback to run an expensive version of `cast_to_python_objects` which exhaustively checks entire lists to avoid the `ArrowInvalid: Could not convert` error in `TypedSequence`. Currently, this error can happen in situations where only some images are decoded in `map`, in which case `cast_to_python_objects` fails to recognize that it needs to cast `PIL.Image` objects if they are not at the beginning of the sequence and stops after the first image dictionary (e.g., if `data` is `[{"bytes": None, "path": "some path"}, PIL.Image(), ...]`)

Fix #4124


Add configs with processed data in medical_dialog dataset

There exist processed data files that do not require parsing the raw data files (which can take long time).

Fix #4122.


dataset viewer issue for common_voice

Yes, it's a known issue, and we expect to fix it soon.

Fixed.

<img width="1393" alt="Capture d’écran 2022-04-25 à 15 42 05" src="https://user-images.githubusercontent.com/1676121/165101176-d729d85b-efff-45a8-bad1-b69223edba5f.png">

## Dataset viewer issue for 'common_voice'

**Link:** https://huggingface.co/datasets/common_voice

Server Error
Status code:   400
Exception:     TypeError
Message:       __init__() got an unexpected keyword argument 'audio_column'

Am I the one who added this dataset ? No


BIG-bench

> It looks like the CI is failing on windows because our windows CI is unable to clone the bigbench repository (maybe it has to do with filenames that are longer than 256 characters, which windows don't like). Could the smaller installation of bigbench via pip solve this issue ?
> Otherwise we can see how to remove this limitation in our windows CI.

I'm not sure.
If it's git's fault that it can't handle the long filenames, it will possibly be resolved by the pip install. If it's an issue with windows not liking long filenames after it's installed, then it will not be resolved.
I don't have a windows computer to try it on, but I might be able to tweek this PR and do an experiment to find out.
We're waiting for a quota increase for the pip install (https://github.com/pypa/pypi-support/issues/1782). It's been pending for 2-3 weeks, and I don't have an estimate for when it will be resolved.

>Regarding the dummy data zip files, I think we can just keep datasets/bigbench/dummy/abstract_narrative_understanding/1.0.0/dummy_data.zip and remove all the other ones. We just require to have at least one dummy_data.zip file.

Sounds great. I will trim that down.

Do you know what are the other tests dependencies that have conflicts with bigbench ? I can try to split the CI to end up with a compatible list of test dependencies

Hi @lhoestq,

I haven't played with eliminating requirements form the test dependencies, and I've been trying to resolve this by modifying the bigbench repo to become compatible.
In the original bigbench repo, the version requirements were strict, and specifically it had a datasets==1.17.0 requirement which was causing trouble.
I'm working on PR https://github.com/google/BIG-bench/pull/766 to get some more flexible versions that might be compatible with the test dependencies in  HF/datasets.
We're somewhat flexible in modifying these version numbers if we can figure out what the exact conflict is.

I've spent some time experimenting with different versions, but I don't have a very efficient way of doing this debugging on my work computer (which for some reason doesn't produce the same sets of errors running python 3.9 instead of 3.6 or 3.7 in the tests).
It currently fails at
> The conflict is caused by:
>    bert-score 0.3.6 depends on matplotlib
>    big-bench 0.0.1 depends on matplotlib<4.0 and >=3.5.1

which doesn't seem like it can be the real issue.

If you have any advice for how to resolve these conflicts, that would be greatly appreciated!

Hi again @lhoestq,
After some more or less random guessing of conflicting packages, I've managed to find a configuration that seems to be compatible with HF/datasets.

The errors went away after removing version limits on matplotlib and scipy, and loosening numpy from 1.19 -> 1.17 in the bigbench requirements.

I might do some more tweaking to see if it lets me set some minimal limits on matplotlib and scipy, but I think we at least can move forward.

The WIN tests are still failing, now because of

> Did not find path entry C:\tools\miniconda3\bin
>C:\tools\miniconda3\envs\py37\python.exe: No module named pytest

I have no way of debugging this locally, and unless there's some way to get more verbose logs, I don't know why it's not finding pytest. Would you be able to take a quick look?

Update: Actually, I see it's still failing because of the long filenames. So perhaps the pytest error is just because the previous steps failed.

One more update on the WIN errors.
I think all the long filenames are in files in the github repo that does not need to be included.
We will try to remove them .

Hi ! The remaining error seems to be a `UnicodeDecodeError` from `setup.py`. I think you can fix your setup.py:
```diff
-     with open(os.path.join(os.path.dirname(__file__), fname)) as f:
+     with open(os.path.join(os.path.dirname(__file__), fname), encoding="utf-8") as f:
```
Indeed on windows, when you `open` a file it doesn't always use "utf-8" by default

Hi @lhoestq,
The dependency issues seems to now be resolved 🎉

Now, the WIN tests are failing at
> ERROR tests/test_arrow_dataset.py::test_dummy_dataset_serialize_s3 - botocore...
> ERROR tests/test_dataset_dict.py::test_dummy_dataset_serialize_s3 - botocore...

Is this testing the dummy dataset that's added in bigbench? If so, I might need some help getting the right format in.

The error message I'm seeing is
> raise EndpointConnectionError(endpoint_url=request.url, error=e)
> E           botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://127.0.0.1:5555/test"

Which seems unrelated, but perhaps the real issue is somewhere I'm not seeing?

Woohoo awesome !

Let me check the CI error

Can you try to re-run the CI, just in case CircleCI messed up ?

Hi @lhoestq,
Rerunning did not seem to solve the problem.
The `test_dummy_dataset_serialize_s3` error still seems to remain.

Hi again @lhoestq,
I'm not sure if this is informative or not in terms of debugging, but I deleted the dummy data and the errors for windows still fail and the others still pass.
Do you have any idea what could be causing this error on windows?

_The documentation is not available anymore as the PR was closed or merged._

Now the last question: let's have the dataset under`google/bigbench` @andersjohanandreassen ?

I think it would be nicer, this way you and anyone in your team can update the dataset card whevener you want without going through a github PR. You just need to join the https://huggingface.co/google page using your google email :)

Hi @lhoestq,

Thank you so much for the help! I really appreciate it!!!

After some discussion with the other bigbench organizers, I think there is a slight preference for bigbench to not be under google/bigbench since this is a collaboration with researchers from many different institutions/organizations beyond Google.

I see the drawback with the updates to the dataset card having to go through a PR, but hopefully that won't be very frequent.

We're finalizing putting the bigbench api on pip, so once that's finalized I just need to update the setup.py with the correct dependency and I think we are ready to merge.

I noticed that in the latest windows CI run it takes forever to install the dependencies, was there any change in the bigbench dependencies recently ?

oh, sorry! I just did a double check on the dependencies, and it seems like there is at least one left that should have been removed.  There's also one new one added.
Let me get those removed again. Will ping you here when it's updated.

It looks like there is a circular dependency in `bigbench` at https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz

```python
>>> import bigbench.api.util as bb_utils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/bigbench/api/util.py", line 29, in <module>
    import bigbench.models.query_logging_model as query_logging_model
  File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/bigbench/models/query_logging_model.py", line 23, in <module>
    import bigbench.api.util as util
AttributeError: module 'bigbench.api' has no attribute 'util'
```

Hi @lhoestq ,
I think we are ready to merge!

I have one minor question that I haven't been able to figure out:
Is there a way to bypass the `verify_infos` from triggering? I have `max_examples` as an argument to allow for selecting a fixed subset of the datasets (some of the tasks have *very* many examples). But this is a variable that's not specified by the configs, so it raises an `NonMatchingSplitsSizesError`.
I wasn't able to work my way around this, but perhaps there is a way to bypass this that I'm not seeing?
If this cannot be done, I'm happy to ignore this for now.

Regarding pypi, we are working on a release there, but I'm told there is some issue that there is a problem regarding the upload, and we are not sure when it will be resolved, and it's not in my control.
I think merging this PR with the GCS is a great idea, and I will open a new PR when the pypi version is ready.

Cool ! Merging then :D

> Is there a way to bypass the verify_infos from triggering? I have max_examples as an argument to allow for selecting a fixed subset of the datasets (some of the tasks have very many examples). But this is a variable that's not specified by the configs, so it raises an NonMatchingSplitsSizesError.

This is a bug, I opened an issue [here](https://github.com/huggingface/datasets/issues/4462). It should be easy to fix :)

The bigbench page is available here ! https://huggingface.co/datasets/bigbench

I think we can update the dataset viewer to install bigbench on it, but since this is production code I'd rather use the version on pypi for bigbench when it comes out

This PR adds all BIG-bench json tasks to huggingface/datasets.


Image decoding often fails when  transforming Image datasets

bug

Something isn't working

A quick hack I have found is that we can call the image first before running the transforms and it makes sure the image is decoded before being passed on.

For this I just needed to add `example['img'] = example['img']` to the top of my `generate_flipped_data` function, defined above, so that image decode in invoked.

After this minor change this function works:
```python
def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    example['img'] = example['img']  # <<< This is the only change
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example
```

Hi @RafayAK, thanks for reporting.

Current implementation of the Image feature performs the decoding only if the "img" field is accessed by the mapped function.

In your original `generate_flipped_data` function:
- it only accesses the "img" field (and thus performs decoding) if `rng.random() > p`;
- on the other hand, for the cases where `rng.random() <= p`, the "img" field is not accessed and thus no decoding is performed for those examples

By adding the code line `example['img'] = example['img']`, you make sure the "img" field is accessed in all cases, and the decoding is done for all examples.

Also note that there is a little bug in your implementation: `p` is not the probability of flipping, but the probability of not-flipping; the larger is `p`, the smaller is the probability of flipping.

Some refactoring (fixing also `p`):
```python
def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down.

    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    do_flip = rng.random() < p  # Note the "<" sign here instead of ">"
    example['img'] = example['img'].transpose(1) if do_flip else example['img']  # Note "img" is always accessed
    example['is_flipped'] = 1 if do_flip else 0
    return example

@albertvillanova Thanks for letting me know this is intended behavior. The docs are severely lacking on this, if I hadn't posted this here I would have never found out how I'm actually supposed to modify images in a Dataset object.

@albertvillanova Secondly if you check the error message it shows that around 1999 images were successfully created, I'm pretty sure some of them were also flipped during the process. Back to my main contention, sometimes the decoding takes place other times it fails.

I suppose to run `map` on any dataset all the examples should be invoked even if on some of them we end up doing nothing, is that right?

Hi @RafayAK! I've opened a PR with the fix, which adds a fallback to reattempt casting to PyArrow format with a more robust (but more expensive) procedure if the first attempt fails. Feel free to test it by installing `datasets` from the PR branch with the following command:
```
pip install git+https://github.com/huggingface/datasets.git@fix-4124
```

@mariosasko I'll try this right away and report back.

@mariosasko Thanks a lot for looking into this, now the `map` function at least behaves as one would expect a function to behave.

Looking forward to exploring Hugging Face more and even contributing 😃.

```bash
 $ conda list | grep datasets
datasets                  2.0.1.dev0               pypi_0    pypi

```

```python
def preprocess_data(dataset):
    """
    Helper funtion to pre-process HuggingFace Cifar-100 Dataset to remove fine_label and coarse_label columns and
    add is_flipped column
    Args:
        dataset: HuggingFace CIFAR-100 Dataset Object

    Returns:
        new_dataset: A Dataset object with "img" and "is_flipped" columns only

    """
    # remove fine_label and coarse_label columns
    new_dataset = dataset.remove_columns(['fine_label', 'coarse_label'])
    # add the column for is_flipped
    new_dataset = new_dataset.add_column(name="is_flipped", column=np.zeros((len(new_dataset)), dtype=np.uint8))

    return new_dataset

def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    # example['img'] = example['img']
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example

my_test = preprocess_data(test_dataset)
my_test = my_test.map(generate_flipped_data)
```

The output now show the function was applied successfully:
``` bash
/home/rafay/anaconda3/envs/pytorch_new/bin/python /home/rafay/Documents/you_only_live_once/upside_down_detector/create_dataset.py
Downloading builder script: 5.61kB [00:00, 3.16MB/s]
Downloading metadata: 4.21kB [00:00, 2.56MB/s]
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
100%|██████████| 10000/10000 [00:01<00:00, 5149.15ex/s]
```

## Describe the bug
When transforming/modifying images in an image dataset using the `map` function the PIL images often fail to decode in time for the image transforms, causing errors.

Using a debugger it is easy to see what the problem is, the Image decode invocation does not take place and the resulting image passed around is still raw bytes:
```
[{'bytes': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00 \x00\x00\x00 \x08\x02\x00\x00\x00\xfc\x18\xed\xa3\x00\x00\x08\x02IDATx\x9cEVIs[\xc7\x11\xeemf\xde\x82\x8d\x80\x08\x89"\xb5V\\\xb6\x94(\xe5\x9f\x90\xca5\x7f$\xa7T\xe5\x9f&9\xd9\x8a\\.\xdb\xa4$J\xa4\x00\x02x\xc0{\xb3t\xe7\x00\xca\x99\xd3\\f\xba\xba\xbf\xa5?|\xfa\xf4\xa2\xeb\xba\xedv\xa3f^\xf8\xd5\x0bY\xb6\x10\xb3\xaaDq\xcd\x83\x87\xdf5\xf3gZ\x1a\x04\x0f\xa0fp\xfa\xe0\xd4\x07?\x9dN\xc4\xb1\x99\xfd\xf2\xcb/\x97\x97\x97H\xa2\xaaf\x16\x82\xaf\xeb\xca{\xbf\xd9l.\xdf\x7f\xfa\xcb_\xff&\x88\x08\x00\x80H\xc0\x80@.;\x0f\x8c@#v\xe3\xe5\xfc\xd1\x9f\xee6q\xbf\xdf\xa6\x14\'\x93\xf1\xc3\xe5\xe3\xd1x\x14c\x8c1\xa5\x1c\x9dsM\xd3\xb4\xed\x08\x89SJ)\xa5\xedv\xbb^\xafNO\x97D\x84Hf ....
```

## Steps to reproduce the bug
```python
from datasets import load_dataset, Dataset
import numpy as np
# seeded NumPy random number generator for reprodducinble results.
rng = np.random.default_rng(seed=0)

test_dataset = load_dataset('cifar100', split="test")

def preprocess_data(dataset):
    """
    Helper function to pre-process HuggingFace Cifar-100 Dataset to remove fine_label and coarse_label columns and
    add is_flipped column
    Args:
        dataset: HuggingFace CIFAR-100 Dataset Object

    Returns:
        new_dataset: A Dataset object with "img" and "is_flipped" columns only

    """
    # remove fine_label and coarse_label columns
    new_dataset = dataset.remove_columns(['fine_label', 'coarse_label'])
    # add the column for is_flipped
    new_dataset = new_dataset.add_column(name="is_flipped", column=np.zeros((len(new_dataset)), dtype=np.uint8))

    return new_dataset

def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping function that transforms some of the images up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: the probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    # example['img'] = example['img']
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example

my_test = preprocess_data(test_dataset)
my_test = my_test.map(generate_flipped_data)

```

## Expected results
The dataset should be transformed without problems.

## Actual results
```
/home/rafay/anaconda3/envs/pytorch_new/bin/python /home/rafay/Documents/you_only_live_once/upside_down_detector/create_dataset.py
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
 20%|█▉        | 1999/10000 [00:00<00:01, 5560.44ex/s]
Traceback (most recent call last):
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2326, in _map_single
    writer.write(example)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 441, in write
    self.write_examples_on_file()
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 399, in write_examples_on_file
    self.write_batch(batch_examples=batch_examples)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 492, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 230, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 185, in __arrow_array__
    out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
  File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB size=32x32 at 0x7F56AEE61DE0> with type Image: did not recognize Python value type when inferring an Arrow data type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rafay/Documents/you_only_live_once/upside_down_detector/create_dataset.py", line 55, in <module>
    my_test = my_test.map(generate_flipped_data)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1953, in map
    return self._map_single(
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 519, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 486, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2360, in _map_single
    writer.finalize()
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 522, in finalize
    self.write_examples_on_file()
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 399, in write_examples_on_file
    self.write_batch(batch_examples=batch_examples)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 492, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 230, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 185, in __arrow_array__
    out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
  File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB size=32x32 at 0x7F56AEE61DE0> with type Image: did not recognize Python value type when inferring an Arrow data type

Process finished with exit code 1
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux(Fedora 35)
- Python version: 3.10
- PyArrow version: 7.0.0


Building C4 takes forever

Something isn't working

Hi @StellaAthena, thanks for reporting.

Please note, that our `datasets` library performs several operations in order to load a dataset, among them:
- it downloads all the required files: for C4 "en", 378.69 GB of JSON GZIPped files
- it parses their content to generate the dataset
- it caches the dataset in an Arrow file: for C4 "en", this file size is 1.87 TB
- it memory-maps the Arrow file

If it suits your use case, you might load this dataset in streaming mode:
- no Arrow file is generated
- you can iterate over elements immediately (no need to wait to download all the entire files)

```python
In [45]: from datasets import load_dataset
    ...: ds = load_dataset("c4", "en", split="train", streaming=True)
    ...: for item in ds:
    ...:     print(item)
    ...:     break
    ...:
{'text': 'Beginners BBQ Class Taking Place in Missoula!\nDo you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\nHe will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\nThe cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.', 'timestamp': '2019-04-25T12:57:54Z', 'url': 'https://klyq.com/beginners-bbq-class-taking-place-in-missoula/'}
```
I hope this is useful for your use case.

## Describe the bug
C4-en is a 300 GB dataset. However, when I try to download it through the hub it takes over _six hours_ to generate the train/test split from the downloaded files. This is an absurd amount of time and an unnecessary waste of resources.

## Steps to reproduce the bug
```python
c4 = datasets.load("c4", "en")
```

## Expected results
I would like to be able to download pre-split data.

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.13.0-35-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


medical_dialog zh has very slow _generate_examples

Something isn't working

Hi @nbroad1881, thanks for reporting.

Let me have a look to try to improve its performance.

Thanks @nbroad1881  for reporting! I don't recall it taking so long. I will also have a look at this.
@albertvillanova please let me know if I am doing something unnecessary or time consuming.

Hi @nbroad1881 and @vrindaprabhu,

As a workaround for the performance of the parsing of the raw data files (this could be addressed in a subsequent PR), I have found that there are also processed data files, that do not require parsing. I have added these as new configurations `processed.en` and `processed.zh`:
```python
ds = load_dataset("medical_dialog", "processed.zh")
```

## Describe the bug
After downloading the files from Google Drive, `load_dataset("medical_dialog", "zh", data_dir="./")` takes an unreasonable amount of time. Generating the train/test split for 33% of the dataset takes over 4.5 hours.

## Steps to reproduce the bug
The easiest way I've found to download files from Google Drive is to use `gdown` and use Google Colab because the download speeds will be very high due to the fact that they are both in Google Cloud.

```python
file_ids = [
        "1AnKxGEuzjeQsDHHqL3NqI_aplq2hVL_E",
        "1tt7weAT1SZknzRFyLXOT2fizceUUVRXX",
        "1A64VBbsQ_z8wZ2LDox586JIyyO6mIwWc",
        "1AKntx-ECnrxjB07B6BlVZcFRS4YPTB-J",
        "1xUk8AAua_x27bHUr-vNoAuhEAjTxOvsu",
        "1ezKTfe7BgqVN5o-8Vdtr9iAF0IueCSjP",
        "1tA7bSOxR1RRNqZst8cShzhuNHnayUf7c",
        "1pA3bCFA5nZDhsQutqsJcH3d712giFb0S",
        "1pTLFMdN1A3ro-KYghk4w4sMz6aGaMOdU",
        "1dUSnG0nUPq9TEQyHd6ZWvaxO0OpxVjXD",
        "1UfCH05nuWiIPbDZxQzHHGAHyMh8dmPQH",
]
for i in file_ids:
    url = f"https://drive.google.com/uc?id={i}"
    !gdown $url

from datasets import load_dataset

ds = load_dataset("medical_dialog", "zh", data_dir="./")
```

## Expected results
Faster load time

## Actual results
`Generating train split: 33%: 625519/1921127 [4:31:03<31:39:20, 11.37 examples/s]`

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

@vrindaprabhu , could you take a look at this since you implemented it? I think the `_generate_examples` function might need to be rewritten