text: Title: Dataset on Hub re-downloads every time?
  Body: ### Describe the bug
  
  Hi, I have a dataset on the hub [here](https://huggingface.co/datasets/manestay/borderlines). It has 1k+ downloads, which I sure is mostly just me and my colleagues working with it. It should have far fewer, since I'm using the same machine with a properly set up HF_HOME variable. However, whenever I run the below function `load_borderlines_hf`, it downloads the entire dataset from the hub and then does the other logic:
  https://github.com/manestay/borderlines/blob/4e161f444661e2ebfe643f3fe149d9258d63a57d/run_gpt/lib.py#L80
  
  Let me know what I'm doing wrong here, or if it's a bug with the `datasets` library itself. On the hub I have my data stored in CSVs, but several columns are lists, so that's why I have the code to map splitting on `;`. I looked into dataset loading scripts, but it seemed difficult to set up. I have verified that other `datasets` and `models` on my system are using the cache properly (e.g. I have a 13B parameter model and large datasets, but those are cached and don't redownload).
  
  __EDIT: __ as pointed out in the discussion below, it may be the `map()` calls that aren't being cached properly. Supposing the `load_dataset()` retrieve from the cache, then it should be the case that the `map()` calls also retrieve from the cached output. But the `map()` commands re-execute sometimes.
  
  ### Steps to reproduce the bug
  
  1. Copy and paste the function from [here](https://github.com/manestay/borderlines/blob/4e161f444661e2ebfe643f3fe149d9258d63a57d/run_gpt/lib.py#L80) (lines 80-100)
  2. Run it in Python `load_borderlines_hf(None)`
  3. It completes successfully, downloading from HF hub, then doing the mapping logic etc.
  4. If you run it again after some time, it will re-download, ignoring the cache
  
  ### Expected behavior
  
  Re-running the code, which calls `datasets.load_dataset('manestay/borderlines', 'territories')`, should use the cached version
  
  ### Environment info
  
  
  - `datasets` version: 2.16.1
  - Platform: Linux-5.14.21-150500.55.7-default-x86_64-with-glibc2.31
  - Python version: 3.10.13
  - `huggingface_hub` version: 0.20.3
  - PyArrow version: 15.0.0
  - Pandas version: 1.5.3
  - `fsspec` version: 2023.10.0