<p>When working with the Pytorch Dataset, we need to implement not just <code>__getitem__</code> but also <code>__len__</code>. When we implement <code>__len__</code> Pytorch uses it for things such as computing how many batches it can make given a batch size and a dataset size. If we request a batch size larger than the dataset size, Pytorch will simply return to us the max-size it can. So if our dataset is of length 10, and we request a batch size of 100, we will actually just get 10.</p>
<p>In your case, your <code>__getitem__</code> takes an index, but does not appear to use it. So you can pass in 10000000000000 it's still just going to return <code>self.value</code> and <code>self.label</code> each as a tensor. You need to implement the method so that it actually uses the index to fetch a specific instance.</p>

CC BY-SA 4.0
2024-06-28T02:45:05.850Z
it seems that adding len does not limit the index. I added 
def __len__
return 20000

and index 50000 still works. should i do something more?
COTHE
CC BY-SA 4.0
2024-06-29T07:54:45.057Z
Would it be possible to update the code snippet on your question so I can see what you're working with :)?
CamWheeler135
CC BY-SA 4.0
2024-06-27T10:30:52.423Z
2024-06-27T10:32:40.480Z
2024-06-27T10:32:40.480Z
CamWheeler135
<p>I defined a dataset as below</p>
<pre><code>import torch
from torch.utils.data import Dataset

class Datagen(Dataset):
    def __init__(self, value, label):
        self.value = value
        self.label = label

    def __getitem__(self, idx):
        data = torch.tensor([self.value])
        label = torch.tensor([self.label])
        return data, label

dataset = Datagen(value=1, label=10)

print(dataset[50000])
</code></pre>
<p>and found that there's no limit to the index.
can anyone explain the mechanism of Dataset object??
and what happens if we make a dataloader as below?</p>
<pre><code>dataloader = data.DataLoader(dataset, batch_size=2000)
</code></pre>
<p>Does the dataloader owns its size?</p>

CC BY-SA 4.0
2024-06-27T10:37:26.347Z
Programming questions (including asking about how an API or library works) are off-topic here. See https://ai.stackexchange.com/help/on-topic
nbro
CC BY-SA 4.0
2024-06-27T10:52:17.247Z
Oh I didn't know it. Thanks for noticing.
COTHE
<datasets><training-datasets>
Size of Dataset object
COTHE
# Size of Dataset object

(Asked by: username_0 on 2024-06-27)

I defined a dataset as below
import torch
from torch.utils.data import Dataset

class Datagen(Dataset):
    def __init__(self, value, label):
        self.value = value
        self.label = label

    def __getitem__(self, idx):
        data = torch.tensor([self.value])
        label = torch.tensor([self.label])
        return data, label

dataset = Datagen(value=1, label=10)

print(dataset[50000])

and found that there's no limit to the index.
can anyone explain the mechanism of Dataset object??
and what happens if we make a dataloader as below?
dataloader = data.DataLoader(dataset, batch_size=2000)

Does the dataloader owns its size?

**Tags:** datasets, training-datasets

> **Comments:**
> **username_1**: Programming questions (including asking about how an API or library works) are off-topic here. See https://ai.stackexchange.com/help/on-topic  
> **username_0**: Oh I didn't know it. Thanks for noticing.  

---

## 1 answers

### Answer by username_2 on 2024-06-27. Score: 0 | Accepted answer

When working with the Pytorch Dataset, we need to implement not just `__getitem__` but also `__len__`. When we implement `__len__` Pytorch uses it for things such as computing how many batches it can make given a batch size and a dataset size. If we request a batch size larger than the dataset size, Pytorch will simply return to us the max-size it can. So if our dataset is of length 10, and we request a batch size of 100, we will actually just get 10.

In your case, your `__getitem__` takes an index, but does not appear to use it. So you can pass in 10000000000000 it's still just going to return `self.value` and `self.label` each as a tensor. You need to implement the method so that it actually uses the index to fetch a specific instance.

> **Comments:**
> **username_0**: it seems that adding len does not limit the index. I added  def __len__ return 20000  and index 50000 still works. should i do something more?  
> **username_2**: Would it be possible to update the code snippet on your question so I can see what you're working with :)?  

---

