Answers.Body: <p>When working with the Pytorch Dataset, we need to implement not just <code>__getitem__</code> but also <code>__len__</code>. When we implement <code>__len__</code> Pytorch uses it for things such as computing how many batches it can make given a batch size and a dataset size. If we request a batch size larger than the dataset size, Pytorch will simply return to us the max-size it can. So if our dataset is of length 10, and we request a batch size of 100, we will actually just get 10.</p>
  <p>In your case, your <code>__getitem__</code> takes an index, but does not appear to use it. So you can pass in 10000000000000 it's still just going to return <code>self.value</code> and <code>self.label</code> each as a tensor. You need to implement the method so that it actually uses the index to fetch a specific instance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-28T02:45:05.850Z
Answers.Text: it seems that adding len does not limit the index. I added 
  def __len__
  return 20000
  
  and index 50000 still works. should i do something more?
Answers.UserDisplayName: COTHE
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-29T07:54:45.057Z
Answers.Text: Would it be possible to update the code snippet on your question so I can see what you're working with :)?
Answers.UserDisplayName: CamWheeler135
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-27T10:30:52.423Z
Answers.LastActivityDate: 2024-06-27T10:32:40.480Z
Answers.LastEditDate: 2024-06-27T10:32:40.480Z
Answers.UserDisplayName: CamWheeler135
Body: <p>I defined a dataset as below</p>
  <pre><code>import torch
  from torch.utils.data import Dataset
  
  class Datagen(Dataset):
      def __init__(self, value, label):
          self.value = value
          self.label = label
  
      def __getitem__(self, idx):
          data = torch.tensor([self.value])
          label = torch.tensor([self.label])
          return data, label
  
  dataset = Datagen(value=1, label=10)
  
  print(dataset[50000])
  </code></pre>
  <p>and found that there's no limit to the index.
  can anyone explain the mechanism of Dataset object??
  and what happens if we make a dataloader as below?</p>
  <pre><code>dataloader = data.DataLoader(dataset, batch_size=2000)
  </code></pre>
  <p>Does the dataloader owns its size?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-27T10:37:26.347Z
Comments.Text: Programming questions (including asking about how an API or library works) are off-topic here. See https://ai.stackexchange.com/help/on-topic
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-27T10:52:17.247Z
Comments.Text: Oh I didn't know it. Thanks for noticing.
Comments.UserDisplayName: COTHE
Tags: <datasets><training-datasets>
Title: Size of Dataset object
UserDisplayName: COTHE
ThreadText: # Size of Dataset object
  
  (Asked by: username_0 on 2024-06-27)
  
  I defined a dataset as below
  
  ```
  import torch
  from torch.utils.data import Dataset
  
  class Datagen(Dataset):
      def __init__(self, value, label):
          self.value = value
          self.label = label
  
      def __getitem__(self, idx):
          data = torch.tensor([self.value])
          label = torch.tensor([self.label])
          return data, label
  
  dataset = Datagen(value=1, label=10)
  
  print(dataset[50000])
  
  ```
  
  and found that there's no limit to the index.
  can anyone explain the mechanism of Dataset object??
  and what happens if we make a dataloader as below?
  
  ```
  dataloader = data.DataLoader(dataset, batch_size=2000)
  
  ```
  
  Does the dataloader owns its size?
  
  **Tags:** datasets, training-datasets
  
  > **Comments:**
  > **username_1**: Programming questions (including asking about how an API or library works) are off-topic here. See https://ai.stackexchange.com/help/on-topic  
  > **username_0**: Oh I didn't know it. Thanks for noticing.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-06-27. Score: 0 | Accepted answer
  
  When working with the Pytorch Dataset, we need to implement not just `__getitem__` but also `__len__`. When we implement `__len__` Pytorch uses it for things such as computing how many batches it can make given a batch size and a dataset size. If we request a batch size larger than the dataset size, Pytorch will simply return to us the max-size it can. So if our dataset is of length 10, and we request a batch size of 100, we will actually just get 10.
  
  In your case, your `__getitem__` takes an index, but does not appear to use it. So you can pass in 10000000000000 it's still just going to return `self.value` and `self.label` each as a tensor. You need to implement the method so that it actually uses the index to fetch a specific instance.
  
  > **Comments:**
  > **username_0**: it seems that adding len does not limit the index. I added  def __len__ return 20000  and index 50000 still works. should i do something more?  
  > **username_2**: Would it be possible to update the code snippet on your question so I can see what you're working with :)?  
  
  ---
  
  