Hi Vladimir,
Thank you for your response. I am not using HF trainer, I have my own trainer, which essentially means I create my own Dataset, DataLoader, training loop etc (I have not used HF trainer till now may be I will explore that). If my understanding is right then, get_train_dataloader returns a Torch DataLoader so working with HF Trainer and my own Trainer should be very similar.
Looking at the Pytorch code it seems like when IterableDataset is used then _InfiniteConstantSampler is the what the sampler is set to. I created a custom IterableDataset in Torch (not HF) and am able to create batches using this. When I create a HF IterableDataset (using streaming=True) then I am not able to iterate on this as it throws error and you can see that the sampler is SequentialSampler.
Here is a reproducible example:

Using Torch IterableDataset and Torch DataLoader:

class CustomIterableDataset(IterableDataset):

  def __init__(self, data):
        self.data = data

  def __iter__(self):
        return iter(self.data)

data = list(range(12))
dataset = CustomIterableDataset(data)
dataloader = DataLoader(dataset, batch_size=4)
print("dataloader: ", dataloader.sampler)
for batch in dataloader:
    print(batch)

Output is:
dataloader:  <torch.utils.data.dataloader._InfiniteConstantSampler object at 0x7f1cc29e2c50>
tensor([0, 1, 2, 3])
tensor([4, 5, 6, 7])
tensor([ 8,  9, 10, 11])

Using HF IterableDataset and Torch DataLoader:

from datasets import load_dataset

dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
print(dataloader.sampler)
<torch.utils.data.sampler.SequentialSampler object at 0x7f245a510208>
for batch in dataloader:
    print(batch)

Error is:
Traceback (most recent call last):
File “”, line 1, in 
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 435, in next
data = self._next_data()
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 474, in _next_data
index = self._next_index()  # may raise StopIteration
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 427, in _next_index
return next(self._sampler_iter)  # may raise StopIteration
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/sampler.py”, line 227, in iter
for idx in self.sampler:
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/sampler.py”, line 67, in iter
return iter(range(len(self.data_source)))
TypeError: object of type ‘IterableDataset’ has no len()
Thanks,
Leena