Solved by VikasRajashekar in post #7 
              
              
                Hey thanks for the input. 
I found the mistake in the understanding  

INFO:./train_decoder_only.py:Train dataset length: 112449 

Here we are printing the length of dataset: 

len(train_dataset) 

Where are 

Epoch: [1/28113] 0%| [00:00<00:01] 

is the length of the dataset loader with batch size 4…