Seq2Seq: Option to not store whole dataset in memory
https://github.com/huggingface/transformers/blob/ef0e9d806c51059b07b98cb0279a20d3ba3cbc1d/examples/seq2seq/utils.py#L93
Do you still need help? I can help out and contribute here. <|||||>Yes that would be super helpful! The goal is to avoid using so much CPU memory here:
https://github.com/huggingface/transformers/blob/353b8f1e7a7361c0afd9e391381bc226b4a5ca8f/examples/seq2seq/utils.py#L101

by only reading a few batches from disk at a time instead of all at once. Will probably require saving in a different format.
Maybe something like this:
https://github.com/pytorch/fairseq/blob/f0a61a2774aff2efbc1adb0b5daee346a8401605/fairseq/data/data_utils.py#L55

Let me know if you need more info!<|||||>Great! 
My idea is to lazily read & encode just the required line numbers from the file when `__getitem__` is called with an index. For this we could create a map of {example number: line numbers} to read. Let me know what you think.<|||||>Sounds reasonable. What does fairseq do?<|||||>Your approach sounds good, feel free to get started.

Another approach would be not pad inputs when they are getting cached and then make a batch at load time. <|||||>I think fairseq had the data in multiple files instead of one big one. Sounds good - I am working on it. Will share when I have a tested version. <|||||>Couple of questions : I plan to get rid of the self.source variable; 
1. can I get rid of this property?https://github.com/huggingface/transformers/blob/353b8f1e7a7361c0afd9e391381bc226b4a5ca8f/examples/seq2seq/utils.py#L146-L148

2. Any ideas on how to use the sampler without the full dataset? In general shuffling and sampling may be limited with lazy datasets: although you should be able to use a random sampler in your loader.
https://github.com/huggingface/transformers/blob/353b8f1e7a7361c0afd9e391381bc226b4a5ca8f/examples/seq2seq/utils.py#L155<|||||>@sshleifer I have a working solution that works for the rest of the training loop and passes the tests. See [here](https://github.com/huggingface/transformers/compare/master...Pradhy729:lazy_loading_seq2seq)
Just need input on my points above. Let me know.<|||||>1) You can get rid of `src_lens` and `tgt_lens`, they are unused afaict

2) I would suggest trying to store the len of each example, (tokenized or untokenized, but not padded), and passing that to `SortishSampler` instead of `self.source`, and then changing
`def key(self, i): len(self.data[i])` -> `def key(self, i): self.data[i]`.

https://github.com/huggingface/transformers/blob/7e86d070c0bbed949b5c922f914f0fec44af72d4/examples/seq2seq/utils.py#L203.

<|||||>Like https://github.com/huggingface/transformers/pull/5818<|||||>Got it. Good idea!