<h1 dir="auto">🚀 Feature request</h1>
<p dir="auto">Thanks a lot for releasing LXMERT model. In the LXMERT model code samples, the visual feature extraction code (using generalized faster-rcnn: <a href="https://github.com/huggingface/transformers/blob/master/examples/lxmert/modeling_frcnn.py">modeling_frcnn</a>) only in the inference step is given. However, the visual feature extraction during the training phase is not given. For this reason if we use the same code for fine-tuning, it raises NotImplementedError as the visual feature extraction during training is not implemented. Is it possible to share the visual feature extraction during training?</p>
<p dir="auto">Right now the Trainer uses <code class="notranslate">IterableDatasetShard</code> to skip examples on each node and avoid ending up with duplicate data.<br>
This is not efficient for vision or audio tasks since we waste I/O and CPU time reading and decoding files that are not used.</p>
<p dir="auto">We consider implementing an optimized sharding for distributed training directly in <code class="notranslate">datasets</code>. Right now a <code class="notranslate">datasets.IterableDataset</code> is already a <code class="notranslate">torch.utils.data.IterableDataset</code> that automatically takes care of distributing the necessary input shards to subprocesses in single node (since <code class="notranslate">datasets</code> 2.3.0).</p>
<p dir="auto">The idea would be to also take into account the rank and world size to distribute the input shards. Maybe distributing the. <code class="notranslate">datasets.IterableDataset</code> across nodes should be asked explicitly by the user.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a> WDYT ? Do you have other ideas in mind to optimize the use of <code class="notranslate">datasets.IterableDataset</code> for distributed training in pytorch ?</p>