<p dir="auto">Hi friends-<br>
I am trying to train a Roberta on a large corpus with a server with time limitation.<br>
Is there any way to save the model like every 3000 steps to keep record of the training, and resume it later?<br>
Really need it with the project…Thanks for helping.</p>
<h3 dir="auto">Feature request</h3>
<p dir="auto">I would like to propose that somewhere in the typical model pipeline, such as the data collators or the Trainer loop, instances of BatchEncoding should be converted to dicts to avoid breaking DataParallel models. Alternatively, I could add to the documentation to state that BatchEncodings should not be passed to PyTorch models. Maintainers, please let me know which solution you would prefer.</p>
<h3 dir="auto">Motivation</h3>
<p dir="auto">I am currently trying to integrate the sentence-transformers library with the nice Trainer API. One of the peculiarities of sentence-transformers is that rather than unpacking the model parameters like so <code class="notranslate">outputs = model(**inputs)</code>, the model expects a dict as input, which is internally unpacked. No big deal, I just override the <code class="notranslate">prediction_step</code> and remove the unpacking. However, this strangely failed when using a DataParallel setup. I realized this is because the following code in DataParallel could not handle BatchEncodings:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def scatter(inputs, target_gpus, dim=0):
    r&quot;&quot;&quot;
    Slices tensors into approximately equal chunks and
    distributes them across given GPUs. Duplicates
    references to objects that are not tensors.
    &quot;&quot;&quot;
    def scatter_map(obj):
        if isinstance(obj, torch.Tensor):
            return Scatter.apply(target_gpus, None, dim, obj)
        if _is_namedtuple(obj):
            return [type(obj)(*args) for args in zip(*map(scatter_map, obj))]
        if isinstance(obj, tuple) and len(obj) &gt; 0:
            return list(zip(*map(scatter_map, obj)))
        if isinstance(obj, list) and len(obj) &gt; 0:
            return [list(i) for i in zip(*map(scatter_map, obj))]
        if isinstance(obj, dict) and len(obj) &gt; 0:
            return [type(obj)(i) for i in zip(*map(scatter_map, obj.items()))]
        return [obj for targets in target_gpus]"><pre class="notranslate"><code class="notranslate">def scatter(inputs, target_gpus, dim=0):
    r"""
    Slices tensors into approximately equal chunks and
    distributes them across given GPUs. Duplicates
    references to objects that are not tensors.
    """
    def scatter_map(obj):
        if isinstance(obj, torch.Tensor):
            return Scatter.apply(target_gpus, None, dim, obj)
        if _is_namedtuple(obj):
            return [type(obj)(*args) for args in zip(*map(scatter_map, obj))]
        if isinstance(obj, tuple) and len(obj) &gt; 0:
            return list(zip(*map(scatter_map, obj)))
        if isinstance(obj, list) and len(obj) &gt; 0:
            return [list(i) for i in zip(*map(scatter_map, obj))]
        if isinstance(obj, dict) and len(obj) &gt; 0:
            return [type(obj)(i) for i in zip(*map(scatter_map, obj.items()))]
        return [obj for targets in target_gpus]
</code></pre></div>
<p dir="auto">DataParallel walks like a dict, talks like a dict, prints out exactly like a dict, but is not a subclass of dict, and therefore breaks this code. The tensors are not distributed across GPUs, and errors result. In my opinion, this violates the idea of duck typing, but I realize that technically it's PyTorch who's breaking duck typing here.</p>
<p dir="auto">Normally, this isn't a problem since Trainer unpacks the model args (BatchEncoding) in its predict_step, but I can't imagine sentence-transformers is the only library that does not adopt this convention.</p>
<h3 dir="auto">Your contribution</h3>
<p dir="auto">I would like to do one of three things:</p>
<ol dir="auto">
<li>Change DataCollators to output dicts or have Trainer check for BatchEncodings and convert them to dicts</li>
<li>Document that BatchEncodings should not be passed to models</li>
<li>Submit a PR to PyTorch that modifies this function to check for UserDicts like BatchEncodings</li>
</ol>