<p dir="auto">Hi</p>
<p dir="auto">One may want to evaluate perplexity in language modelling task. In this case it is necessary to average over sequence length, then take exponent and only then average over batch size.</p>
<p dir="auto">For now it is impossible to implement such logic with <code class="notranslate">torch.nn.NLLLoss</code> in case of implicitly result averaging or summing up.</p>
<p dir="auto">From my point of view, it may be reasonable not to perform any implicitly operations with result of loss functions to make them more flexible.</p>
<p dir="auto">Cheers!</p>
<p dir="auto">After it was verified on <a href="https://discuss.pytorch.org/t/loaded-network-has-different-shape-but-works-anyway/19398" rel="nofollow">discuss.pytorch</a> that this is indeed unwanted behaviour, I am forwarding this to you:</p>
<h2 dir="auto">Issue description</h2>
<p dir="auto">I trained a model with among others had the following layer:<br>
<code class="notranslate">final_layer.append(nn.Conv2d(64, 1, kernel_size=1))</code><br>
and then saved it to a file with state_dict and torch.save.</p>
<p dir="auto">Then, when I wanted to load that model using load_state_dict, by accident the same layer was setup as follows:<br>
<code class="notranslate">final_layer.append(nn.Conv2d(64, 32, kernel_size=1))</code></p>
<p dir="auto">Nevertheless, the model was loaded without error. It seemed that the weights were just duplicated 32 times, but I have not verified. So the question is how this is consistent with API documentation. I have not found a statement that says load_state_dict would somehow fix shape inconsistencies automatically. It seems you have a documentation vs reality mismatch here. (Now you need to decide which one to fix)</p>
<h2 dir="auto">Code example</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="path = './test_model.pth'

model = nn.Conv2d(64, 1, 3, 1, 1)
torch.save(model.state_dict(), path)

model = nn.Conv2d(64, 32, 3, 1, 1)
model.load_state_dict(torch.load(path))

for w in model.weight[:, :, 0, 0]:
    print(w)"><pre class="notranslate"><code class="notranslate">path = './test_model.pth'

model = nn.Conv2d(64, 1, 3, 1, 1)
torch.save(model.state_dict(), path)

model = nn.Conv2d(64, 32, 3, 1, 1)
model.load_state_dict(torch.load(path))

for w in model.weight[:, :, 0, 0]:
    print(w)
</code></pre></div>
<p dir="auto">Provided by discuss .pytorch user ptrblck</p>
<h2 dir="auto">System Info</h2>
<p dir="auto">pytorch 0.4 release</p>