HI everyone. I’m experiencing an issue that I can’t explain: I have two models which produce an output tensor with the same size (8, 8192), however calling the F.mse_loss on the output of one takes less time than calling it on the other.
See this gist to run the comparison with synthetic data:

  
      gist.github.com
  
  
    https://gist.github.com/iacolippo/9a06449fb4b819083dc61dc020dfed63 3
model_comparison
from time import time

import torch
import torch.nn as nn
import torch.nn.functional as F


class ConvNetV0(nn.Module):
    def __init__(self):
        super(ConvNetV0, self).__init__()
This file has been truncated. show original




  
  
    
    
  
  


V0 Total: 1.3554 Loss forward: 0.0042 Loss backward: 0.9177
V0 Total: 0.0344 Loss forward: 0.0132 Loss backward: 0.0091
V0 Total: 0.1393 Loss forward: 0.1089 Loss backward: 0.0085
V0 Total: 0.1405 Loss forward: 0.1095 Loss backward: 0.0085
V0 Total: 0.1380 Loss forward: 0.1079 Loss backward: 0.0083
V0 Total: 0.1379 Loss forward: 0.1077 Loss backward: 0.0083
V0 Total: 0.1380 Loss forward: 0.1077 Loss backward: 0.0084
V0 Total: 0.1388 Loss forward: 0.1085 Loss backward: 0.0084

V1 Total: 0.7927 Loss forward: 0.0001 Loss backward: 0.5020
V1 Total: 0.0289 Loss forward: 0.0066 Loss backward: 0.0195
V1 Total: 0.0267 Loss forward: 0.0211 Loss backward: 0.0026
V1 Total: 0.0337 Loss forward: 0.0283 Loss backward: 0.0026
V1 Total: 0.0341 Loss forward: 0.0287 Loss backward: 0.0030
V1 Total: 0.0336 Loss forward: 0.0285 Loss backward: 0.0030
V1 Total: 0.0335 Loss forward: 0.0287 Loss backward: 0.0030
V1 Total: 0.0335 Loss forward: 0.0289 Loss backward: 0.0030

Where does this difference come from?