Hi,
I am observing some confusing performance issues when profiling using emit_nvtx 4 and Nvidia Visual Profiler.
For example, performance of MSELoss 1 in the following 2 scenarios:

A simple 3 layer BLSTM network 2
1.1 Output is contiguous
1.2 Time spent on MSELoss varies between ~800 microseconds to ~5 milliseconds
A complex network like MMDenseLSTM 1
2.1 Output is not contiguous, lots of other in-contiguities within the network
2.2 Time spent on MSELoss varies between ~150 milliseconds to ~200 milliseconds

The outputs have the same shape for both the scenarios.
This is a very simple example, similar differences are observed in the backward computations and gradient accumulation and parameter update as well.
I tried making the output contiguous just before calculating MSELoss, but without much effect (just ~5 millisecond reduction)
So my question is, what impact does contiguity of the data have on the performance of different steps, esp. MSELoss, Gradient Accumulation and Optimizer.step()? Or if there are some other considerations to be made?
@albanD, @apaszke Would be great if you could throw some light on this. 
Regards