vadimkantorov:

I’ve tried and got an error:


That should not happen on the latest version. We added that 

Feature will-be native mini-batching support for torch.autograd.backward. Probably they could not be made parallel because of memory saving goal.

The thing is that this is not really an autograd thing, this is only true under a specific assumption that you have a batch of independent samples. Which is not true in general when not considering cummulative loss or(as soon as you have a batchnorm.
The workaround of using a for-loop looks simple enough here.