Hi,
The gradient history is based on Variables. So if you make the loss a Variable at the last moment, the only history you have is empty.
At the moment the autograd engine only works with Tensors that are within the current process. Adding support for distributed is currently being done. You can find the PR with all the doc here 5.