I’ll try to answer some questions, but feel free to dig deeper, if anything is still unclear.


You don’t need to use Variable since PyTorch 0.4.0 anymore. It was used to define tensors which require gradients, i.e. where autograd can automatically calculate gradients for them. In the current stable and preview release you can just use tensor as the base class. There are some special cases, e.g. where you require gradients in the input. In these cases, just pass requires_grad=True to the constructor of the tensor.


The gradient calculation and optimization can be performed as long as you use PyTorch functions throughout the forward pass. If you call any numpy functions etc. you would need to implement the backward function yourself. So, in the usual use case, your claim is right. You just need to pass the tensors and it’ll work.


Regarding the Variable question, see 1. If you detach a tensor, you make sure that autograd won’t calculate the gradients beyond this tensor. This is useful if you want to calculate the gradients for a certain amount of (time)steps, but don’t want to go further than e.g. 10 steps in the past.


Gradients are accumulated in PyTorch. If you don’t zero out the gradients after each optimizer step, your gradients would be accumulated and potentially grow. You can use this behavior to e.g. simulate larger batch sizes by accumulating the gradients for a certain amount of smaller batches.


I think it might be worth looking into this tutorial 7.


If you pass the data to your model, the intermediate tensors will be calculated and stored as they are needed for the backward pass. You could use a smaller batch size if you run out of memory or use torch.utils.checkpoint 2 to trade compute for memory.