Yes, only the gradients w.r.t. explicitly created Variables (called leaf Variables) are saved. If you want to get the grad w.r.t. some intermediate values (i.e. computed form leaf Variables) you need to use hooks (search for register_hook in the docs).