ptrblck:

Do you require gradients in the samples themselves or are you also training the network used in the sampler?


I’m training the network within the sampler. (I do require gradients for the samples, but it’s a 2nd order derivative and I’m following this neat trick here which reshapes the input and ensure they require_grads)
  
    
    
    How to calculate Laplacian (sum of 2nd derivatives) in one step autograd
  
  
    Can you use the new Hessian function introduced in 1.5? https://pytorch.org/docs/stable/autograd.html#torch.autograd.functional.hessian
  





 ptrblck:

Also, your current forward method is using the no_grad context manager. I assume the forward is used in the sampler network?


Yes, it is used. This method creates my input data, which to give it a more mathematical context rather than purely computational, are a set of positions within some N-dimensional vector space. These positions are initially randomly distributed (via torch.randn then via the Metropolis-Hastings algorthim 1 are distrubted according to my probability distribution. I don’t require any gradient computations here, only function evaluations so I decided to run the forward method with a no_grad context manager in order to 1) get a speed-up and 2) not backprop through the sampling algorithm when I call my loss function!
Does that make things clearer as to why I implemented the forward method like it that way? (If there’s a mistake, please let me know!)



 ptrblck:

If so, does the code also raise the error, if you call detach_().requires_grad_() outside of the forward on the returned samples?


Good idea. I’ll have a look and follow up with this!