All variables that don’t need a gradient have the flag requires_grad set to False, and I call the optimiser with the call torch.optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), lr)
I’ve gone back and had a look at how the number of spatial dimensions affects the time taken to do the backwards pass, and it seems to be quadratic, which I suppose makes sense as a covariance matrix is a DxD matrix. The question is, does taking 200x longer to do a backwards pass than a forwards pass make sense?