You could access the gradient directly: model.gradient = model.net.fc.weight.grad.clone() (replace fc with the name of your classification layer) The speed is probably an issue with the loss.backward() stage, but this step is probably not the bottleneck.