Thak you!
Regarding #2, my actual model is on gpu and I have an extra copy of the model (which resides on cpu) and when I calculate gradient for a layer, I copy the gradients to the cpu model too. So can I save the cpu model with ctx like ctx.cpu_model = cpu_model in forward pass, receive it in the backward pass, copy gradients from gpu model to cpu model during backward and after calling loss.backward(), get the gradients from the cpu model?