Also; I’m wondering that for a deep network parameters. What is the best structure to save the the most recent iteration’s gradients and each gradients can also be access through similar thing as index(list isn’t the best choice here I think)? I don’t need the all history but only the gradient information from last iteration.
For more details, I’m trying to implement algorithm 1 in this paper

  
      arxiv.org
  
  
    
1309.2388.pdf 3

1179.19 KB