Hello,
I like to implement the following project in PyTorch:

  
      
      GitHub
  
  
    

kutoga/going_deeper 2

going_deeper - Going Deeper: Infinite Deep Neural Networks


  
  
    
    
  
  


I already started to implement some simple networks (which is quite elegant in PyTorch).
The linked project uses a dynamic amount of layers and weights. If required, the model adds new layers. Is it possible to optimize such a newtork with a dynamic amount of layers / weights with stateful optimizers (e.g. Adam, Adagrad, etc.) with PyTorch without loosing the advantages of the internal state?
Thank you very much:slight_smile: