arxiv.org
  
  
    
1211.5063.pdf 18

610.95 KB

  
  
    
    
  
  



One good heuristic for setting this threshold is to look at statistics on the average norm over a sufficiently large number of updates. In our experiments we have noticed that for a given task and model size, training is not very sensitive to this hyper-parameter and the algorithm behaves well even for rather small thresholds.