Hi everyone,
I read the paper about LR range test

  
      arxiv.org
  
  
    
1506.01186.pdf 6

965.07 KB

  
  
    
    
  
  


And the one about OneCyclePolicy

  
      arxiv.org
  
  
    
1708.07120.pdf 5

2.88 MB

  
  
    
    
  
  


I copied this implementation of the LR range test

  
      Another data science student's blog – 20 Mar 18
  
  
    

How Do You Find A Good Learning Rate 11

This is the main hyper-parameter to set when we train a neural net, but how do you determine the best value? Here's a technique to quickly decide on one.


  
  
    
    
  
  


I’ve also documented myself about de OneCyclePolicy

  
      Another data science student's blog – 7 Apr 18
  
  
    

The 1cycle policy 8

Properly setting the hyper-parameters of a neural network can be challenging, fortunately, there are some recipe that can help.


  
  
    
    
  
  


https://towardsdatascience.com/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6 3
It seems to be something many people wonder:

  
      stackoverflow.com
  
  
      
    
  

  Have I implemented implemenation of learning rate finder correctly? 9



  deep-learning, computer-vision, pytorch, mnist



  asked by
  
  
    blue-sky
  
  on 12:12PM - 06 Feb 19 UTC


  
  
    
    
  
  



  
      stackoverflow.com
  
  
      
    
  

  How to use Pytorch OneCycleLR in a training loop (and optimizer/scheduler interactions)? 7



  pytorch



  asked by
  
  
    GeneC
  
  on 01:31AM - 31 Jan 20 UTC


  
  
    
    
  
  



  
      
      mc.ai – 14 Feb 20
  
  
    

Super-Convergence with JUST PyTorch 3

Source: Deep Learning on Medium Super-Convergence with JUST PyTorchA guide to decrease training time whilst increasing results with built-in PyTorch functions and classesWhy?When creating Snaked, m…


  
  
    
    
  
  



  
      stackoverflow.com
  
  
      
    
  

  How does one use torch.optim.lr_scheduler.OneCycleLR()? 9



  python-3.x, deep-learning, pytorch, torchvision



  asked by
  
  
    Ayushman Buragohain
  
  on 02:33PM - 15 Jul 20 UTC


  
  
    
    
  
  



  
    
    
    LR parameter in optimizer larger than max_lr in OneCycleLr vision
  
  
    Hello, 
I made a little mistake in my code (I think at least …), where I put the following : 
# Optimizer
wd = 0.01
optimizer = optim.Adam(model.parameters(), lr=10e-4, weight_decay=wd)

# LR scheduler
scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=1e-4, steps_per_epoch=len(train_loader), epochs=epochs, pct_start=0.3) # only available on pytorch v1.3.0

Notice that in the optimizer I put a learning rate of 10e-4, whereas I put the max_lr on OneCycleLR to 1e-4. So in this situation w…
  


Suppose I have implemented correctly a LR Test

How is the correct orchestation between the optimizer and the one cycle scheduler? Because, the optimizer has the param lr, and the scheduler max_lr?
Can somebody provide me some example?, since I’ve been looking for it with no success

Thanks