title: pytorch freeze weights and update param_groups
body: <p>Freezing weights in pytorch for <code>param_groups</code> setting.</p>
  
  <p>So if one wants to freeze weights during training:</p>
  
  <pre><code>for param in child.parameters():
      param.requires_grad = False
  </code></pre>
  
  <p>the optimizer also has to be updated to not include the non gradient weights:</p>
  
  <pre><code>optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=opt.lr, amsgrad=True)
  </code></pre>
  
  <p>If one wants to use different <code>weight_decay</code> / learning rates for bias and weights/this also allows for differing learning rates: </p>
  
  <pre><code>param_groups = [{'params': model.module.bias_parameters(), 'weight_decay': args.bias_decay},
                  {'params': model.module.weight_parameters(), 'weight_decay': args.weight_decay}]
  </code></pre>
  
  <p><code>param_groups</code> a <em>list</em> of <em>dics</em> is defined and passed into <code>SGD</code> as follows:</p>
  
  <pre><code>optimizer = torch.optim.Adam(param_groups, args.lr,
                                   betas=(args.momentum, args.beta))
  </code></pre>
  
  <p><strong>How can this be achieved with freezing individual weights</strong>? Running filter over a list of dics or is there a way of adding tensors to the optimizer separately?</p>