Answers.Body: <p>If anything, you want the learning rate to <em>decrease</em> as the number of iterations increases.</p>
  <p>When you're looking for a good spot and you're clueless, take large steps. When you've found a pretty good spot, take small steps, so you don't end up far away.</p>
  <p>In other fields of machine learning, there are studies of how the learning rate should scale. For example, in traditional reinforcement learning methods, if <span class="math-container">$\alpha_i$</span> is the learning rate at step <span class="math-container">$i$</span>, then we want to have the following two criteria, to make sure we get convergence to the optimal policy:</p>
  <ol>
  <li><span class="math-container">$\sum_{i=0}^{\infty} \alpha_i = \infty$</span>. This makes sure that, no matter how bad our initial experience was, we can eventually forget it and replace it with better information.</li>
  <li><span class="math-container">$\sum_{i=0}^{\infty} \alpha_i^2 &lt; \infty$</span>. This guarantees eventual convergence.</li>
  </ol>
  <p>A typical choice here is <span class="math-container">$\alpha_i = \frac{1}{1+i}$</span>, which fits both criteria.</p>
  <p>I am unaware of similar criteria for MLPs, but if you're going to modify the step sizes, I would follow a similar approach. Make the step sizes decrease, but not too fast.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-20T03:22:40.510Z
Answers.Text: I didn't aim anything - I just wanted to know what happens when an additional factor is introduced, @RobbyGoetschalckx
Answers.UserDisplayName: Spectre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-20T03:23:20.790Z
Answers.Text: I tried to view it from the mathematical side .
Answers.UserDisplayName: Spectre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-20T03:25:08.570Z
Answers.Text: Also, I only wanted to know what happens when the factor introduced is the number of iterations at the time of correction.
Answers.UserDisplayName: Spectre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-20T03:25:25.413Z
Answers.Text: Yes, and from the mathematical side, you want to decrease the step size as you're going along. When you've already learned a lot, you want step size to be small, so you don't overshoot the mark. And when you're far away, you want step sizes to be large.
Answers.UserDisplayName: Robby Goetschalckx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-20T03:26:04.077Z
Answers.Text: I answered exactly what happens if you do that. If you scale it by the number of iterations, you will get a very unstable process.
Answers.UserDisplayName: Robby Goetschalckx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-20T03:26:31.537Z
Answers.Text: Oh... sorry for my misunderstanding - I am just 15.
Answers.UserDisplayName: Spectre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-20T03:21:11.897Z
Answers.LastActivityDate: 2020-09-20T03:21:11.897Z
Answers.UserDisplayName: Robby Goetschalckx
Body: <p>My doubt is like this :</p>
  <hr />
  <p>Suppose we have an MLP. In an MLP, as per the backprop algorithm (back-propagation algorithm), the correction applied to each weight is :</p>
  <blockquote>
  <p><span class="math-container">$$ w_{ij} := -\eta\frac{\partial E}{\partial w_{ij}}$$</span>
  (<span class="math-container">$\eta$</span> = learning rate, <span class="math-container">$E$</span> = error in the output, <span class="math-container">$w_{ij}$</span> = <span class="math-container">$i^{\text{th}}$</span> neuron in the <span class="math-container">$j^{\text{th}}$</span> row or layer)</p>
  </blockquote>
  <p>Now, if we put an extra factor in the correction as:</p>
  <blockquote>
  <p><span class="math-container">$$ w_{ij} := -k\eta  \frac{\partial E}{\partial w_{ij}}$$</span> (<span class="math-container">$k$</span> denotes the number of iterations at the time of correction)</p>
  </blockquote>
  <p>how much will that factor affect the learning of the network ? Will it affect the convergence of the  network such that it takes time to fit to the data ?</p>
  <blockquote>
  <p>NB : I am only asking this as a doubt. I haven't tried any ML projects recently, so this is not related to anything I am doing.</p>
  </blockquote>
  
Tags: <neural-networks><backpropagation><convergence>
Title: How much can an inclusion of the number of iterations have on the training of an MLP?
UserDisplayName: Spectre
ThreadText: # How much can an inclusion of the number of iterations have on the training of an MLP?
  
  (Asked by: username_0 on 2020-09-19)
  
  My doubt is like this :
  
  ---
  
  Suppose we have an MLP. In an MLP, as per the backprop algorithm (back-propagation algorithm), the correction applied to each weight is :
  
  > $$ w\_{ij} := -\eta\frac{\partial E}{\partial w\_{ij}}$$
  > ($\eta$ = learning rate, $E$ = error in the output, $w\_{ij}$ = $i^{\text{th}}$ neuron in the $j^{\text{th}}$ row or layer)
  
  Now, if we put an extra factor in the correction as:
  
  > $$ w\_{ij} := -k\eta \frac{\partial E}{\partial w\_{ij}}$$ ($k$ denotes the number of iterations at the time of correction)
  
  how much will that factor affect the learning of the network ? Will it affect the convergence of the network such that it takes time to fit to the data ?
  
  > NB : I am only asking this as a doubt. I haven't tried any ML projects recently, so this is not related to anything I am doing.
  
  **Tags:** neural-networks, backpropagation, convergence
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-20. Score: 2 | Accepted answer
  
  If anything, you want the learning rate to *decrease* as the number of iterations increases.
  
  When you're looking for a good spot and you're clueless, take large steps. When you've found a pretty good spot, take small steps, so you don't end up far away.
  
  In other fields of machine learning, there are studies of how the learning rate should scale. For example, in traditional reinforcement learning methods, if $\alpha\_i$ is the learning rate at step $i$, then we want to have the following two criteria, to make sure we get convergence to the optimal policy:
  
  1. $\sum\_{i=0}^{\infty} \alpha\_i = \infty$. This makes sure that, no matter how bad our initial experience was, we can eventually forget it and replace it with better information.
  2. $\sum\_{i=0}^{\infty} \alpha\_i^2 < \infty$. This guarantees eventual convergence.
  
  A typical choice here is $\alpha\_i = \frac{1}{1+i}$, which fits both criteria.
  
  I am unaware of similar criteria for MLPs, but if you're going to modify the step sizes, I would follow a similar approach. Make the step sizes decrease, but not too fast.
  
  > **Comments:**
  > **username_0**: I didn't aim anything - I just wanted to know what happens when an additional factor is introduced, @RobbyGoetschalckx  
  > **username_0**: I tried to view it from the mathematical side .  
  > **username_0**: Also, I only wanted to know what happens when the factor introduced is the number of iterations at the time of correction.  
  > **username_1**: Yes, and from the mathematical side, you want to decrease the step size as you're going along. When you've already learned a lot, you want step size to be small, so you don't overshoot the mark. And when you're far away, you want step sizes to be large.  
  > **username_1**: I answered exactly what happens if you do that. If you scale it by the number of iterations, you will get a very unstable process.  
  > **username_0**: Oh... sorry for my misunderstanding - I am just 15.  
  
  ---
  
  