Answers.Body: <p>The current parameter values are the values of the weights <span class="math-container">$w$</span> and the biases <span class="math-container">$b$</span> in the neurons in each layer.  What you are calculating is the gradient of the average loss of the batch relative to the weight and the bias.  Since the gradient points &quot;up&quot;, or toward greater loss, you multiply by the learning rate and subtract it from the current values.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-09T13:29:36.700Z
Answers.LastActivityDate: 2022-10-09T13:29:36.700Z
Answers.UserDisplayName: David Hoelzer
Body: <p>I was reading a book on Deep Learning when I came across a line, more like a few words that didn't make apparent sense.</p>
  <blockquote>
  <p>Thus, we will often settle for sampling a random minibatch of examples every time we need to compute the update, a variant called minibatch stochastic gradient descent.
  In each iteration, we first randomly sample a minibatch B consisting of a fixed number of training examples. We then compute the derivative (gradient) of the average loss on the minibatch with regard to the model parameters. Finally, we multiply the gradient by a predetermined positive value η and subtract the resulting term from the <em><strong>current parameter values.</strong></em></p>
  </blockquote>
  <blockquote>
  <p>We can express the update mathematically as follows (<span class="math-container">$∂$</span> denotes the partial derivative):</p>
  </blockquote>
  <p><span class="math-container">$(w,b) \leftarrow (w,b) - {η\over |B|} {\Large \Sigma}_{I \in B} \partial_{(w,b)}l^{(i)}(w,b).$</span></p>
  <p>The set cardinality <span class="math-container">$|B|$</span> represents the number of examples in each minibatch (the batch size) and <span class="math-container">$η$</span> denotes the learning rate.</p>
  <p>What does &quot;current&quot; parameter values mean in this context?</p>
  <p>You can <strong>find the book here</strong> -</p>
  <p><a href="https://d2l.ai" rel="nofollow noreferrer">https://d2l.ai</a> <span class="math-container">$\rightarrow$</span> Chapter <em>Linear Neural Networks, Part 3.1.1.4</em></p>
  
Tags: <machine-learning><deep-learning><optimization><gradient-descent><mini-batch-gradient-descent>
Title: What are your "current parameters" in Minibatch Stochastic Gradient Descent?
UserDisplayName: HarshDarji
ThreadText: # What are your "current parameters" in Minibatch Stochastic Gradient Descent?
  
  (Asked by: username_0 on 2022-10-09)
  
  I was reading a book on Deep Learning when I came across a line, more like a few words that didn't make apparent sense.
  
  > Thus, we will often settle for sampling a random minibatch of examples every time we need to compute the update, a variant called minibatch stochastic gradient descent.
  > In each iteration, we first randomly sample a minibatch B consisting of a fixed number of training examples. We then compute the derivative (gradient) of the average loss on the minibatch with regard to the model parameters. Finally, we multiply the gradient by a predetermined positive value η and subtract the resulting term from the ***current parameter values.***
  
  > We can express the update mathematically as follows ($∂$ denotes the partial derivative):
  
  $(w,b) \leftarrow (w,b) - {η\over |B|} {\Large \Sigma}\_{I \in B} \partial\_{(w,b)}l^{(i)}(w,b).$
  
  The set cardinality $|B|$ represents the number of examples in each minibatch (the batch size) and $η$ denotes the learning rate.
  
  What does "current" parameter values mean in this context?
  
  You can **find the book here** -
  
  <https://d2l.ai> $\rightarrow$ Chapter *Linear Neural Networks, Part 3.1.1.4*
  
  **Tags:** machine-learning, deep-learning, optimization, gradient-descent, mini-batch-gradient-descent
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-09. Score: 1
  
  The current parameter values are the values of the weights $w$ and the biases $b$ in the neurons in each layer. What you are calculating is the gradient of the average loss of the batch relative to the weight and the bias. Since the gradient points "up", or toward greater loss, you multiply by the learning rate and subtract it from the current values.
  
  
  ---
  
  