Answers.Body: <p>After researching a bit I've reached the conclusion that it must be behaving just like other networks with shared parameters do i.e. they accumulate gradients in the backpropagation stage and update it once they've backpropagated through all layers. This answer clearly explains it in the case of <a href="https://datascience.stackexchange.com/a/30923/101495">CNNs</a>. I'm sure this is the case after going through their codebase which does not have any special implementations for handling backpropagation and hence must follow the traditional route which DL libraries automatically use!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-25T17:08:28.137Z
Answers.LastActivityDate: 2022-12-25T17:08:28.137Z
Answers.UserDisplayName: FoundABetterName
Answers.Body: <p>ALBERT (A Lite BERT) is a variant of the BERT (Bidirectional Encoder Representations from Transformers) model, which is a type of transformer-based language model for natural language processing tasks such as language understanding and machine translation. Like BERT, ALBERT uses self-attention mechanisms and transformer blocks to process input sequences and generate output representations.</p>
  <p>In ALBERT, gradients are backpropagated (also known as &quot;backprop&quot;) during the training process in order to update the model's parameters and improve its performance. This is done using the standard backpropagation algorithm, which is a widely-used method for training neural networks.</p>
  <p>During backpropagation, the model computes the gradients of the loss function with respect to the model's parameters, and then uses these gradients to update the parameters in the opposite direction of the gradient. This helps to reduce the loss and improve the model's ability to make predictions.</p>
  <p>In the case of ALBERT, the model's parameters include the weights and biases of the self-attention mechanisms and transformer blocks, as well as any other trainable parameters in the model. The gradients of the loss function with respect to these parameters are computed using the chain rule, and are then used to update the parameters using an optimization algorithm such as stochastic gradient descent (SGD) or Adam.</p>
  <p>I hope this helps! Let me know if you have any other questions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-27T22:11:08.177Z
Answers.LastActivityDate: 2022-12-27T22:11:08.177Z
Answers.UserDisplayName: Manindra Sammanathegreat
Body: <p>I was reading the <a href="https://arxiv.org/abs/1909.11942" rel="nofollow noreferrer">ALBERT</a> paper and saw that they use the same parameters in each layer hence reducing the number of unique parameters. From what I could gather it seems if the all the layers have say parameters W then parameters of different layers would be updated differently which would destroy the parameter sharing.</p>
  <p>So one way I can think of is say we have only one set of parameters W and after each layer completes it's weight update W changes to W' and the preceding layers now use W'.</p>
  <p>Is this the right way to think about it or does something else happen under the hood?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-23T12:45:25.967Z
Comments.Text: It might be a good idea to provide the link to the paper.
Comments.UserDisplayName: nbro
Tags: <deep-learning><natural-language-processing><bert>
Title: How are gradients backpropogated in ALBERT?
UserDisplayName: FoundABetterName
ThreadText: # How are gradients backpropogated in ALBERT?
  
  (Asked by: username_0 on 2022-11-26)
  
  I was reading the [ALBERT](https://arxiv.org/abs/1909.11942) paper and saw that they use the same parameters in each layer hence reducing the number of unique parameters. From what I could gather it seems if the all the layers have say parameters W then parameters of different layers would be updated differently which would destroy the parameter sharing.
  
  So one way I can think of is say we have only one set of parameters W and after each layer completes it's weight update W changes to W' and the preceding layers now use W'.
  
  Is this the right way to think about it or does something else happen under the hood?
  
  **Tags:** deep-learning, natural-language-processing, bert
  > **Comments:**
  > **username_1**: It might be a good idea to provide the link to the paper.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2022-12-25. Score: 0 | Accepted answer
  
  After researching a bit I've reached the conclusion that it must be behaving just like other networks with shared parameters do i.e. they accumulate gradients in the backpropagation stage and update it once they've backpropagated through all layers. This answer clearly explains it in the case of [CNNs](https://datascience.stackexchange.com/a/30923/101495). I'm sure this is the case after going through their codebase which does not have any special implementations for handling backpropagation and hence must follow the traditional route which DL libraries automatically use!
  
  
  ---
  
  ### Answer by username_2 on 2022-12-27. Score: 0
  
  ALBERT (A Lite BERT) is a variant of the BERT (Bidirectional Encoder Representations from Transformers) model, which is a type of transformer-based language model for natural language processing tasks such as language understanding and machine translation. Like BERT, ALBERT uses self-attention mechanisms and transformer blocks to process input sequences and generate output representations.
  
  In ALBERT, gradients are backpropagated (also known as "backprop") during the training process in order to update the model's parameters and improve its performance. This is done using the standard backpropagation algorithm, which is a widely-used method for training neural networks.
  
  During backpropagation, the model computes the gradients of the loss function with respect to the model's parameters, and then uses these gradients to update the parameters in the opposite direction of the gradient. This helps to reduce the loss and improve the model's ability to make predictions.
  
  In the case of ALBERT, the model's parameters include the weights and biases of the self-attention mechanisms and transformer blocks, as well as any other trainable parameters in the model. The gradients of the loss function with respect to these parameters are computed using the chain rule, and are then used to update the parameters using an optimization algorithm such as stochastic gradient descent (SGD) or Adam.
  
  I hope this helps! Let me know if you have any other questions.
  
  
  ---
  
  