Answers.Body: <p>Yes, your reading of the paper is correct.
  Vectors generated from sinusoid functions are fixed, and are not modified during training.</p>
  <p>There exists an alternative variant -- initialize vectors as random and then update during training as usual. It also works in practice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T01:22:03.217Z
Answers.LastActivityDate: 2022-03-08T16:21:35.850Z
Answers.LastEditDate: 2022-03-08T16:21:35.850Z
Answers.UserDisplayName: user2907934
Body: <p>I'm trying to practically frame the concept of <em>positional embeddings</em> as introduced in the original <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">paper</a>.</p>
  <p>As far as I've understood, what we do is basically creating some other vectors in addition to the original <em>embeddings</em> of our inputs. So if I have my input <span class="math-container">$X \in \mathbb{R}^{n \times d}$</span> with shape <code>(64,103)</code> (so <span class="math-container">$n$</span> here is the batch size), I will be creating a matrix <span class="math-container">$P \in \mathbb{R}^{n \times d}$</span> where each <span class="math-container">$d$</span> dimensional vector will contain information about positions. Now, these vectors are generated from sinusoid functions with an initial frequency of <code>1e-4</code>, and then initial embeddings <span class="math-container">$X \in \mathbb{R}^{n \times d}$</span> and positional embeddings <span class="math-container">$P \in \mathbb{R}^{n \times d}$</span> are summed together to get a new input <span class="math-container">$X' \in \mathbb{R}^{n \times d}$</span>.</p>
  <p>Now, all this process is happening <em>before</em> the start of training right? Positional embeddings are not <em>learned/modified</em> during backpropagation?</p>
  
Tags: <natural-language-processing><transformer><positional-encoding>
Title: Are positional embeddings computed during or before training?
UserDisplayName: James Arten
ThreadText: # Are positional embeddings computed during or before training?
  
  (Asked by: username_0 on 2022-03-03)
  
  I'm trying to practically frame the concept of *positional embeddings* as introduced in the original [paper](https://arxiv.org/abs/1706.03762).
  
  As far as I've understood, what we do is basically creating some other vectors in addition to the original *embeddings* of our inputs. So if I have my input $X \in \mathbb{R}^{n \times d}$ with shape `(64,103)` (so $n$ here is the batch size), I will be creating a matrix $P \in \mathbb{R}^{n \times d}$ where each $d$ dimensional vector will contain information about positions. Now, these vectors are generated from sinusoid functions with an initial frequency of `1e-4`, and then initial embeddings $X \in \mathbb{R}^{n \times d}$ and positional embeddings $P \in \mathbb{R}^{n \times d}$ are summed together to get a new input $X' \in \mathbb{R}^{n \times d}$.
  
  Now, all this process is happening *before* the start of training right? Positional embeddings are not *learned/modified* during backpropagation?
  
  **Tags:** natural-language-processing, transformer, positional-encoding
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-08. Score: 2
  
  Yes, your reading of the paper is correct.
  Vectors generated from sinusoid functions are fixed, and are not modified during training.
  
  There exists an alternative variant -- initialize vectors as random and then update during training as usual. It also works in practice.
  
  
  ---
  
  