Answers.Body: <p><a href="https://arxiv.org/pdf/2103.05247.pdf" rel="nofollow noreferrer">Read the paper. It tells you.</a> (page 3)</p>
  <blockquote>
  <p><strong>Bit memory.</strong> Similar to the task proposed by Miconi et al. (2018), we consider a bit memory
  task where the model is shown 5 bitstrings each of length 1000. Afterwards, the model is shown
  a masked version of one of the bitstrings, where each bit is masked with probability 0.5, and the
  model is tasked with producing the original bitstring. The bitstrings are broken up into sequences of
  length 50, so that the models are fed 120 tokens of dimension 50.</p>
  </blockquote>
  <p>It wasn't immediately clear to me whether they train the model on the bitstrings (in which case 5 is quite low and a Transformer model is not required) or just present them to it when running the model. Figure 4 (page 8) makes it clear that it's the latter, as the original bitstrings and the query both appear on the same attention plot.</p>
  <p>i.e. the input is something like:</p>
  <pre><code>00011110011100100001000101110011...
  01000101001101100011001110010110...
  11000101111111001010011010111010...
  01011101001100100011111000000100...
  11110100011010011000101100010101...
  query
  *10**10*1**1110*****0110**11**1*...
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-10T16:13:59.743Z
Answers.LastActivityDate: 2021-08-10T16:13:59.743Z
Answers.UserDisplayName: Criticize SE actions means ban
Body: <p>I learned from <a href="https://read.deeplearning.ai/the-batch/issue-102/" rel="nofollow noreferrer">this post</a> about the so-called bit memory:</p>
  <blockquote>
  <p>They froze its self-attention and feed-forward layers and, in separate copies, fine-tuned peripheral layers on each on a wide range of tasks: Bit memory (memorizing strings of bits), Bit XOR (performing logical operations on pairs of strings of bits), ListOps (parsing and performing mathematical operations), MNIST, CIFAR-10 (classification of images), CFAR-10 LRA (classification of flattened, greyscale images), and remote homology detection (predicting what kind of protein structure an amino acid is part of).</p>
  </blockquote>
  <p>I wonder what the &quot;bit memory&quot; task is? Is it an identity function as described in <a href="https://krbnite.github.io/Deep-Learning-Nanodegree-Notes-on-Autoencoders/" rel="nofollow noreferrer">this post</a>? Or the <a href="https://paperswithcode.com/method/memory-network" rel="nofollow noreferrer">memory network</a>?</p>
  
Tags: <transformer><attention><memory>
Title: What is the bit memory task?
UserDisplayName: Lerner Zhang
ThreadText: # What is the bit memory task?
  
  (Asked by: username_0 on 2021-08-09)
  
  I learned from [this post](https://read.deeplearning.ai/the-batch/issue-102/) about the so-called bit memory:
  
  > They froze its self-attention and feed-forward layers and, in separate copies, fine-tuned peripheral layers on each on a wide range of tasks: Bit memory (memorizing strings of bits), Bit XOR (performing logical operations on pairs of strings of bits), ListOps (parsing and performing mathematical operations), MNIST, CIFAR-10 (classification of images), CFAR-10 LRA (classification of flattened, greyscale images), and remote homology detection (predicting what kind of protein structure an amino acid is part of).
  
  I wonder what the "bit memory" task is? Is it an identity function as described in [this post](https://krbnite.github.io/Deep-Learning-Nanodegree-Notes-on-Autoencoders/)? Or the [memory network](https://paperswithcode.com/method/memory-network)?
  
  **Tags:** transformer, attention, memory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-10. Score: 2 | Accepted answer
  
  [Read the paper. It tells you.](https://arxiv.org/pdf/2103.05247.pdf) (page 3)
  
  > **Bit memory.** Similar to the task proposed by Miconi et al. (2018), we consider a bit memory
  > task where the model is shown 5 bitstrings each of length 1000. Afterwards, the model is shown
  > a masked version of one of the bitstrings, where each bit is masked with probability 0.5, and the
  > model is tasked with producing the original bitstring. The bitstrings are broken up into sequences of
  > length 50, so that the models are fed 120 tokens of dimension 50.
  
  It wasn't immediately clear to me whether they train the model on the bitstrings (in which case 5 is quite low and a Transformer model is not required) or just present them to it when running the model. Figure 4 (page 8) makes it clear that it's the latter, as the original bitstrings and the query both appear on the same attention plot.
  
  i.e. the input is something like:
  
  ```
  00011110011100100001000101110011...
  01000101001101100011001110010110...
  11000101111111001010011010111010...
  01011101001100100011111000000100...
  11110100011010011000101100010101...
  query
  *10**10*1**1110*****0110**11**1*...
  
  ```
  
  
  ---
  
  