Answers.Body: <p>1-bit quantization does exist, at least at the inference stage: a common approach is to constrain weights (and sometimes activations) to be -1 or +1. I'd recommend this <a href="https://arxiv.org/abs/2004.03333" rel="nofollow noreferrer">survey paper</a> for a good overview.</p>
  <p>However, as @programjames mentioned, binarization during training is much more difficult (at least with traditional SGD) <a href="https://arxiv.org/abs/1602.02830" rel="nofollow noreferrer">as SGD explores through &quot;small and noisy steps&quot;, which isn't really possible with such low precision</a>.</p>
  <p>During inference, however, binary networks can have some nice properties, <a href="https://arxiv.org/abs/1603.05279" rel="nofollow noreferrer">like being able to perform convolutions through XNOR and bit-counting operations</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-01T22:10:51.380Z
Answers.Text: I thought Adam was better than SGD for sparse or noisy gradients.
Answers.UserDisplayName: Geremia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-01T22:16:15.450Z
Answers.Text: XNOR-Net seems interesting, as XNOR can be used to find the similarity between bit sequences.
Answers.UserDisplayName: Geremia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-01T18:36:41.357Z
Answers.LastActivityDate: 2024-06-02T03:31:20.190Z
Answers.LastEditDate: 2024-06-02T03:31:20.190Z
Answers.UserDisplayName: Alexander Wan
Answers.Body: <p>Yes, 1-bit LLMs are a thing (although, technically, they are 1.58 bit LLMs because they use a ternary system using {-1, 0, +1}). <a href="https://arxiv.org/abs/2402.17764" rel="nofollow noreferrer">This paper</a> explores practical implementations of 1.58 bit LLMs.</p>
  <p>1-bit quantization of LLMs is explored in <a href="https://arxiv.org/abs/2310.11453" rel="nofollow noreferrer">this 2023 paper</a> about BitNet models. They use <code>BitLinear</code> instead of the standard <code>nn.Linear</code>. <a href="https://github.com/kyegomez/BitNet" rel="nofollow noreferrer">This git page</a> has the full implementation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-24T06:05:20.390Z
Answers.LastActivityDate: 2024-06-24T06:05:20.390Z
Answers.UserDisplayName: ahron
Body: <p>Does 1-bit quantization machine learning exist?</p>
  <p>Pytorch's docs on &quot;<a href="https://pytorch.org/docs/stable/quantization.html" rel="nofollow noreferrer">Quantization</a>&quot; define it as:</p>
  <blockquote>
  <p>techniques for performing computations and storing tensors at lower bitwidths than floating point precision.</p>
  </blockquote>
  <p><a href="https://pytorch.org/docs/stable/tensors.html#id5" rel="nofollow noreferrer"><code>torch.bool</code> tensors exist with Pytorch</a>, but what's the use for them if most layers use floats? I see the smallest bitwidth besides <a href="https://pytorch.org/docs/stable/tensors.html#id5" rel="nofollow noreferrer"><code>torch.bool</code></a> is <a href="https://pytorch.org/docs/stable/tensors.html#id6" rel="nofollow noreferrer"><code>torch.quint4x2</code></a> (unsigned 4-bit integer), but only one sort of layer (<a href="https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html#torch.nn.EmbeddingBag" rel="nofollow noreferrer"><code>nn.EmbeddingBag</code></a> <a href="https://pytorch.org/docs/stable/nn.html#sparse-layers" rel="nofollow noreferrer">sparse layer</a>) supports it.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-01T13:15:40.667Z
Comments.Text: I don't have the time for a longer answer, but essentially no. There is 1.58-bit (ternary) networks, but only for inference (the "forward" pass). You need more bits to do gradient descent (the "backward" pass), otherwise the gradients end up pretty much random. I've tried implementing networks purely with logic gates (e.g. 3-input majority, where one input is the weight), but it couldn't train past a single layer.
Comments.UserDisplayName: programjames
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-24T06:09:14.630Z
Comments.Text: @programjames The 1-bit  BitNet model was the precursor to the 1.58 bit models. If you read The Era of 1-bit LLMs paper, where they talk about 1.58 bit models, they clearly reference the original BitNet paper that introduced 1-bit quantizations.
Comments.UserDisplayName: ahron
Tags: <machine-learning><pytorch><tensor><layers><quantization>
Title: Does 1-bit quantization (layers with boolean tensors) machine learning exist?
UserDisplayName: Geremia
ThreadText: # Does 1-bit quantization (layers with boolean tensors) machine learning exist?
  
  (Asked by: username_0 on 2024-06-01)
  
  Does 1-bit quantization machine learning exist?
  
  Pytorch's docs on "[Quantization](https://pytorch.org/docs/stable/quantization.html)" define it as:
  
  > techniques for performing computations and storing tensors at lower bitwidths than floating point precision.
  
  [`torch.bool` tensors exist with Pytorch](https://pytorch.org/docs/stable/tensors.html#id5), but what's the use for them if most layers use floats? I see the smallest bitwidth besides [`torch.bool`](https://pytorch.org/docs/stable/tensors.html#id5) is [`torch.quint4x2`](https://pytorch.org/docs/stable/tensors.html#id6) (unsigned 4-bit integer), but only one sort of layer ([`nn.EmbeddingBag`](https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html#torch.nn.EmbeddingBag) [sparse layer](https://pytorch.org/docs/stable/nn.html#sparse-layers)) supports it.
  
  **Tags:** machine-learning, pytorch, tensor, layers, quantization
  
  > **Comments:**
  > **username_1**: I don't have the time for a longer answer, but essentially no. There is 1.58-bit (ternary) networks, but only for inference (the "forward" pass). You need more bits to do gradient descent (the "backward" pass), otherwise the gradients end up pretty much random. I've tried implementing networks purely with logic gates (e.g. 3-input majority, where one input is the weight), but it couldn't train past a single layer.  
  > **username_2**: @username_1 The 1-bit  BitNet model was the precursor to the 1.58 bit models. If you read The Era of 1-bit LLMs paper, where they talk about 1.58 bit models, they clearly reference the original BitNet paper that introduced 1-bit quantizations.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-06-01. Score: 1 | Accepted answer
  
  1-bit quantization does exist, at least at the inference stage: a common approach is to constrain weights (and sometimes activations) to be -1 or +1. I'd recommend this [survey paper](https://arxiv.org/abs/2004.03333) for a good overview.
  
  However, as @username_1 mentioned, binarization during training is much more difficult (at least with traditional SGD) [as SGD explores through "small and noisy steps", which isn't really possible with such low precision](https://arxiv.org/abs/1602.02830).
  
  During inference, however, binary networks can have some nice properties, [like being able to perform convolutions through XNOR and bit-counting operations](https://arxiv.org/abs/1603.05279).
  
  > **Comments:**
  > **username_0**: I thought Adam was better than SGD for sparse or noisy gradients.  
  > **username_0**: XNOR-Net seems interesting, as XNOR can be used to find the similarity between bit sequences.  
  
  ---
  
  ### Answer by username_2 on 2024-06-24. Score: 1
  
  Yes, 1-bit LLMs are a thing (although, technically, they are 1.58 bit LLMs because they use a ternary system using {-1, 0, +1}). [This paper](https://arxiv.org/abs/2402.17764) explores practical implementations of 1.58 bit LLMs.
  
  1-bit quantization of LLMs is explored in [this 2023 paper](https://arxiv.org/abs/2310.11453) about BitNet models. They use `BitLinear` instead of the standard `nn.Linear`. [This git page](https://github.com/kyegomez/BitNet) has the full implementation.
  
  
  ---
  
  