Answers.Body: <p>SBERT.net gives an example that does exactly (or very close to) what you requested. The approach is based on PCA that projects high dimensional vectors (768 dim in this case) to lower dimensions with some performance drop. It even mentioned that it's possible to reduce the original vector to dimension as low as 32.</p>
  <p>For more details, please refer to <a href="https://www.sbert.net/examples/training/distillation/README.html" rel="nofollow noreferrer">Model Distillation</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-12T18:07:04.933Z
Answers.LastActivityDate: 2022-10-12T18:07:04.933Z
Answers.UserDisplayName: RandomWalker
Body: <p>The output dimensions of BERT are 768-dimensional, is it possible to reduce them to a lower, custom number? For example, if there is another BERT-based transformer model which has a lower count of ouput dimensions, if it's possible to fine tune BERT on MLM to output lower dimension encodings etc.</p>
  <p>And if not, are there any possible workarounds for this issue?</p>
  
Tags: <machine-learning><natural-language-processing><transformer><bert>
Title: How do we reduce the output dimensions of BERT?
UserDisplayName: micawber
ThreadText: # How do we reduce the output dimensions of BERT?
  
  (Asked by: username_0 on 2022-03-05)
  
  The output dimensions of BERT are 768-dimensional, is it possible to reduce them to a lower, custom number? For example, if there is another BERT-based transformer model which has a lower count of ouput dimensions, if it's possible to fine tune BERT on MLM to output lower dimension encodings etc.
  
  And if not, are there any possible workarounds for this issue?
  
  **Tags:** machine-learning, natural-language-processing, transformer, bert
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-12. Score: 1
  
  SBERT.net gives an example that does exactly (or very close to) what you requested. The approach is based on PCA that projects high dimensional vectors (768 dim in this case) to lower dimensions with some performance drop. It even mentioned that it's possible to reduce the original vector to dimension as low as 32.
  
  For more details, please refer to [Model Distillation](https://www.sbert.net/examples/training/distillation/README.html)
  
  
  ---
  
  