Answers.Body: <p>To create custom embeddings for a dataset of legal texts in Uzbek, you can fine-tune a pre-trained encoder-only language model that is capable of capturing nuances in legal language. <a href="https://huggingface.co/docs/transformers/model_doc/xlm-roberta" rel="nofollow noreferrer">XLM-RoBERTa</a> is a multilingual variant of RoBERTa, designed to work across many languages, including low-resource languages like Uzbek. It has strong performance across a variety of tasks and can be fine-tuned for specific domains like legal texts.</p>
  <blockquote>
  <p>It is based on Facebook’s RoBERTa model released in 2019. It is a large multi-lingual language model, trained on 2.5TB of filtered CommonCrawl data... trained on 100 different languages. Unlike some XLM multilingual models, it does not require <em>lang</em> tensors to understand which language is used, and should be able to determine the correct language from the input ids.</p>
  </blockquote>
  <p>After loading the pretrained model in your fine tuning process, prepare your legal texts in a format compatible with the <code>datasets</code> library of Hugging Face, such as <code>DatasetDict</code> for train, validation, and test splits. You can fine-tune the model on a specific task like legal text classification, named entity recognition (NER), or directly train it to produce embeddings.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-12T01:46:08.573Z
Answers.LastActivityDate: 2024-08-12T01:46:08.573Z
Answers.UserDisplayName: cinch
Answers.Body: <p>We've released a framework to make LLM embeddings easier to finetune (i.e., customize), evaluate and use: <a href="https://github.com/nlp-uoregon/ullme" rel="nofollow noreferrer">github.com/nlp-uoregon/ullme</a> (paper: <a href="https://www.arxiv.org/abs/2408.03402" rel="nofollow noreferrer">arxiv.org/abs/2408.03402</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-11T01:47:37.160Z
Answers.LastActivityDate: 2024-08-11T01:47:37.160Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>I'm working on creating custom embeddings for a dataset of legal texts in a specific one language(for example, Uzbek. It can be any).</p>
  <p>How should I select a model for fine-tuning to generate accurate embeddings, particularly for specialized language like legal terminology? Should I consider the languages/scripts on which the base model was trained? What other factors should I consider when selecting the model?</p>
  <p>If possible, could you also outline the steps for doing this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-12T12:16:14.267Z
Comments.Text: Please, focus on one question and specific problem per post. You can always create multiple posts.
Comments.UserDisplayName: nbro
Tags: <natural-language-processing><large-language-models><embeddings><fine-tuning><model-request>
Title: Fine-tuning to generate accurate embeddings for specialized language?
UserDisplayName: Asadov Alisher
ThreadText: # Fine-tuning to generate accurate embeddings for specialized language?
  
  (Asked by: username_0 on 2024-08-10)
  
  I'm working on creating custom embeddings for a dataset of legal texts in a specific one language(for example, Uzbek. It can be any).
  
  How should I select a model for fine-tuning to generate accurate embeddings, particularly for specialized language like legal terminology? Should I consider the languages/scripts on which the base model was trained? What other factors should I consider when selecting the model?
  
  If possible, could you also outline the steps for doing this?
  
  **Tags:** natural-language-processing, large-language-models, embeddings, fine-tuning, model-request
  > **Comments:**
  > **username_1**: Please, focus on one question and specific problem per post. You can always create multiple posts.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-08-12. Score: 1
  
  To create custom embeddings for a dataset of legal texts in Uzbek, you can fine-tune a pre-trained encoder-only language model that is capable of capturing nuances in legal language. [XLM-RoBERTa](https://huggingface.co/docs/transformers/model_doc/xlm-roberta) is a multilingual variant of RoBERTa, designed to work across many languages, including low-resource languages like Uzbek. It has strong performance across a variety of tasks and can be fine-tuned for specific domains like legal texts.
  
  > It is based on Facebook’s RoBERTa model released in 2019. It is a large multi-lingual language model, trained on 2.5TB of filtered CommonCrawl data... trained on 100 different languages. Unlike some XLM multilingual models, it does not require *lang* tensors to understand which language is used, and should be able to determine the correct language from the input ids.
  
  After loading the pretrained model in your fine tuning process, prepare your legal texts in a format compatible with the `datasets` library of Hugging Face, such as `DatasetDict` for train, validation, and test splits. You can fine-tune the model on a specific task like legal text classification, named entity recognition (NER), or directly train it to produce embeddings.
  
  
  ---
  
  ### Answer by username_3 on 2024-08-11. Score: 1
  
  We've released a framework to make LLM embeddings easier to finetune (i.e., customize), evaluate and use: [github.com/nlp-uoregon/ullme](https://github.com/nlp-uoregon/ullme) (paper: [arxiv.org/abs/2408.03402](https://www.arxiv.org/abs/2408.03402)).
  
  
  ---
  
  