readme: ---
  language: he
  
  thumbnail: https://avatars1.githubusercontent.com/u/3617152?norod.jpg
  widget:
  - text: "מתמטיקה:"
  - text: "עליית המכונות"
  - text: "ויקיפדיה העברית"
  - text: "האירוויזיון הוא"
  - text: "דוד בן-גוריון היה"
  
  license: mit
  ---
  
  # hebrew-bad_wiki-gpt_neo-tiny
  
  ## Table of Contents
  - [Model Details](#model-details)
  - [Uses](#uses)
  - [Risks, Limitations and Biases](#risks-limitations-and-biases)
  - [Training](#training)
  - [Evaluation](#evaluation)
  - [Environmental Impact](#environmental-impact)
  - [How to Get Started With the Model](#how-to-get-started-with-the-model)
  
  ## Model Details
  **Model Description:**
  
  The model developer notes that the model is 
  > Hebrew nonsense generation model which produces really bad wiki-abstract text. 
  
  
  - **Developed by:** [Doron Adler](https://github.com/Norod)
  - **Model Type:** Text Generation
  - **Language(s):** Hebrew
  - **License:** MIT
  - **Resources for more information:**
  - [GitHub Repo](https://github.com/Norod/hebrew-gpt_neo)
  - [HuggingFace Space](https://huggingface.co/spaces/Norod78/Hebrew-GPT-Neo-Small)
  
  
  ## Uses
  
  #### Direct Use
  
  This model can be used for text generation.
  
  #### Misuse and Out-of-scope Use
  
  
  ## Risks, Limitations and Biases
  **CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.**
  
  Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
  
  
  
  
  ## Training
  
  
  #### Training Data
   [Hebrew Wikipedia Dump](https://dumps.wikimedia.org/hewiki/latest/) (hewiki abstract) from May 2020
  
  
  
  
  #### Training Procedure
  
  
  This model was fined tuned upon [hebrew-gpt_neo-tiny](https://huggingface.co/Norod78/hebrew-gpt_neo-tiny) which was previously trained using [EleutherAI's gpt-neo](https://github.com/EleutherAI/gpt-neo). 
  
  Fine-tuning on the wiki-absract text was done using [@minimaxir](https://twitter.com/minimaxir)'s [aitextgen](https://github.com/minimaxir/aitextgen).
  
  
  
  ## Evaluation
  
  
  #### Configs
  
  Model configs for the hebrew-gpt_neo-tiny is available on the [hebrew-gpt_neo model github](https://github.com/Norod/hebrew-gpt_neo/tree/main/hebrew-gpt_neo-tiny/configs) 
  
  * **Activation Function:** gelu
  * **Number_Head:** 12
  * **Number_Vocab:** 50257
  * **Train batch size:** 250
  * **Eval batch size:** 64
  * **Predict batch size:** 1
  
  
  
  
  ## Environmental Impact
  
  Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). We present the hardware type based on the [associated paper](https://arxiv.org/pdf/2105.09680.pdf).
  
  
  - **Hardware Type:** [More information needed]
  
  - **Hours used:** Unknown
  
  - **Cloud Provider:** GCP  tpu-v8s
  
  - **Compute Region:** europe-west4
  
  - **Carbon Emitted:** [More information needed]
  
  
  ## How to Get Started With the Model
  
  A Google Colab Notebook is also available [here](https://colab.research.google.com/github/Norod/hebrew-gpt_neo/blob/main/hebrew-gpt_neo-tiny/Norod78_hebrew_gpt_neo_tiny_Colab.ipynb)
  
  
  ​​
  ```
  from transformers import AutoTokenizer, AutoModelForCausalLM
  
  tokenizer = AutoTokenizer.from_pretrained("Norod78/hebrew-bad_wiki-gpt_neo-tiny")
  
  model = AutoModelForCausalLM.from_pretrained("Norod78/hebrew-bad_wiki-gpt_neo-tiny")
  
  
  ```
  
  
  