readme: ---
  language: en
  license: mit
  ---
  
  # OpenAI GPT
  
  ## Table of Contents
  - [Model Details](#model-details)
  - [How To Get Started With the Model](#how-to-get-started-with-the-model)
  - [Uses](#uses)
  - [Risks, Limitations and Biases](#risks-limitations-and-biases)
  - [Training](#training)
  - [Evaluation](#evaluation)
  - [Environmental Impact](#environmental-impact)
  - [Technical Specifications](#technical-specifications)
  - [Citation Information](#citation-information)
  - [Model Card Authors](#model-card-authors)
  
  ## Model Details
  
  **Model Description:** `openai-gpt` is a transformer-based language model created and released by OpenAI. The model is a causal (unidirectional) transformer pre-trained using language modeling on a large corpus with long range dependencies.
  
  - **Developed by:** Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever. See [associated research paper](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) and [GitHub repo](https://github.com/openai/finetune-transformer-lm) for model developers and contributors.
  - **Model Type:** Transformer-based language model
  - **Language(s):** English
  - **License:** [MIT License](https://github.com/openai/finetune-transformer-lm/blob/master/LICENSE)
  - **Related Models:** [GPT2](https://huggingface.co/gpt2), [GPT2-Medium](https://huggingface.co/gpt2-medium), [GPT2-Large](https://huggingface.co/gpt2-large) and [GPT2-XL](https://huggingface.co/gpt2-xl)
  - **Resources for more information:**
    - [Research Paper](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)
    - [OpenAI Blog Post](https://openai.com/blog/language-unsupervised/)
    - [GitHub Repo](https://github.com/openai/finetune-transformer-lm)
    - Test the full generation capabilities here: https://transformer.huggingface.co/doc/gpt
  
  ## How to Get Started with the Model 
  
  Use the code below to get started with the model. You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
  set a seed for reproducibility:
  
  ```python
  >>> from transformers import pipeline, set_seed
  >>> generator = pipeline('text-generation', model='openai-gpt')
  >>> set_seed(42)
  >>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)
  
  [{'generated_text': "Hello, I'm a language model,'he said, when i was finished.'ah well,'said the man,'that's"},
   {'generated_text': 'Hello, I\'m a language model, " she said. \n she reached the bottom of the shaft and leaned a little further out. it was'},
   {'generated_text': 'Hello, I\'m a language model, " she laughed. " we call that a\'white girl.\'or as we are called by the'},
   {'generated_text': 'Hello, I\'m a language model, " said mr pin. " an\'the ones with the funny hats don\'t. " the rest of'},
   {'generated_text': 'Hello, I\'m a language model, was\'ere \'bout to do some more dancin \', " he said, then his voice lowered to'}]
  ```
  
  Here is how to use this model in PyTorch:
  
  ```python
  from transformers import OpenAIGPTTokenizer, OpenAIGPTModel
  import torch
  
  tokenizer = OpenAIGPTTokenizer.from_pretrained("openai-gpt")
  model = OpenAIGPTModel.from_pretrained("openai-gpt")
  
  inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
  outputs = model(**inputs)
  
  last_hidden_states = outputs.last_hidden_state
  ```
  
  and in TensorFlow:
  
  ```python
  from transformers import OpenAIGPTTokenizer, TFOpenAIGPTModel
  
  tokenizer = OpenAIGPTTokenizer.from_pretrained("openai-gpt")
  model = TFOpenAIGPTModel.from_pretrained("openai-gpt")
  
  inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
  outputs = model(inputs)
  
  last_hidden_states = outputs.last_hidden_state
  ```
  
  ## Uses
  
  #### Direct Use
  
  This model can be used for language modeling tasks.
  
  #### Downstream Use
  
  Potential downstream uses of this model include tasks that leverage language models. In the [associated paper](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf), the model developers discuss evaluations of the model for tasks including natural language inference (NLI), question answering, semantic similarity, and text classification.
  
  #### Misuse and Out-of-scope Use
  
  The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
  
  ## Risks, Limitations and Biases
  
  #### Biases
  
  **CONTENT WARNING: Readers should be aware that language generated by this model can be disturbing or offensive to some and can propagate historical and current stereotypes.**
  
  Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). 
  Predictions generated by this model can include disturbing and harmful stereotypes across protected classes; identit