readme: ---
  language: en
  license: mit
  ---
  
  # GPT-2 Large
  
  ## Table of Contents
  - [Model Details](#model-details)
  - [How To Get Started With the Model](#how-to-get-started-with-the-model)
  - [Uses](#uses)
  - [Risks, Limitations and Biases](#risks-limitations-and-biases)
  - [Training](#training)
  - [Evaluation](#evaluation)
  - [Environmental Impact](#environmental-impact)
  - [Technical Specifications](#technical-specifications)
  - [Citation Information](#citation-information)
  - [Model Card Authors](#model-card-author)
  
  ## Model Details
  
  **Model Description:** GPT-2 Large is the **774M parameter** version of GPT-2, a transformer-based language model created and released by OpenAI. The model is a pretrained model on English language using a causal language modeling (CLM) objective. 
  
  - **Developed by:** OpenAI, see [associated research paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) and [GitHub repo](https://github.com/openai/gpt-2) for model developers.
  - **Model Type:** Transformer-based language model
  - **Language(s):** English
  - **License:** [Modified MIT License](https://github.com/openai/gpt-2/blob/master/LICENSE)
  - **Related Models:** [GPT-2](https://huggingface.co/gpt2), [GPT-Medium](https://huggingface.co/gpt2-medium) and [GPT-XL](https://huggingface.co/gpt2-xl)
  - **Resources for more information:**
    - [Research Paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
    - [OpenAI Blog Post](https://openai.com/blog/better-language-models/)
    - [GitHub Repo](https://github.com/openai/gpt-2)
    - [OpenAI Model Card for GPT-2](https://github.com/openai/gpt-2/blob/master/model_card.md)
    - Test the full generation capabilities here: https://transformer.huggingface.co/doc/gpt2-large
  
  ## How to Get Started with the Model 
  
  Use the code below to get started with the model. You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
  set a seed for reproducibility:
  
  ```python
  >>> from transformers import pipeline, set_seed
  >>> generator = pipeline('text-generation', model='gpt2-large')
  >>> set_seed(42)
  >>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)
  
  [{'generated_text': "Hello, I'm a language model, I can do language modeling. In fact, this is one of the reasons I use languages. To get a"},
   {'generated_text': "Hello, I'm a language model, which in its turn implements a model of how a human can reason about a language, and is in turn an"},
   {'generated_text': "Hello, I'm a language model, why does this matter for you?\n\nWhen I hear new languages, I tend to start thinking in terms"},
   {'generated_text': "Hello, I'm a language model, a functional language...\n\nI don't need to know anything else. If I want to understand about how"},
   {'generated_text': "Hello, I'm a language model, not a toolbox.\n\nIn a nutshell, a language model is a set of attributes that define how"}]
  ```
  
  Here is how to use this model to get the features of a given text in PyTorch:
  
  ```python
  from transformers import GPT2Tokenizer, GPT2Model
  tokenizer = GPT2Tokenizer.from_pretrained('gpt2-large')
  model = GPT2Model.from_pretrained('gpt2-large')
  text = "Replace me by any text you'd like."
  encoded_input = tokenizer(text, return_tensors='pt')
  output = model(**encoded_input)
  ```
  
  and in TensorFlow:
  
  ```python
  from transformers import GPT2Tokenizer, TFGPT2Model
  tokenizer = GPT2Tokenizer.from_pretrained('gpt2-large')
  model = TFGPT2Model.from_pretrained('gpt2-large')
  text = "Replace me by any text you'd like."
  encoded_input = tokenizer(text, return_tensors='tf')
  output = model(encoded_input)
  ```
  
  ## Uses
  
  #### Direct Use
  
  In their [model card about GPT-2](https://github.com/openai/gpt-2/blob/master/model_card.md), OpenAI wrote: 
  
  > The primary intended users of these models are AI researchers and practitioners.
  > 
  > We primarily imagine these language models will be used by researchers to better understand the behaviors, capabilities, biases, and constraints of large-scale generative language models.
  
  #### Downstream Use
  
  In their [model card about GPT-2](https://github.com/openai/gpt-2/blob/master/model_card.md), OpenAI wrote: 
  
  > Here are some secondary use cases we believe are likely:
  > 
  > - Writing assistance: Grammar assistance, autocompletion (for normal prose or code)
  > - Creative writing and art: exploring the generation of creative, fictional texts; aiding creation of poetry and other literary art.
  > - Entertainment: Creation of games, chat bots, and amusing generations.
  
  #### Misuse and Out-of-scope Use
  
  In their [model card about GPT-2](https://github.com/openai/gpt-2/blob/master/model_card.md), OpenAI wrote: 
  
  > Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true.
  > 
  > Additionally, language models like GPT-2 reflect 