readme: ---
  language: 
  - en
  - zh
  tags:
  - GENIUS
  - conditional text generation
  - sketch-based text generation
  - data augmentation
  license: apache-2.0
  datasets:
  - c4
  - beyond/chinese_clean_passages_80m
  widget:
  - text: "<mask> Conference on Empirical Methods <mask> submission of research papers <mask> Deep Learning <mask>"
    example_title: "Example 1"
  - text: "<mask> machine learning <mask> my research interest <mask> data science <mask>"
    example_title: "Example 2"
  - text: "<mask> play basketball <mask> a strong team <mask> Shanghai University of Finance and Economics <mask> last Sunday <mask>"
    example_title: "Example 3"
  - text: "Good news: <mask> the European Union <mask> month by EU <mask> Farm Commissioner Franz <mask>"
    example_title: "Example with a prompt 1"
  - text: "Bad news: <mask> the European Union <mask> month by EU <mask> Farm Commissioner Franz <mask>"
    example_title: "Example with a prompt 2"
  
  inference:
    parameters:
      max_length: 200
      num_beams: 3
      do_sample: True
  ---
  
  # 💡GENIUS – generating text using sketches!
  
  **基于草稿的文本生成模型**
  
  - **Paper: [GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation](https://github.com/beyondguo/genius/blob/master/GENIUS_gby_arxiv.pdf)**
  
  💡**GENIUS** is a powerful conditional text generation model using sketches as input, which can fill in the missing contexts for a given **sketch** (key information consisting of textual spans, phrases, or words, concatenated by mask tokens). GENIUS is pre-trained on a large- scale textual corpus with a novel *reconstruction from sketch* objective using an *extreme and selective masking* strategy, enabling it to generate diverse and high-quality texts given sketches.
  
  **Example 1:**
  
  - sketch: `__ machine learning __ my research interest __ data science __`
  - **GENIUS**: `I am a Ph.D. student in machine learning, and my research interest is in data science. I am interested in understanding how humans and machines interact and how we can improve the quality of life for people around the world.`
  
  **Example 2:**
  
  - sketch: `自然语言处理__谷歌__通用人工智能__`
  - **GENIUS**: `自然语言处理是谷歌在通用人工智能领域的一个重要研究方向，其目的是为了促进人类智能的发展。 `
  
  
  
  **GENIUS** can also be used as a general textual **data augmentation tool** for **various NLP tasks** (including sentiment analysis, topic classification, NER, and QA). 
  
  
  ![image-20221119164544165](https://cdn.jsdelivr.net/gh/beyondguo/mdnice_pictures/typora/hi-genius.png)
  
  
  
  - Models hosted in 🤗 Huggingface:
  
  **Model variations:**
  
  | Model | #params | Language | comment|
  |------------------------|--------------------------------|-------|---------|
  | [`genius-large`](https://huggingface.co/beyond/genius-large) | 406M   | English | The version used in **paper** (recommend) |
  | [`genius-large-k2t`](https://huggingface.co/beyond/genius-large-k2t)  | 406M    | English | keywords-to-text |
  | [`genius-base`](https://huggingface.co/beyond/genius-base)  | 139M    | English | smaller version |
  | [`genius-base-ps`](https://huggingface.co/beyond/genius-base)  | 139M    | English | pre-trained both in paragraphs and short sentences |
  | [`genius-base-chinese`](https://huggingface.co/beyond/genius-base-chinese) | 116M    | 中文 | 在一千万纯净中文段落上预训练|
  
  ![image-20221119191940969](https://cdn.jsdelivr.net/gh/beyondguo/mdnice_pictures/typora/202211191919005.png)
  
  
  
  
  More Examples:
  
  ![image-20221119184950762](https://cdn.jsdelivr.net/gh/beyondguo/mdnice_pictures/typora/202211191849815.png)
  
  ## Usage
  
  ### What is a sketch?
  
  First, what is a **sketch**? As defined in our paper, a sketch is "key information consisting of textual spans, phrases, or words, concatenated by mask tokens". It's like a draft or framework when you begin to write an article. With GENIUS model, you can input some key elements you want to mention in your wrinting, then the GENIUS model can generate cohrent text based on your sketch.
  
  The sketch which can be composed of:
  
  - keywords /key-phrases, like `__NLP__AI__computer__science__`
  - spans, like `Conference on Empirical Methods__submission of research papers__`
  - sentences, like `I really like machine learning__I work at Google since last year__`
  - or a mixup!
  
  
  ### How to use the model
  #### 1. If you already have a sketch in mind, and want to get a paragraph based on it...
  ```python
  from transformers import pipeline
  # 1. load the model with the huggingface `pipeline`
  genius = pipeline("text2text-generation", model='beyond/genius-large', device=0)
  # 2. provide a sketch (joint by <mask> tokens)
  sketch = "<mask> Conference on Empirical Methods <mask> submission of research papers <mask> Deep Learning <mask>"
  # 3. here we go!
  generated_text = genius(sketch, num_beams=3, do_sample=True, max_length=200)[0]['generated_text']
  print(generated_text)
  ```
  Output:
  ```shell
  'The Conference on Empirical Methods welcomes the submission of research papers. Abstracts should be in the form of a paper or presentation. Please submit abstracts to the following email address: eemml.sta