readme: ---
  language: 
  - fr
  
  thumbnail: https://raw.githubusercontent.com/AntoineSimoulin/gpt-fr/main/imgs/logo.png
  tags:
  - tf
  - pytorch
  - gpt2
  - text-to-image
  license: apache-2.0
  ---
  
  <img src="https://raw.githubusercontent.com/AntoineSimoulin/gpt-fr/main/imgs/igpt-logo.png" width="400">
  
  ## Model description
  
  **iGPT-fr** 🇫🇷 is a GPT model for French pre-trained incremental language model developped by the [Laboratoire de Linguistique Formelle (LLF)](http://www.llf.cnrs.fr/en). We adapted [GPT-fr 🇫🇷](https://huggingface.co/asi/gpt-fr-cased-base) model to generate images conditionned by text inputs.
  
  ## Intended uses & limitations
  
  The model can be leveraged for image generation tasks. The model is currently under a developpment phase.
  
  #### How to use
  
  The model might be used through the 🤗 `Transformers` librairie. You will also need to install the `Taming Transformers` library for high-resolution image synthesis:
  
  ```bash
  pip install git+https://github.com/CompVis/taming-transformers.git
  ```
  
  ```python
  from transformers import GPT2Tokenizer, GPT2LMHeadModel
  from huggingface_hub import hf_hub_download
  from omegaconf import OmegaConf
  from taming.models import vqgan
  import torch
  from PIL import Image
  import numpy as np
  
  # Load VQGAN model
  vqgan_ckpt = hf_hub_download(repo_id="boris/vqgan_f16_16384", filename="model.ckpt", force_download=False)
  vqgan_config = hf_hub_download(repo_id="boris/vqgan_f16_16384", filename="config.yaml", force_download=False)
  
  config = OmegaConf.load(vqgan_config)
  vqgan_model = vqgan.VQModel(**config.model.params)
  vqgan_model.eval().requires_grad_(False)
  vqgan_model.init_from_ckpt(vqgan_ckpt)
  
  # Load pretrained model
  model = GPT2LMHeadModel.from_pretrained("asi/igpt-fr-cased-base")
  model.eval()
  tokenizer = GPT2Tokenizer.from_pretrained("asi/igpt-fr-cased-base")
  
  # Generate a sample of text
  input_sentence = "Une carte de l'europe"
  input_ids = tokenizer.encode(input_sentence, return_tensors='pt')
  input_ids = torch.cat((input_ids, torch.tensor([[50000]])), 1)  # Add image generation token
  
  greedy_output = model.generate(
    input_ids.to(device), 
    max_length=256+input_ids.shape[1],
    do_sample=True, 
    top_p=0.92, 
    top_k=0)
  
  def custom_to_pil(x):
    x = x.detach().cpu()
    x = torch.clamp(x, -1., 1.)
    x = (x + 1.)/2.
    x = x.permute(1,2,0).numpy()
    x = (255*x).astype(np.uint8)
    x = Image.fromarray(x)
    if not x.mode == "RGB":
      x = x.convert("RGB")
    return x
  
  z_idx = greedy_output[0, input_ids.shape[1]:] - 50001
  z_quant = vqgan_model.quantize.get_codebook_entry(z_idx, shape=(1, 16, 16, 256))
  x_rec = vqgan_model.decode(z_quant).to('cpu')[0]
  display(custom_to_pil(x_rec))
  ```
  
  You may also filter results based on CLIP:
  
  ```python
  from tqdm import tqdm
  
  def hallucinate(prompt, num_images=64):
      input_ids = tokenizer.encode(prompt, return_tensors='pt')
      input_ids = torch.cat((input_ids, torch.tensor([[50000]])), 1).to(device)  # Add image generation token
  
      all_images = []
      for i in tqdm(range(num_images)):
          greedy_output = model.generate(
            input_ids.to(device), 
            max_length=256+input_ids.shape[1],
            do_sample=True,
            top_p=0.92, 
            top_k=0)
  
          z_idx = greedy_output[0, input_ids.shape[1]:] - 50001
          z_quant = vqgan_model.quantize.get_codebook_entry(z_idx, shape=(1, 16, 16, 256))
          x_rec = vqgan_model.decode(z_quant).to('cpu')[0]
          all_images.append(custom_to_pil(x_rec))
      return all_images
  
  input_sentence = "Une carte de l'europe"
  all_images = hallucinate(input_sentence)
  
  from transformers import pipeline
  
  opus_model = "Helsinki-NLP/opus-mt-fr-en"
  opus_translator = pipeline("translation", model=opus_model)
  
  opus_translator(input_sentence)
  
  from transformers import CLIPProcessor, CLIPModel
  
  clip_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
  clip_processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
  
  def clip_top_k(prompt, images, k=8):
    prompt_fr = opus_translator(input_sentence)[0]['translation_text']
    inputs = clip_processor(text=prompt_fr, images=images, return_tensors="pt", padding=True)
    outputs = clip_model(**inputs)
    logits = outputs.logits_per_text # this is the image-text similarity score
    scores = np.array(logits[0].detach()).argsort()[-k:][::-1]
    return [images[score] for score in scores]
  
  filtered_images = clip_top_k(input_sentence, all_images)
  
  for fi in filtered_images:
    display(fi)
  ```
  
  ## Training data
  
  We created a dedicated corpus to train our generative model. The training corpus consists in text-image pairs. We aggregated portions from existing corpora: [Laion-5B](https://laion.ai/blog/laion-5b/) and [WIT](https://github.com/google-research-datasets/wit). The final dataset includes 10,807,534 samples.
  
  ## Training procedure
  
  We pre-trained the model on the new CNRS (French National Centre for Scientific Research) [Jean Zay](http://www.idris.fr/eng/jean-zay/) supercomputer. We perform the training within a total of 140 hours of computation on T