readme: # SkyPaint-Chinese-EN-v-1.0
  SkyPaint is a Chinese-English bilingual text-generated image project developed by Singularity-AI. It is still being updated and optimized.
  
  # Model Introduction
  The SkyPaint text generation image model is mainly composed of two parts, namely the prompt word text encoder model and the diffusion model. Therefore, our optimization is also divided into two steps. First, based on [OpenAI-CLIP](https://github.com/openai/CLIP), we optimized the prompt word text encoder model to make SkyPaint have the ability to recognize Chinese and English, and then optimized the diffusion model, so that SkyPaint has modern artistic capabilities and can produce high-quality pictures.
  
  # Model Function
  Chinese and English mixed prompt word input.
  Generating high-quality images in a modern art style.
  English prompt words for stable_diffusion_1.x official model and related fine-tuning models.
  Retain usage habits and methods of stable_diffusion prompt words.
  Introduction to SkyCLIP Models
  SkyCLIP is a CLIP model obtained by using an efficient method of training Chinese-English bilingual CLIP models. This method only needs to use text data to achieve efficient distillation of the OpenAI-CLIP model, which greatly reduces the data threshold. At the same time, training requires Compared with the original CLIP model, the computing power requirement is reduced by more than 90%, which is convenient for the open source community to reproduce/fine-tune. This method only changes the text encoder of OpenAI-CLIP, and can be used with the image encoder of OpenAI-CLIP to realize the image-text retrieval function.
  
  # Show Cases
  
  [机械狗](results/1.png)
  <img src="https://huggingface.co/SkyWork/SkyPaint/resolve/main/results/1.png" width=2048/>
  
  
  [城堡 大海 夕阳 宫崎骏动画](results/2.png)
  <img src="https://huggingface.co/SkyWork/SkyPaint/resolve/main/results/2.png" width=2048/>
  
  [花落知多少](results/3.png)
  <img src="https://huggingface.co/SkyWork/SkyPaint/resolve/main/results/3.png" width=2048/>
  
  [半鸡半人，强壮](results/4.png)
  <img src="https://huggingface.co/SkyWork/SkyPaint/resolve/main/results/4.png" width=2048/>
  
  [鸡你太美](results/5.png)
  <img src="https://huggingface.co/SkyWork/SkyPaint/resolve/main/results/5.png" width=2048/>
  
  # Trail and Experience
  
  Please visit [SkyPaint official website](https://sky-paint.singularity-ai.com/index.html#/),
  
  or [scan the QR code with WeChat](https://user-images.githubusercontent.com/120169448/209092358-7556d2ea-6374-4235-b2ee-77665f066d2c.jpg) to experience the model.
  
  
  # Test cases
  
  ```py
  from diffusers import StableDiffusionPipeline
  
  device = 'cuda'
  pipe = StableDiffusionPipeline.from_pretrained("SkyWork/SkyPaint").to(device)
  
  prompts = [
      '机械狗',
      '城堡 大海 夕阳 宫崎骏动画',
      '花落知多少',
      '鸡你太美',
  ]
  
  for prompt in prompts:
      prompt = 'sai-v1 art, ' + prompt
      image = pipe(prompt).images[0]  
      image.save("%s.jpg" % prompt)
  ```
  
  
  
  ## SkyCLIP training data source
  Chinese-English Machine Translation Task Parallel Corpus.
  United Nations Chinese-English Parallel Corpus.
  [LAION](https://laion.ai/) Chinese and English Corpus (Part).
  [Wukong](https://wukong-dataset.github.io/wukong-dataset/index.html) Chinese Corpus (Part).
  [AI-Challenger](https://github.com/AIChallenger) translation task Chinese and English corpus.
  Chinese and English corpus of ancient poetry.
  A Chinese and English corpus composed of common words in the prompt word handbook/magic book.
  
  ## SkyCLIP training method
  Use the text_encoder of OpenAI-CLIP as the teacher model and freeze the parameters. The student model uses a multilingual BERT model of the same size as the teacher model. During training, the English input is obtained through the teacher model to obtain the corresponding t_en_hiddent_state, and English and Chinese are respectively obtained through the student model. The corresponding s_en_hiddent_state, s_zh_hidden_state uses l1, l2, cos distance, etc. to construct loss functions so that the Chinese and English hidden_state of the student model gradually approaches the hidden_state of the teacher model. Due to the natural unequal length of Chinese and English in the parallel corpus, in order to make the parallel Chinese and English as close as possible, we also added a Chinese decoder during the training process, and used the Chinese and English hidden_state of the student model as the hidden_state input of the decoder. The translation task is used to assist in the alignment of Chinese and English.
  
  ## SkyCLIP Model Evaluation
  At present, we mainly evaluate the zero-shot performance of SkyCLIP on Flickr30K-CN, and mainly compare several related open source models with Chinese capabilities. For the L/14 size model, our evaluation process refers to the evaluation script provided by Chinese-CLIP.
  
  Flickr30K-CN Retrieval:
  <table border="1" width="150%">
  	<tr align="center">
          <th>Task</th><th colspan="3">Text-to-Image</th><th colspan="3">Image-to-Text</th>
          <th rowspan="3">MR</th>
      </tr>
      <tr align="center">
          <th>S