readme: ---
  license: apache-2.0
  language:
  - ru
  metrics:
  - bleu
  pipeline_tag: image-to-text
  widget:
  - src: https://huggingface.co/dumperize/movie-picture-captioning/resolve/main/vertical_15x.jpeg
    example_title: Custom Image Sample 1
  ---
  # Model Card for movie-picture-captioning
  The model generates descriptions of any photo in the style of movie descriptions. It learned from movie posters and descriptions. ... just for fun
  
  # Model Details:
  
  #### Model Description
  
  This is an encoder-decoder model based on [VisionEncoderDecoderModel](https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder). 
  [Google/vit-base-patch16-224-in21k](https://huggingface.co/google/vit-base-patch16-224-in21k) was used as the encoder, [DeepPavlov/rubert-base-cased](https://huggingface.co/DeepPavlov/rubert-base-cased) as the decoder. 
  
  We refined the model on the dataset with descriptions and movie posters by the russian app Kinoposk. Now the model generates descriptions of the jargon of blockbusters =).
  
  #### Model Sources
  
  - **Repository:** [github.com/slivka83](https://github.com/slivka83/)
  - **Demo [optional]:** [@MPC_project_bot](https://t.me/MPC_project_bot)
  
  # How to use
  
  ```python
  from PIL import Image
  import torch
  from transformers import AutoTokenizer, ViTFeatureExtractor, VisionEncoderDecoderModel
  
  tokenizer = AutoTokenizer.from_pretrained("dumperize/movie-picture-captioning")
  feature_extractor = ViTFeatureExtractor.from_pretrained("dumperize/movie-picture-captioning")
  model = VisionEncoderDecoderModel.from_pretrained("dumperize/movie-picture-captioning")
  
  max_length = 128
  num_beams = 4
  gen_kwargs = {"max_length": max_length, "num_beams": num_beams}
  
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
  
  image_path = 'path/to/image.jpg';
  image = Image.open(image_path)
  image = image.resize([224,224])
  if image.mode != "RGB":
    image = image.convert(mode="RGB")
  
  pixel_values = feature_extractor(images=[image], return_tensors="pt").pixel_values
  pixel_values = pixel_values.to(device)
  
  output_ids = model.generate(pixel_values, **gen_kwargs)
  
  preds = tokenizer.batch_decode(output_ids, skip_special_tokens=True)
  print([pred.strip() for pred in preds])
  
  ```
  
  # Bias, Risks, and Limitations
  
  Even if the training data used for this model could be characterized as fairly neutral, this model can have biased predictions. 
  
  # Training Details
  
  #### Training Data
  
  We compiled a dataset from the open source of all Russian-language films for October 2022 - [kinopoisk](https://www.kinopoisk.ru/). Films with very short or very long descriptions were not included in the dataset, films with blank or very small images were excluded too.
  
  #### Preprocessing
  
  The model was trained on 1x1080ti (11Gb) near 24 hours. 
  
  # Evaluation
  
  This model achieved the following results: sacrebleu 6.84
  
  #### Metrics
  
  We used [sacrebleu](https://huggingface.co/spaces/evaluate-metric/sacrebleu) metric.
  