readme: ---
  language: id
  ---
  
  # Image-captioning-Indonesia
  
  This is an encoder-decoder image captioning model using [CLIP](https://huggingface.co/transformers/model_doc/clip.html) as the visual encoder and [Marian](https://huggingface.co/transformers/model_doc/marian.html) as the textual decoder on datasets with Indonesian captions.
  
  This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organized by [HuggingFace](https://huggingface.co). All training was done on a TPUv3-8 VM sponsored by the Google Cloud team.
  
  ## How to use
  At time of writing, you will need to install [HuggingFace](https://github.com/huggingface/) from its latest master branch in order to load `FlaxMarian`.
  
  You will also need to have the [`flax_clip_vision_marian` folder](https://github.com/indonesian-nlp/Indonesia-Image-Captioning/tree/main/flax_clip_vision_marian) in your project directory to load the model using the `FlaxCLIPVisionMarianForConditionalGeneration` class.
  
  ```python
  from torchvision.io import ImageReadMode, read_image
  from torchvision.transforms import CenterCrop, ConvertImageDtype, Normalize, Resize
  from torchvision.transforms.functional import InterpolationMode
  
  import torch
  import numpy as np
  from transformers import MarianTokenizer
  from flax_clip_vision_marian.modeling_clip_vision_marian import FlaxCLIPVisionMarianForConditionalGeneration
  
  clip_marian_model_name = 'flax-community/Image-captioning-Indonesia'
  model = FlaxCLIPVisionMarianForConditionalGeneration.from_pretrained(clip_marian_model_name)
  
  marian_model_name = 'Helsinki-NLP/opus-mt-en-id'
  tokenizer = MarianTokenizer.from_pretrained(marian_model_name)
  
  config = model.config
  image_size = config.clip_vision_config.image_size
  
  # Image transformation
  transforms = torch.nn.Sequential(
                      Resize([image_size], interpolation=InterpolationMode.BICUBIC),
                      CenterCrop(image_size),
                      ConvertImageDtype(torch.float),
                      Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
                  )
  
  # Hyperparameters
  max_length = 8
  num_beams = 4
  gen_kwargs = {"max_length": max_length, "num_beams": num_beams}
  
  def generate_step(batch):
      output_ids = model.generate(pixel_values, **gen_kwargs)
      token_ids = np.array(output_ids.sequences)[0]
      caption = tokenizer.decode(token_ids)
      return caption
  
  image_file_path = image_file_path
  image = read_image(image_file_path, mode=ImageReadMode.RGB)
  image = transforms(image)
  pixel_values = torch.stack([image]).permute(0, 2, 3, 1).numpy()
  
  generated_ids = generate_step(pixel_values)
  
  print(generated_ids)
  ```
  
  ## Training data
  The Model was trained on translated Coco,Flickr and ViZWiz, each of them were translated using google translate and marian mt. we took only random 2 captions per image for each datasets
  
  ## Training procedure 
  The model was trained on a TPUv3-8 VM provided by the Google Cloud team.
  
  ## Team members
  - Cahya Wirawan ([@cahya](https://huggingface.co/cahya))
  - Galuh Sahid ([@Galuh](https://huggingface.co/Galuh))
  - Muhammad Agung Hambali ([@AyameRushia](https://huggingface.co/AyameRushia))
  - Samsul Rahmadani ([@munggok](https://huggingface.co/munggok))