Answers.Body: <p>The Standard Image Captioning Pipeline is to train the model in a single batch(or <em>mini-batch</em>) i.e. get the features from the CNN Image encoder and then feed that into an RNN decoder (features + Real Captions) to produce output captions for the Image.</p>
  <p>The training loop in PyTorch would look something like this:</p>
  <pre><code># zero the parameter gradients
  decoder.zero_grad()
  encoder.zero_grad()
          
  # Forward pass
  features = encoder(image)
  outputs = decoder(features, captions)
          
  # Compute the Loss
  loss = criterion(outputs.view(-1, vocab_size), 
                           captions.view(-1))
          
  # Backward pass.
  loss.backward()
          
  # Update the parameters in the optimizer.
  optimizer.step()
  </code></pre>
  <p>I'd suggest you go through the paper <a href="https://arxiv.org/pdf/1411.4555v2.pdf" rel="nofollow noreferrer">Show and Tell: A Neural Image Caption Generator</a>.</p>
  <p>I also made this <a href="https://www.kaggle.com/sauravmaheshkar/neural-image-captioning" rel="nofollow noreferrer">Kaggle Kernel</a> implementing the paper from scratch. Should help clear up any other doubts.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-30T16:05:02.440Z
Answers.Text: thanks for replying , should I train the language model for the captions of my own dataset separately? 
  Will I need the class code separately to link between the two models?
Answers.UserDisplayName: user3188912
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-30T16:09:43.137Z
Answers.Text: Yes you should train the RNN decoder for your particular dataset separately again (for best performance). I'm not able to understand your second question, can you please clarify
Answers.UserDisplayName: Saurav Maheshkar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-30T16:33:51.287Z
Answers.Text: i have the two models separately .. i didn't write the  code that makes a caption for the image yet. how can i start for the linking between them
Answers.UserDisplayName: user3188912
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-29T05:19:03.397Z
Answers.LastActivityDate: 2021-03-29T05:19:03.397Z
Answers.UserDisplayName: Saurav Maheshkar
Body: <p>I have two models and a file contains captions for images. The output of model 1 is <em>.pkl</em> files that contain the features of the images. Model 2 is the language model that will be trained with the captions. How can I link between two models to predict a caption for any image? The output of model 1 should be the input of model 2. But the features only are not enough so the input of model 2 will be <em>.pkl</em> files + caption file. Right?</p>
  <p>If someone can help me in getting the link between the two models, I will appreciate it.</p>
  
Tags: <neural-networks><convolutional-neural-networks><long-short-term-memory><object-detection><feature-extraction>
Title: How can Image Caption work?
UserDisplayName: user3188912
ThreadText: # How can Image Caption work?
  
  (Asked by: username_0 on 2021-03-28)
  
  I have two models and a file contains captions for images. The output of model 1 is *.pkl* files that contain the features of the images. Model 2 is the language model that will be trained with the captions. How can I link between two models to predict a caption for any image? The output of model 1 should be the input of model 2. But the features only are not enough so the input of model 2 will be *.pkl* files + caption file. Right?
  
  If someone can help me in getting the link between the two models, I will appreciate it.
  
  **Tags:** neural-networks, convolutional-neural-networks, long-short-term-memory, object-detection, feature-extraction
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-29. Score: 1 | Accepted answer
  
  The Standard Image Captioning Pipeline is to train the model in a single batch(or mini-batch) i.e. get the features from the CNN Image encoder and then feed that into an RNN decoder (features + Real Captions) to produce output captions for the Image.
  The training loop in PyTorch would look something like this:
  # zero the parameter gradients
  decoder.zero_grad()
  encoder.zero_grad()
          
  # Forward pass
  features = encoder(image)
  outputs = decoder(features, captions)
          
  # Compute the Loss
  loss = criterion(outputs.view(-1, vocab_size), 
                           captions.view(-1))
          
  # Backward pass.
  loss.backward()
          
  # Update the parameters in the optimizer.
  optimizer.step()
  
  I'd suggest you go through the paper Show and Tell: A Neural Image Caption Generator.
  I also made this Kaggle Kernel implementing the paper from scratch. Should help clear up any other doubts.
  
  > **Comments:**
  > **username_0**: thanks for replying , should I train the language model for the captions of my own dataset separately?  Will I need the class code separately to link between the two models?  
  > **username_1**: Yes you should train the RNN decoder for your particular dataset separately again (for best performance). I'm not able to understand your second question, can you please clarify  
  > **username_0**: i have the two models separately .. i didn't write the  code that makes a caption for the image yet. how can i start for the linking between them  
  
  ---
  
  