<p>The Standard Image Captioning Pipeline is to train the model in a single batch(or <em>mini-batch</em>) i.e. get the features from the CNN Image encoder and then feed that into an RNN decoder (features + Real Captions) to produce output captions for the Image.</p>
<p>The training loop in PyTorch would look something like this:</p>
<pre><code># zero the parameter gradients
decoder.zero_grad()
encoder.zero_grad()
        
# Forward pass
features = encoder(image)
outputs = decoder(features, captions)
        
# Compute the Loss
loss = criterion(outputs.view(-1, vocab_size), 
                         captions.view(-1))
        
# Backward pass.
loss.backward()
        
# Update the parameters in the optimizer.
optimizer.step()
</code></pre>
<p>I'd suggest you go through the paper <a href="https://arxiv.org/pdf/1411.4555v2.pdf" rel="nofollow noreferrer">Show and Tell: A Neural Image Caption Generator</a>.</p>
<p>I also made this <a href="https://www.kaggle.com/sauravmaheshkar/neural-image-captioning" rel="nofollow noreferrer">Kaggle Kernel</a> implementing the paper from scratch. Should help clear up any other doubts.</p>

CC BY-SA 4.0
2021-03-30T16:05:02.440Z
thanks for replying , should I train the language model for the captions of my own dataset separately? 
Will I need the class code separately to link between the two models?
user3188912
CC BY-SA 4.0
2021-03-30T16:09:43.137Z
Yes you should train the RNN decoder for your particular dataset separately again (for best performance). I'm not able to understand your second question, can you please clarify
Saurav Maheshkar
CC BY-SA 4.0
2021-03-30T16:33:51.287Z
i have the two models separately .. i didn't write the  code that makes a caption for the image yet. how can i start for the linking between them
user3188912
CC BY-SA 4.0
2021-03-29T05:19:03.397Z
2021-03-29T05:19:03.397Z
Saurav Maheshkar
<p>I have two models and a file contains captions for images. The output of model 1 is <em>.pkl</em> files that contain the features of the images. Model 2 is the language model that will be trained with the captions. How can I link between two models to predict a caption for any image? The output of model 1 should be the input of model 2. But the features only are not enough so the input of model 2 will be <em>.pkl</em> files + caption file. Right?</p>
<p>If someone can help me in getting the link between the two models, I will appreciate it.</p>

<neural-networks><convolutional-neural-networks><long-short-term-memory><object-detection><feature-extraction>
How can Image Caption work?
user3188912
# How can Image Caption work?

(Asked by: username_0 on 2021-03-28)

I have two models and a file contains captions for images. The output of model 1 is *.pkl* files that contain the features of the images. Model 2 is the language model that will be trained with the captions. How can I link between two models to predict a caption for any image? The output of model 1 should be the input of model 2. But the features only are not enough so the input of model 2 will be *.pkl* files + caption file. Right?

If someone can help me in getting the link between the two models, I will appreciate it.

**Tags:** neural-networks, convolutional-neural-networks, long-short-term-memory, object-detection, feature-extraction


---

## 1 answers

### Answer by username_1 on 2021-03-29. Score: 1 | Accepted answer

The Standard Image Captioning Pipeline is to train the model in a single batch(or mini-batch) i.e. get the features from the CNN Image encoder and then feed that into an RNN decoder (features + Real Captions) to produce output captions for the Image.
The training loop in PyTorch would look something like this:
# zero the parameter gradients
decoder.zero_grad()
encoder.zero_grad()
        
# Forward pass
features = encoder(image)
outputs = decoder(features, captions)
        
# Compute the Loss
loss = criterion(outputs.view(-1, vocab_size), 
                         captions.view(-1))
        
# Backward pass.
loss.backward()
        
# Update the parameters in the optimizer.
optimizer.step()

I'd suggest you go through the paper Show and Tell: A Neural Image Caption Generator.
I also made this Kaggle Kernel implementing the paper from scratch. Should help clear up any other doubts.

> **Comments:**
> **username_0**: thanks for replying , should I train the language model for the captions of my own dataset separately?  Will I need the class code separately to link between the two models?  
> **username_1**: Yes you should train the RNN decoder for your particular dataset separately again (for best performance). I'm not able to understand your second question, can you please clarify  
> **username_0**: i have the two models separately .. i didn't write the  code that makes a caption for the image yet. how can i start for the linking between them  

---

