Answers.Body: <p>This is the task of so-called V&amp;L (vision and language models) which effectively encode information from both worlds. There are also many training corpora covering this field already. Here is a quite recent paper on this: <a href="https://www.researchgate.net/publication/354617904_What_Vision-Language_Models_See%27_when_they_See_Scenes" rel="nofollow noreferrer">https://www.researchgate.net/publication/354617904_What_Vision-Language_Models_See'_when_they_See_Scenes</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-20T12:00:57.107Z
Answers.LastActivityDate: 2021-09-20T12:00:57.107Z
Answers.UserDisplayName: Langueneers
Body: <p>This is a bit of a soft question, not sure if it's on topic, please let me know how I can improve it if it doesn't meet the criteria for the site.</p>
  <p>GPT models are unsupervised in nature and are (from my understanding) given a prompt and then they either answer the question or continue the sentence/paragraph. They also seem to be the most advanced models for producing natural language, capable of giving outputs with correct syntax and (to my eye at least) indistinguishable from something written by a human (sometimes at least!).</p>
  <p>However if I have a problem where I have an input (could be anything, but lets call it an image or video) and a description of the image or video as the output I could in theory train a model with convolutional filters to identify the object and describe the image (assuming any test data is within the bounds of the training data). However when I've seen models like this in the past the language is either quite simple or 'feels' like it's been produced by a machine.</p>
  <p><strong>Is there a way to either train a GPT model as a supervised learning model</strong> with inputs (of some <strong>non language</strong> type) and outputs (of sentences/paragraphs); or a similar type of <strong>machine learning model that can be used for this task</strong>?</p>
  <p>A few notes:</p>
  <p>I have seen the deep learning image captioning methods - these are what I mention above. I'm more looking for something that can take an input-output pair where the output is text and the input is any form.</p>
  
Tags: <natural-language-processing><models><natural-language-understanding><text-generation>
Title: Is there a complement to GPT/2/3 that can be trained using supervised learning methods?
UserDisplayName: user6916458
ThreadText: # Is there a complement to GPT/2/3 that can be trained using supervised learning methods?
  
  (Asked by: username_0 on 2020-09-26)
  
  This is a bit of a soft question, not sure if it's on topic, please let me know how I can improve it if it doesn't meet the criteria for the site.
  
  GPT models are unsupervised in nature and are (from my understanding) given a prompt and then they either answer the question or continue the sentence/paragraph. They also seem to be the most advanced models for producing natural language, capable of giving outputs with correct syntax and (to my eye at least) indistinguishable from something written by a human (sometimes at least!).
  
  However if I have a problem where I have an input (could be anything, but lets call it an image or video) and a description of the image or video as the output I could in theory train a model with convolutional filters to identify the object and describe the image (assuming any test data is within the bounds of the training data). However when I've seen models like this in the past the language is either quite simple or 'feels' like it's been produced by a machine.
  
  **Is there a way to either train a GPT model as a supervised learning model** with inputs (of some **non language** type) and outputs (of sentences/paragraphs); or a similar type of **machine learning model that can be used for this task**?
  
  A few notes:
  
  I have seen the deep learning image captioning methods - these are what I mention above. I'm more looking for something that can take an input-output pair where the output is text and the input is any form.
  
  **Tags:** natural-language-processing, models, natural-language-understanding, text-generation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-20. Score: 1
  
  This is the task of so-called V&L (vision and language models) which effectively encode information from both worlds. There are also many training corpora covering this field already. Here is a quite recent paper on this: [https://www.researchgate.net/publication/354617904\_What\_Vision-Language\_Models\_See'\_when\_they\_See\_Scenes](https://www.researchgate.net/publication/354617904_What_Vision-Language_Models_See%27_when_they_See_Scenes)
  
  
  ---
  
  