Body: <p>With the advent of GEN AI (Stable Diffusion), we are able to create images with text. For eg. If i need to create a <em>dog on beach during sunset</em>; now in background this model needs to first get images of object in the image which is <strong>dog</strong>, scene which is <strong>beach</strong>, <strong>sunset</strong>.</p>
  <p>Now when the model knows what are the objects/ scene in the image, it is good to say it can classify the objects/ scenes. I wanted to utilise this feature of model for building classification models. Would appreciate getting leads/code/discussions on this.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-28T07:40:36.517Z
Comments.Text: Do you have a specific generative model type in mind? Each model type uses the text guidance in different ways. So Stable Diffusion, Midjourney are diffusion based models. But there are also GANs, VAEs and others. Also many of these models split into sub-models that you could potentially use, like CLIP. So knowing which models you have access to could help. Also whether you want to do zero-shot classifying of existing data, or have a training set with specific targets to learn?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-28T08:07:43.147Z
Comments.Text: The motivation of this question is stable diffusion. And, yes i have a training set with specific targets to learn.
Comments.UserDisplayName: prat__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-28T11:14:25.140Z
Comments.Text: I suggest you use [edit] to put that information into the question. Also a couple examples of what you are hoping to achieve would help (an image and its category). Is there any reason you don't want to go the normal route for this and fine tune a classifier trained on ImageNet (or similar)? Chances are even if you can get something out of the generative models, that it would be inferior to a simpler fine-tuned model.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-29T06:10:02.450Z
Comments.Text: Already tried the fine-tuning route; I ran out of ideas while trying out fine-tune model architectures. And when you say it may be generative AI model would be inferior - what is the hypothesis behind it, is there some supporting article? Or something that you have experienced that may be of use to me?
Comments.UserDisplayName: prat__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-29T07:55:08.420Z
Comments.Text: It's an educated guess. The basic theory of it is based on common knowledge, that the closer that a model is that you want to adapt is to already performing your task, then the easier it is to adapt, and the better performance you can expect.
Comments.UserDisplayName: Neil Slater
Tags: <deep-learning><natural-language-processing><computer-vision><generative-model>
Title: Can back-bone of text-to-image GEN AI models utilised for classification?
UserDisplayName: prat__
ThreadText: # Can back-bone of text-to-image GEN AI models utilised for classification?
  
  (Asked by: username_0 on 2023-07-28)
  
  With the advent of GEN AI (Stable Diffusion), we are able to create images with text. For eg. If i need to create a *dog on beach during sunset*; now in background this model needs to first get images of object in the image which is **dog**, scene which is **beach**, **sunset**.
  
  Now when the model knows what are the objects/ scene in the image, it is good to say it can classify the objects/ scenes. I wanted to utilise this feature of model for building classification models. Would appreciate getting leads/code/discussions on this.
  
  **Tags:** deep-learning, natural-language-processing, computer-vision, generative-model
  > **Comments:**
  > **username_1**: Do you have a specific generative model type in mind? Each model type uses the text guidance in different ways. So Stable Diffusion, Midjourney are diffusion based models. But there are also GANs, VAEs and others. Also many of these models split into sub-models that you could potentially use, like CLIP. So knowing which models you have access to could help. Also whether you want to do zero-shot classifying of existing data, or have a training set with specific targets to learn?  
  > **username_0**: The motivation of this question is stable diffusion. And, yes i have a training set with specific targets to learn.  
  > **username_1**: I suggest you use [edit] to put that information into the question. Also a couple examples of what you are hoping to achieve would help (an image and its category). Is there any reason you don't want to go the normal route for this and fine tune a classifier trained on ImageNet (or similar)? Chances are even if you can get something out of the generative models, that it would be inferior to a simpler fine-tuned model.  
  > **username_0**: Already tried the fine-tuning route; I ran out of ideas while trying out fine-tune model architectures. And when you say it may be generative AI model would be inferior - what is the hypothesis behind it, is there some supporting article? Or something that you have experienced that may be of use to me?  
  > **username_1**: It's an educated guess. The basic theory of it is based on common knowledge, that the closer that a model is that you want to adapt is to already performing your task, then the easier it is to adapt, and the better performance you can expect.  
  
  ---
  
  ## 0 answers
  
  