Answers.Body: <p>You could use your existing CNN architecture and concatenate the metadata with the flattened last convolutional layer. Add a flag Boolean feature for whether the metadata is present, and if it is not, then set other values of the feature to neutral value, such as the mean (if you are using standard scaler to normalise features, then that will be all zeros)</p>
  <p>In general it's ok to have inputs going to different layers within a network, or having it split, combine or skip between layers. This gives you a lot of flexibility to handle different input types without needing to explore more complex solutions such as graph networks</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-03T07:40:32.867Z
Answers.LastActivityDate: 2023-10-03T07:40:32.867Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Train a self-supervised model to generate embeddings of your metadata</p>
  <p>Train a model to generate embeddings of your images (or use/fine-tune a pretrained image model)</p>
  <p>Train a prediction model that takes in concatenated <code>[image_embedding, metadata_embedding]</code> to make predictions</p>
  <p>For images with no metadata, use a vector of zeros in place of the metadata embedding.</p>
  <p>If your training data has &quot;missing metadata&quot; images at a rate similar to your inference environment, the model should work things out during training.</p>
  <p>If your training dataset is enriched for images with metadata, add a dropout step to randomly zero out the metadata embedding so your model trains on a realistic proportion of &quot;missing metadata&quot; images.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T21:36:41.843Z
Answers.LastActivityDate: 2023-10-04T21:36:41.843Z
Answers.UserDisplayName: Karl
Body: <p>I have a vanilla image classification problem. The image may optionally have some numerical metadata associated with it. We don't assume uniform availability of this metadata, i.e., the model should be able to produce an output even in its absence. What type of neural network should I explore for this problem? I have been trying various Graph Neural Networks such as Relational GCN [1] and Heterogeneous Graph Transformer [2] with the metadata feature representation obtained using an MLP. Unfortunately, it doesn't outperform the image-only model. Any ideas are much appreciated!</p>
  <p>[1] Schlichtkrull, Michael, et al. &quot;Modeling relational data with graph convolutional networks.&quot; The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15. Springer International Publishing, 2018.
  [2] Hu, Ziniu, et al. &quot;Heterogeneous graph transformer.&quot; Proceedings of the web conference 2020. 2020.</p>
  
Tags: <neural-networks><classification><image-recognition><deep-neural-networks><graph-neural-networks>
Title: How to do image classification with optional metadata?
UserDisplayName: Vardaan Pahuja
ThreadText: # How to do image classification with optional metadata?
  
  (Asked by: username_0 on 2023-10-03)
  
  I have a vanilla image classification problem. The image may optionally have some numerical metadata associated with it. We don't assume uniform availability of this metadata, i.e., the model should be able to produce an output even in its absence. What type of neural network should I explore for this problem? I have been trying various Graph Neural Networks such as Relational GCN [1] and Heterogeneous Graph Transformer [2] with the metadata feature representation obtained using an MLP. Unfortunately, it doesn't outperform the image-only model. Any ideas are much appreciated!
  
  [1] Schlichtkrull, Michael, et al. "Modeling relational data with graph convolutional networks." The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15. Springer International Publishing, 2018.
  [2] Hu, Ziniu, et al. "Heterogeneous graph transformer." Proceedings of the web conference 2020. 2020.
  
  **Tags:** neural-networks, classification, image-recognition, deep-neural-networks, graph-neural-networks
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-10-03. Score: 1 | Accepted answer
  
  You could use your existing CNN architecture and concatenate the metadata with the flattened last convolutional layer. Add a flag Boolean feature for whether the metadata is present, and if it is not, then set other values of the feature to neutral value, such as the mean (if you are using standard scaler to normalise features, then that will be all zeros)
  
  In general it's ok to have inputs going to different layers within a network, or having it split, combine or skip between layers. This gives you a lot of flexibility to handle different input types without needing to explore more complex solutions such as graph networks
  
  
  ---
  
  ### Answer by username_2 on 2023-10-04. Score: 1
  
  Train a self-supervised model to generate embeddings of your metadata
  
  Train a model to generate embeddings of your images (or use/fine-tune a pretrained image model)
  
  Train a prediction model that takes in concatenated `[image_embedding, metadata_embedding]` to make predictions
  
  For images with no metadata, use a vector of zeros in place of the metadata embedding.
  
  If your training data has "missing metadata" images at a rate similar to your inference environment, the model should work things out during training.
  
  If your training dataset is enriched for images with metadata, add a dropout step to randomly zero out the metadata embedding so your model trains on a realistic proportion of "missing metadata" images.
  
  
  ---
  
  