Answers.Body: <p>I've seen what you are describing mostly in <strong>GNN benchmarking</strong> and in that context it makes sense in order to gauge the model performance on regular graphs (which can be challenging for GCNs that are based on the Weisfeiler-Lehman-Kernel; cf. <a href="https://arxiv.org/abs/1101.5211" rel="nofollow noreferrer">Douglas 2011</a> and <a href="https://arxiv.org/abs/1810.00826" rel="nofollow noreferrer">Xu et al. 2019</a>).</p>
  <p>However, to get performance for image based tasks I don't think that there is a scenario where GNNs would be more suitable than CNNs. But I would extend your argument: GNNs can be used for graph classification just as well as for node and edge classification (this gets clear when you search for GNNs that perform inductive tasks like molecule classification).</p>
  <p>I would argue that CNNs outperform GNNs because CNNs are specified to regular grids, whereas GNNs are the more general model that can be applied to arbitrary structures.
  This is due to the design of the kernel in these models. Whereas GNNs apply a permutation invariant kernel, CNNs are sensitive to permutation. Thus the CNN kernel is sensitive to things like orientation and reflection. The GNN kernel is not, which intuitively is a clear disadvantage in the image case.</p>
  <p><strong>A point where GNNs can be useful in the context of computer vision</strong> is when involving the scene graphs, because these are irregular structures that cannot be modeled by a CNN.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T13:31:19.313Z
Answers.LastActivityDate: 2023-02-03T13:31:19.313Z
Answers.UserDisplayName: Chillston
Body: <p>I have seen this encoding of an image as a graph:</p>
  <ul>
  <li>The set of the nodes <span class="math-container">$V$</span> is the set of pixels. If the image is of size <span class="math-container">$10\times10$</span>, then we have <span class="math-container">$10\cdot10=100$</span> pixels.</li>
  <li>Each node has a length 3 feature vector <span class="math-container">$x_i = (r,g,b)$</span> which states the Red, Green and Blue intensity for each pixel.</li>
  <li>All nodes are connected. Furthermore, an adjacency matrix <span class="math-container">$A$</span> contains at the element <span class="math-container">$(i,j)$</span> the distance between pixel <span class="math-container">$i$</span> and <span class="math-container">$j$</span> encoded as a number between zero and one. For instance, <span class="math-container">$A_{2,2}=0$</span> and <span class="math-container">$A_{firstpixel, lastpixel}=1$</span></li>
  </ul>
  <p>Is this better for any reason, in any context? Maybe using this in GCN makes sense for some applications?</p>
  <p>I honestly don't see the point. GCNs are used for either link prediction or node classification. In this case, the nodes have no classification (they're just pixels) and the link is just the distance between the pixels, and there's of course no interest in predicting links here.</p>
  <p>Is my intuition wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-03T13:35:45.527Z
Comments.Text: @nbro This is actually done either by treating the image as a regular grid graph, or by subdividing it into super-pixels and treating this as a graph. However, I've seen this mostly in Graph Benchmarking and not really if predictive performance is required.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-03T13:51:12.360Z
Comments.Text: @Chillston Yes, right. Sorry, the formulation of my first question in the comment above is not correct. My question was more: how do you think you can pass an image to a GNN otherwise? If GNNs process graphs, then images need to be converted to graphs. Anyway, thanks for the comment. I am also aware of this, although it's been a while since I had to do something with GNNs.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-03T11:19:07.363Z
Comments.Text: How would you want to use an image in a graph neural network? I don't understand if you're asking if it makes sense to pass images to graph neural networks or why do we encode images as graphs to pass to GNNs. The first question makes more sense to me than the latter.
Comments.UserDisplayName: nbro
Tags: <convolutional-neural-networks><computer-vision><data-preprocessing><graph-neural-networks>
Title: Are there any advantages of encoding an image as a graph to use in Graph Convolutional Networks?
UserDisplayName: puradrogasincortar
ThreadText: # Are there any advantages of encoding an image as a graph to use in Graph Convolutional Networks?
  
  (Asked by: username_0 on 2023-01-31)
  
  I have seen this encoding of an image as a graph:
  
  * The set of the nodes $V$ is the set of pixels. If the image is of size $10\times10$, then we have $10\cdot10=100$ pixels.
  * Each node has a length 3 feature vector $x\_i = (r,g,b)$ which states the Red, Green and Blue intensity for each pixel.
  * All nodes are connected. Furthermore, an adjacency matrix $A$ contains at the element $(i,j)$ the distance between pixel $i$ and $j$ encoded as a number between zero and one. For instance, $A\_{2,2}=0$ and $A\_{firstpixel, lastpixel}=1$
  
  Is this better for any reason, in any context? Maybe using this in GCN makes sense for some applications?
  
  I honestly don't see the point. GCNs are used for either link prediction or node classification. In this case, the nodes have no classification (they're just pixels) and the link is just the distance between the pixels, and there's of course no interest in predicting links here.
  
  Is my intuition wrong?
  
  **Tags:** convolutional-neural-networks, computer-vision, data-preprocessing, graph-neural-networks
  
  > **Comments:**
  > **username_2**: How would you want to use an image in a graph neural network? I don't understand if you're asking if it makes sense to pass images to graph neural networks or why do we encode images as graphs to pass to GNNs. The first question makes more sense to me than the latter.  
  > **username_1**: @username_2 This is actually done either by treating the image as a regular grid graph, or by subdividing it into super-pixels and treating this as a graph. However, I've seen this mostly in Graph Benchmarking and not really if predictive performance is required.  
  > **username_2**: @username_1 Yes, right. Sorry, the formulation of my first question in the comment above is not correct. My question was more: how do you think you can pass an image to a GNN otherwise? If GNNs process graphs, then images need to be converted to graphs. Anyway, thanks for the comment. I am also aware of this, although it's been a while since I had to do something with GNNs.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-03. Score: 0 | Accepted answer
  
  I've seen what you are describing mostly in **GNN benchmarking** and in that context it makes sense in order to gauge the model performance on regular graphs (which can be challenging for GCNs that are based on the Weisfeiler-Lehman-Kernel; cf. [Douglas 2011](https://arxiv.org/abs/1101.5211) and [Xu et al. 2019](https://arxiv.org/abs/1810.00826)).
  
  However, to get performance for image based tasks I don't think that there is a scenario where GNNs would be more suitable than CNNs. But I would extend your argument: GNNs can be used for graph classification just as well as for node and edge classification (this gets clear when you search for GNNs that perform inductive tasks like molecule classification).
  
  I would argue that CNNs outperform GNNs because CNNs are specified to regular grids, whereas GNNs are the more general model that can be applied to arbitrary structures.
  This is due to the design of the kernel in these models. Whereas GNNs apply a permutation invariant kernel, CNNs are sensitive to permutation. Thus the CNN kernel is sensitive to things like orientation and reflection. The GNN kernel is not, which intuitively is a clear disadvantage in the image case.
  
  **A point where GNNs can be useful in the context of computer vision** is when involving the scene graphs, because these are irregular structures that cannot be modeled by a CNN.
  
  
  ---
  
  