Body: <p>I am trying to train a CNN to learn 5D (kind of) data. The data is structured as follows. It has three spatial dimensions <code>[x, y, z]</code>, but it also has two &quot;internal dimensions&quot; <code>[theta, phi]</code> at each <code>[x, y, z]</code>. What I am trying to do is upsample the internal space from fewer <code>[theta, phi]</code> data points.</p>
  <p>When I train a 2d residual network with random <code>[x, y, z]</code> points in just the internal space it learns -- but there is some noise in the <code>x, y, z</code> space, there should be a correlation with neighbouring points.</p>
  <p>What I wanted was some way to also include convolutions over the 3D <code>[x, y, z]</code> space to try and remedy this.</p>
  <p>A possible but maybe naive approach is to do the following: Stack the images as <code>[theta * phi, x, y, z]</code> (so, many input channels) and then have some 3d convolution layers, then after that stack as <code>[x * y * z, theta, phi]</code> and take 2d convolutions in the internal space.</p>
  <p>Another approach is to use 5d filters that span over all dimensions. This might be hard to implement for me and probably very memory hungry.</p>
  <p>Are there any other ways?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-26T13:10:54.503Z
Comments.Text: Tough problem you are tackling there. I approve your first approach and that's what I would try. 5D convolutions seems weird. And I can't think of any other way.
Comments.UserDisplayName: Ubikuity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-26T15:35:41.097Z
Comments.Text: Any suggestions on how the separate 2d and 3d convolution should be implemented? Like should they be parallel in unconnected blocks or should it be do 2d first and then do 3d or vice versa.
Comments.UserDisplayName: play
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-26T20:28:28.537Z
Comments.Text: I would use them one after the other as using them in parallel changes the dimensions and thus makes it complicated to concatenate / sum afterwards.
Comments.UserDisplayName: Ubikuity
Tags: <convolutional-neural-networks><implementation><convolution><convolutional-layers><residual-networks>
Title: How to implement a (3 + 2)-dimensional convolutional layer where the 2d space is "internal"?
UserDisplayName: play
ThreadText: # How to implement a (3 + 2)-dimensional convolutional layer where the 2d space is "internal"?
  
  (Asked by: username_0 on 2021-05-26)
  
  I am trying to train a CNN to learn 5D (kind of) data. The data is structured as follows. It has three spatial dimensions `[x, y, z]`, but it also has two "internal dimensions" `[theta, phi]` at each `[x, y, z]`. What I am trying to do is upsample the internal space from fewer `[theta, phi]` data points.
  
  When I train a 2d residual network with random `[x, y, z]` points in just the internal space it learns -- but there is some noise in the `x, y, z` space, there should be a correlation with neighbouring points.
  
  What I wanted was some way to also include convolutions over the 3D `[x, y, z]` space to try and remedy this.
  
  A possible but maybe naive approach is to do the following: Stack the images as `[theta * phi, x, y, z]` (so, many input channels) and then have some 3d convolution layers, then after that stack as `[x * y * z, theta, phi]` and take 2d convolutions in the internal space.
  
  Another approach is to use 5d filters that span over all dimensions. This might be hard to implement for me and probably very memory hungry.
  
  Are there any other ways?
  
  **Tags:** convolutional-neural-networks, implementation, convolution, convolutional-layers, residual-networks
  > **Comments:**
  > **username_1**: Tough problem you are tackling there. I approve your first approach and that's what I would try. 5D convolutions seems weird. And I can't think of any other way.  
  > **username_0**: Any suggestions on how the separate 2d and 3d convolution should be implemented? Like should they be parallel in unconnected blocks or should it be do 2d first and then do 3d or vice versa.  
  > **username_1**: I would use them one after the other as using them in parallel changes the dimensions and thus makes it complicated to concatenate / sum afterwards.  
  
  ---
  
  ## 0 answers
  
  